Oct 31 12:31:15.136752 ip-10-0-131-157 systemd[1]: Starting Kubernetes Kubelet... Oct 31 12:31:15.516623 ip-10-0-131-157 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 12:31:15.516623 ip-10-0-131-157 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 12:31:15.516623 ip-10-0-131-157 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 12:31:15.516623 ip-10-0-131-157 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 12:31:15.522564 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522478 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:31:15.522564 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522560 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:31:15.522564 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522565 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:31:15.522564 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522568 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:31:15.522691 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522571 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:31:15.522691 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522574 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:31:15.522691 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522577 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:31:15.522691 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522579 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:31:15.522691 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522582 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:31:15.522691 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522585 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:31:15.522691 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522596 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:31:15.522691 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522599 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:31:15.522691 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522602 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:31:15.522691 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522604 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:31:15.522691 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522607 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:31:15.522691 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522610 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:31:15.522691 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522613 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:31:15.522691 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522616 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:31:15.522691 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522618 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:31:15.522691 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522621 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:31:15.522691 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522623 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:31:15.522691 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522628 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:31:15.522691 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522632 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:31:15.523184 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522634 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:31:15.523184 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522637 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:31:15.523184 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522639 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:31:15.523184 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522643 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:31:15.523184 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522645 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:31:15.523184 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522648 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:31:15.523184 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522651 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:31:15.523184 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522654 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:31:15.523184 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522656 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:31:15.523184 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522659 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:31:15.523184 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522662 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:31:15.523184 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522664 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:31:15.523184 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522667 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:31:15.523184 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522670 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:31:15.523184 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522673 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:31:15.523184 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522675 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:31:15.523184 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522678 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:31:15.523184 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522680 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:31:15.523184 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522683 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:31:15.523645 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522685 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:31:15.523645 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522688 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:31:15.523645 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522691 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:31:15.523645 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522694 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:31:15.523645 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522697 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:31:15.523645 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522699 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:31:15.523645 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522702 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:31:15.523645 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522705 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:31:15.523645 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522707 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:31:15.523645 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522710 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:31:15.523645 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522716 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:31:15.523645 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522719 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:31:15.523645 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522722 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:31:15.523645 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522724 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:31:15.523645 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522727 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:31:15.523645 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522729 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:31:15.523645 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522732 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:31:15.523645 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522735 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:31:15.523645 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522738 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:31:15.523645 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522741 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:31:15.524153 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522743 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:31:15.524153 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522746 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:31:15.524153 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522749 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:31:15.524153 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522751 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:31:15.524153 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522754 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:31:15.524153 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522757 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:31:15.524153 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522759 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:31:15.524153 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522762 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:31:15.524153 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522765 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:31:15.524153 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522767 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:31:15.524153 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522770 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:31:15.524153 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522772 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:31:15.524153 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522775 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:31:15.524153 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522777 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:31:15.524153 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522780 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:31:15.524153 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522783 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:31:15.524153 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522785 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:31:15.524153 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522788 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:31:15.524153 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522790 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:31:15.524153 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522793 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:31:15.524643 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522795 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:31:15.524643 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522798 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:31:15.524643 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522801 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:31:15.524643 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522804 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:31:15.524643 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522807 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:31:15.524643 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522809 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:31:15.524643 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522811 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:31:15.524643 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522814 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:31:15.524643 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522816 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:31:15.524643 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522819 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:31:15.524643 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522822 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:31:15.524643 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522824 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:31:15.524643 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522828 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:31:15.524643 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522832 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:31:15.524643 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522835 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:31:15.524643 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522838 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:31:15.524643 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522841 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:31:15.524643 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522844 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:31:15.524643 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522848 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:31:15.524643 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522851 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:31:15.525103 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522854 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:31:15.525103 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522856 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:31:15.525103 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522858 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:31:15.525103 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.522861 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:31:15.525103 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523655 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:31:15.525103 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523662 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:31:15.525103 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523666 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:31:15.525103 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523669 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:31:15.525103 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523672 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:31:15.525103 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523675 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:31:15.525103 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523677 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:31:15.525103 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523680 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:31:15.525103 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523682 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:31:15.525103 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523685 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:31:15.525103 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523689 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:31:15.525103 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523692 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:31:15.525103 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523695 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:31:15.525103 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523697 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:31:15.525103 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523700 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:31:15.525103 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523702 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:31:15.525616 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523705 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:31:15.525616 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523707 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:31:15.525616 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523710 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:31:15.525616 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523713 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:31:15.525616 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523717 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:31:15.525616 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523721 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:31:15.525616 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523724 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:31:15.525616 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523727 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:31:15.525616 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523730 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:31:15.525616 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523732 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:31:15.525616 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523735 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:31:15.525616 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523738 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:31:15.525616 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523741 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:31:15.525616 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523743 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:31:15.525616 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523746 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:31:15.525616 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523749 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:31:15.525616 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523751 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:31:15.525616 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523754 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:31:15.525616 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523757 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:31:15.526066 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523759 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:31:15.526066 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523762 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:31:15.526066 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523765 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:31:15.526066 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523767 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:31:15.526066 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523770 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:31:15.526066 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523772 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:31:15.526066 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523775 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:31:15.526066 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523779 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:31:15.526066 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523782 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:31:15.526066 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523784 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:31:15.526066 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523787 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:31:15.526066 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523789 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:31:15.526066 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523792 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:31:15.526066 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523794 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:31:15.526066 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523797 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:31:15.526066 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523800 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:31:15.526066 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523802 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:31:15.526066 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523805 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:31:15.526066 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523807 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:31:15.526066 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523810 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:31:15.526554 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523813 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:31:15.526554 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523815 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:31:15.526554 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523817 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:31:15.526554 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523820 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:31:15.526554 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523824 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:31:15.526554 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523827 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:31:15.526554 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523830 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:31:15.526554 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523833 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:31:15.526554 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523835 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:31:15.526554 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523838 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:31:15.526554 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523840 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:31:15.526554 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523843 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:31:15.526554 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523845 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:31:15.526554 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523849 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:31:15.526554 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523855 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:31:15.526554 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523858 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:31:15.526554 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523860 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:31:15.526554 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523863 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:31:15.526554 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523865 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:31:15.527044 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523869 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:31:15.527044 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523872 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:31:15.527044 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523875 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:31:15.527044 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523877 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:31:15.527044 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523880 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:31:15.527044 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523882 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:31:15.527044 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523885 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:31:15.527044 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523888 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:31:15.527044 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523890 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:31:15.527044 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523893 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:31:15.527044 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523896 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:31:15.527044 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523898 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:31:15.527044 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523901 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:31:15.527044 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523903 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:31:15.527044 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523905 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:31:15.527044 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523908 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:31:15.527044 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523910 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:31:15.527044 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523914 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:31:15.527044 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523917 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:31:15.527044 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523920 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:31:15.527516 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523922 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:31:15.527516 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523925 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:31:15.527516 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523927 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:31:15.527516 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523930 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:31:15.527516 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523932 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:31:15.527516 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523935 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:31:15.527516 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523937 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:31:15.527516 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523940 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:31:15.527516 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523944 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:31:15.527516 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523947 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:31:15.527516 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523949 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:31:15.527516 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.523952 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:31:15.527516 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525223 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:31:15.527516 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525230 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:31:15.527516 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525233 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:31:15.527516 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525236 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:31:15.527516 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525240 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:31:15.527516 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525242 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:31:15.527516 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525246 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:31:15.527963 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525248 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:31:15.527963 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525251 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:31:15.527963 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525254 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:31:15.527963 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525256 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:31:15.527963 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525259 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:31:15.527963 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525262 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:31:15.527963 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525264 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:31:15.527963 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525267 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:31:15.527963 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525269 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:31:15.527963 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525272 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:31:15.527963 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525276 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:31:15.527963 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525280 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:31:15.527963 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525284 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:31:15.527963 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525287 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:31:15.527963 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525290 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:31:15.527963 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525293 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:31:15.527963 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525295 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:31:15.527963 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525298 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:31:15.527963 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525302 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:31:15.528425 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525304 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:31:15.528425 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525306 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:31:15.528425 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525310 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:31:15.528425 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525313 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:31:15.528425 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525316 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:31:15.528425 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525319 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:31:15.528425 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525322 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:31:15.528425 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525324 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:31:15.528425 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525327 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:31:15.528425 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525329 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:31:15.528425 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525332 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:31:15.528425 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525335 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:31:15.528425 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525337 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:31:15.528425 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525340 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:31:15.528425 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525342 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:31:15.528425 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525345 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:31:15.528425 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525347 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:31:15.528425 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525349 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:31:15.528425 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525353 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:31:15.528885 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525357 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:31:15.528885 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525360 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:31:15.528885 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525363 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:31:15.528885 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525366 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:31:15.528885 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525369 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:31:15.528885 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525372 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:31:15.528885 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525374 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:31:15.528885 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525377 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:31:15.528885 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525380 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:31:15.528885 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525382 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:31:15.528885 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525385 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:31:15.528885 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525387 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:31:15.528885 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525389 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:31:15.528885 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525392 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:31:15.528885 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525394 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:31:15.528885 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525397 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:31:15.528885 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525400 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:31:15.528885 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525402 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:31:15.528885 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525405 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:31:15.529316 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525408 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:31:15.529316 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525411 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:31:15.529316 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525414 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:31:15.529316 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525416 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:31:15.529316 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525419 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:31:15.529316 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525421 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:31:15.529316 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525424 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:31:15.529316 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525427 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:31:15.529316 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525429 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:31:15.529316 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525432 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:31:15.529316 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525453 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:31:15.529316 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525457 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:31:15.529316 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525459 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:31:15.529316 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525462 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:31:15.529316 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525465 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:31:15.529316 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525467 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:31:15.529316 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525470 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:31:15.529316 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525473 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:31:15.529316 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525476 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:31:15.529316 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525478 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:31:15.529791 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525481 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:31:15.529791 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525483 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:31:15.529791 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525485 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:31:15.529791 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525488 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:31:15.529791 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525491 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:31:15.529791 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525493 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:31:15.529791 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525496 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:31:15.529791 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525498 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:31:15.529791 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525501 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:31:15.529791 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525503 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:31:15.529791 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525506 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:31:15.529791 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525509 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:31:15.529791 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525511 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:31:15.529791 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525514 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:31:15.529791 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525517 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:31:15.529791 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525519 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:31:15.529791 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525522 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:31:15.529791 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525525 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:31:15.529791 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525528 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:31:15.529791 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525530 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:31:15.530255 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525532 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:31:15.530255 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.525535 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:31:15.530255 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526673 2577 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 12:31:15.530255 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526682 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 12:31:15.530255 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526688 2577 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 12:31:15.530255 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526693 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 12:31:15.530255 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526698 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 12:31:15.530255 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526701 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 12:31:15.530255 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526706 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 12:31:15.530255 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526710 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 12:31:15.530255 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526714 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 12:31:15.530255 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526717 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 12:31:15.530255 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526720 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 12:31:15.530255 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526723 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 12:31:15.530255 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526726 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 12:31:15.530255 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526729 2577 flags.go:64] FLAG: --cgroup-root="" Oct 31 12:31:15.530255 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526732 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 12:31:15.530255 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526735 2577 flags.go:64] FLAG: --client-ca-file="" Oct 31 12:31:15.530255 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526738 2577 flags.go:64] FLAG: --cloud-provider="external" Oct 31 12:31:15.530255 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526741 2577 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 12:31:15.530255 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526746 2577 flags.go:64] FLAG: --cluster-domain="" Oct 31 12:31:15.530255 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526749 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 12:31:15.530255 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526752 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 12:31:15.530255 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526756 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 12:31:15.530835 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526759 2577 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 12:31:15.530835 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526763 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 12:31:15.530835 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526766 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 12:31:15.530835 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526769 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 12:31:15.530835 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526772 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 12:31:15.530835 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526775 2577 flags.go:64] FLAG: --contention-profiling="false" Oct 31 12:31:15.530835 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526778 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 12:31:15.530835 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526781 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 12:31:15.530835 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526784 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 12:31:15.530835 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526787 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 12:31:15.530835 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526792 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 12:31:15.530835 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526795 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 12:31:15.530835 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526798 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 12:31:15.530835 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526800 2577 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 12:31:15.530835 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526803 2577 flags.go:64] FLAG: --enable-server="true" Oct 31 12:31:15.530835 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526808 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 12:31:15.530835 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526813 2577 flags.go:64] FLAG: --event-burst="100" Oct 31 12:31:15.530835 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526816 2577 flags.go:64] FLAG: --event-qps="50" Oct 31 12:31:15.530835 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526819 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 12:31:15.530835 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526822 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 12:31:15.530835 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526825 2577 flags.go:64] FLAG: --eviction-hard="" Oct 31 12:31:15.530835 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526830 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 12:31:15.530835 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526833 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 12:31:15.530835 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526836 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 12:31:15.530835 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526839 2577 flags.go:64] FLAG: --eviction-soft="" Oct 31 12:31:15.531491 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526842 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 12:31:15.531491 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526846 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 12:31:15.531491 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526849 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 12:31:15.531491 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526852 2577 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 12:31:15.531491 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526855 2577 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 12:31:15.531491 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526858 2577 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 12:31:15.531491 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526861 2577 flags.go:64] FLAG: --feature-gates="" Oct 31 12:31:15.531491 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526866 2577 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 12:31:15.531491 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526869 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 12:31:15.531491 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526872 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 12:31:15.531491 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526875 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 12:31:15.531491 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526878 2577 flags.go:64] FLAG: --healthz-port="10248" Oct 31 12:31:15.531491 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526881 2577 flags.go:64] FLAG: --help="false" Oct 31 12:31:15.531491 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526884 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-131-157.us-east-2.compute.internal" Oct 31 12:31:15.531491 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526887 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 12:31:15.531491 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526890 2577 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 12:31:15.531491 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526893 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 12:31:15.531491 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526897 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 12:31:15.531491 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526900 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 12:31:15.531491 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526903 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 12:31:15.531491 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526906 2577 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 12:31:15.531491 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526909 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 12:31:15.531491 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526913 2577 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 12:31:15.532040 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526917 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 12:31:15.532040 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526920 2577 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 12:31:15.532040 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526923 2577 flags.go:64] FLAG: --kube-reserved="" Oct 31 12:31:15.532040 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526926 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 12:31:15.532040 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526929 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 12:31:15.532040 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526932 2577 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 12:31:15.532040 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526934 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 12:31:15.532040 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526937 2577 flags.go:64] FLAG: --lock-file="" Oct 31 12:31:15.532040 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526940 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 12:31:15.532040 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526943 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 12:31:15.532040 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526946 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 12:31:15.532040 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526951 2577 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 12:31:15.532040 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526954 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 12:31:15.532040 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526957 2577 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 12:31:15.532040 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526960 2577 flags.go:64] FLAG: --logging-format="text" Oct 31 12:31:15.532040 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526962 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 12:31:15.532040 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526966 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 12:31:15.532040 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526969 2577 flags.go:64] FLAG: --manifest-url="" Oct 31 12:31:15.532040 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526972 2577 flags.go:64] FLAG: --manifest-url-header="" Oct 31 12:31:15.532040 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526977 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 12:31:15.532040 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526980 2577 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 12:31:15.532040 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526984 2577 flags.go:64] FLAG: --max-pods="110" Oct 31 12:31:15.532040 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526987 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 12:31:15.532040 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526990 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 12:31:15.532040 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526993 2577 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 12:31:15.532641 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526996 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 12:31:15.532641 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.526999 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 12:31:15.532641 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527001 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 12:31:15.532641 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527004 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 12:31:15.532641 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527012 2577 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 12:31:15.532641 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527016 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 12:31:15.532641 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527020 2577 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 12:31:15.532641 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527025 2577 flags.go:64] FLAG: --pod-cidr="" Oct 31 12:31:15.532641 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527028 2577 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 12:31:15.532641 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527031 2577 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 12:31:15.532641 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527034 2577 flags.go:64] FLAG: --pods-per-core="0" Oct 31 12:31:15.532641 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527037 2577 flags.go:64] FLAG: --port="10250" Oct 31 12:31:15.532641 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527040 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 12:31:15.532641 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527043 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0bc5251286bf527fa" Oct 31 12:31:15.532641 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527046 2577 flags.go:64] FLAG: --qos-reserved="" Oct 31 12:31:15.532641 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527049 2577 flags.go:64] FLAG: --read-only-port="10255" Oct 31 12:31:15.532641 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527052 2577 flags.go:64] FLAG: --register-node="true" Oct 31 12:31:15.532641 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527055 2577 flags.go:64] FLAG: --register-with-taints="" Oct 31 12:31:15.532641 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527059 2577 flags.go:64] FLAG: --registry-burst="10" Oct 31 12:31:15.532641 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527062 2577 flags.go:64] FLAG: --registry-qps="5" Oct 31 12:31:15.532641 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527065 2577 flags.go:64] FLAG: --reserved-cpus="" Oct 31 12:31:15.532641 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527068 2577 flags.go:64] FLAG: --reserved-memory="" Oct 31 12:31:15.532641 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527072 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 12:31:15.532641 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527075 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 12:31:15.532641 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527078 2577 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 12:31:15.532641 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527081 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 12:31:15.533387 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527084 2577 flags.go:64] FLAG: --runonce="false" Oct 31 12:31:15.533387 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527087 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 12:31:15.533387 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527090 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 12:31:15.533387 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527093 2577 flags.go:64] FLAG: --seccomp-default="false" Oct 31 12:31:15.533387 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527096 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 12:31:15.533387 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527099 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 12:31:15.533387 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527102 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 12:31:15.533387 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527105 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 12:31:15.533387 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527108 2577 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 12:31:15.533387 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527111 2577 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 12:31:15.533387 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527114 2577 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 12:31:15.533387 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527116 2577 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 12:31:15.533387 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527121 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 12:31:15.533387 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527125 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 12:31:15.533387 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527128 2577 flags.go:64] FLAG: --system-cgroups="" Oct 31 12:31:15.533387 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527130 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 12:31:15.533387 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527136 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 12:31:15.533387 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527138 2577 flags.go:64] FLAG: --tls-cert-file="" Oct 31 12:31:15.533387 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527141 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 12:31:15.533387 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527146 2577 flags.go:64] FLAG: --tls-min-version="" Oct 31 12:31:15.533387 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527149 2577 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 12:31:15.533387 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527152 2577 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 12:31:15.533387 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527155 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 12:31:15.533387 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527157 2577 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 12:31:15.533387 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527160 2577 flags.go:64] FLAG: --v="2" Oct 31 12:31:15.534333 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527165 2577 flags.go:64] FLAG: --version="false" Oct 31 12:31:15.534333 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527169 2577 flags.go:64] FLAG: --vmodule="" Oct 31 12:31:15.534333 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527172 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 12:31:15.534333 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527176 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 12:31:15.534333 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527310 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:31:15.534333 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527313 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:31:15.534333 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527316 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:31:15.534333 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527319 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:31:15.534333 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527322 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:31:15.534333 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527325 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:31:15.534333 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527327 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:31:15.534333 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527330 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:31:15.534333 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527333 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:31:15.534333 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527336 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:31:15.534333 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527338 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:31:15.534333 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527340 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:31:15.534333 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527343 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:31:15.534333 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527346 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:31:15.534333 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527348 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:31:15.534333 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527352 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:31:15.534837 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527354 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:31:15.534837 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527357 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:31:15.534837 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527360 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:31:15.534837 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527362 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:31:15.534837 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527365 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:31:15.534837 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527368 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:31:15.534837 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527370 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:31:15.534837 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527373 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:31:15.534837 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527375 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:31:15.534837 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527378 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:31:15.534837 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527380 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:31:15.534837 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527383 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:31:15.534837 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527387 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:31:15.534837 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527390 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:31:15.534837 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527393 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:31:15.534837 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527397 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:31:15.534837 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527399 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:31:15.534837 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527402 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:31:15.534837 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527404 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:31:15.535276 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527407 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:31:15.535276 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527410 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:31:15.535276 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527413 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:31:15.535276 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527415 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:31:15.535276 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527418 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:31:15.535276 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527422 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:31:15.535276 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527426 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:31:15.535276 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527429 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:31:15.535276 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527431 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:31:15.535276 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527446 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:31:15.535276 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527449 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:31:15.535276 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527451 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:31:15.535276 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527455 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:31:15.535276 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527458 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:31:15.535276 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527461 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:31:15.535276 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527463 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:31:15.535276 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527466 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:31:15.535276 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527468 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:31:15.535276 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527471 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:31:15.535739 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527474 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:31:15.535739 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527477 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:31:15.535739 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527479 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:31:15.535739 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527482 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:31:15.535739 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527485 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:31:15.535739 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527488 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:31:15.535739 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527490 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:31:15.535739 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527493 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:31:15.535739 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527495 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:31:15.535739 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527498 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:31:15.535739 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527500 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:31:15.535739 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527503 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:31:15.535739 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527505 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:31:15.535739 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527508 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:31:15.535739 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527510 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:31:15.535739 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527514 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:31:15.535739 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527517 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:31:15.535739 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527520 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:31:15.535739 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527522 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:31:15.535739 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527525 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:31:15.536216 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527528 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:31:15.536216 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527530 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:31:15.536216 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527533 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:31:15.536216 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527535 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:31:15.536216 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527538 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:31:15.536216 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527541 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:31:15.536216 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527544 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:31:15.536216 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527546 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:31:15.536216 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527548 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:31:15.536216 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527551 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:31:15.536216 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527554 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:31:15.536216 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527556 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:31:15.536216 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527559 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:31:15.536216 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527561 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:31:15.536216 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527564 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:31:15.536216 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527566 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:31:15.536216 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527569 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:31:15.536216 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527571 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:31:15.536216 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527574 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:31:15.536216 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527576 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:31:15.536705 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527579 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:31:15.536705 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527581 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:31:15.536705 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527583 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:31:15.536705 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527586 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:31:15.536705 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527589 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:31:15.536705 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527591 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:31:15.536705 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527593 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:31:15.536705 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527596 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:31:15.536705 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527599 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:31:15.536705 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527601 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:31:15.536705 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527604 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:31:15.536705 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527606 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:31:15.536705 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.527633 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 12:31:15.536705 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527669 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:31:15.536705 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527674 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:31:15.536705 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527678 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:31:15.537085 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527682 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:31:15.537085 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527685 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:31:15.537085 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527689 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:31:15.537085 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527692 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:31:15.537085 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527695 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:31:15.537085 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527697 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:31:15.537085 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527700 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:31:15.537085 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527703 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:31:15.537085 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527705 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:31:15.537085 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527707 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:31:15.537085 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527710 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:31:15.537085 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527714 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:31:15.537085 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527718 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:31:15.537085 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527721 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:31:15.537085 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527723 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:31:15.537085 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527726 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:31:15.537085 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527728 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:31:15.537085 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527731 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:31:15.537085 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527733 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:31:15.537556 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527736 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:31:15.537556 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527738 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:31:15.537556 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527741 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:31:15.537556 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527743 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:31:15.537556 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527745 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:31:15.537556 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527748 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:31:15.537556 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527755 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:31:15.537556 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527758 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:31:15.537556 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527760 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:31:15.537556 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527763 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:31:15.537556 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527765 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:31:15.537556 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527767 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:31:15.537556 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527770 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:31:15.537556 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527772 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:31:15.537556 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527775 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:31:15.537556 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527779 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:31:15.537556 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527781 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:31:15.537556 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527784 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:31:15.537556 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527786 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:31:15.537556 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527789 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:31:15.538014 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527791 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:31:15.538014 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527794 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:31:15.538014 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527797 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:31:15.538014 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527799 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:31:15.538014 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527802 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:31:15.538014 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527804 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:31:15.538014 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527806 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:31:15.538014 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527809 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:31:15.538014 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527811 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:31:15.538014 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527814 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:31:15.538014 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527816 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:31:15.538014 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527819 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:31:15.538014 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527821 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:31:15.538014 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527823 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:31:15.538014 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527826 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:31:15.538014 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527828 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:31:15.538014 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527831 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:31:15.538014 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527833 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:31:15.538014 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527836 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:31:15.538014 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527839 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:31:15.538504 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527841 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:31:15.538504 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527844 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:31:15.538504 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527846 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:31:15.538504 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527849 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:31:15.538504 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527853 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:31:15.538504 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527855 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:31:15.538504 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527858 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:31:15.538504 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527862 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:31:15.538504 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527864 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:31:15.538504 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527867 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:31:15.538504 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527869 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:31:15.538504 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527872 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:31:15.538504 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527874 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:31:15.538504 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527877 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:31:15.538504 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527879 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:31:15.538504 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527882 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:31:15.538504 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527884 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:31:15.538504 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527887 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:31:15.538504 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527889 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:31:15.538972 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527892 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:31:15.538972 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527894 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:31:15.538972 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527897 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:31:15.538972 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527899 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:31:15.538972 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527902 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:31:15.538972 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527904 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:31:15.538972 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527906 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:31:15.538972 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527909 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:31:15.538972 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527912 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:31:15.538972 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527914 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:31:15.538972 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527916 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:31:15.538972 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527919 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:31:15.538972 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527921 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:31:15.538972 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527924 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:31:15.538972 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527927 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:31:15.538972 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527929 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:31:15.538972 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527932 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:31:15.538972 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527934 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:31:15.538972 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527936 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:31:15.539416 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527939 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:31:15.539416 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527944 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:31:15.539416 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527947 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:31:15.539416 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527950 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:31:15.539416 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527952 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:31:15.539416 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.527955 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:31:15.539416 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.535905 2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 12:31:15.539416 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.535924 2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 12:31:15.539416 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.535949 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 12:31:15.539416 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.535955 2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 12:31:15.539416 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536022 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:31:15.539416 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536027 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:31:15.539416 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536031 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:31:15.539416 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536035 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:31:15.539416 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536038 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:31:15.539416 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536041 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:31:15.539416 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536044 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:31:15.539416 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536046 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:31:15.539416 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536050 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:31:15.539883 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536052 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:31:15.539883 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536055 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:31:15.539883 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536058 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:31:15.539883 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536060 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:31:15.539883 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536063 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:31:15.539883 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536066 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:31:15.539883 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536068 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:31:15.539883 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536071 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:31:15.539883 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536074 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:31:15.539883 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536078 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:31:15.539883 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536082 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:31:15.539883 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536085 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:31:15.539883 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536088 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:31:15.539883 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536095 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:31:15.539883 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536098 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:31:15.539883 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536101 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:31:15.539883 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536103 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:31:15.539883 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536106 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:31:15.539883 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536108 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:31:15.540314 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536111 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:31:15.540314 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536114 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:31:15.540314 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536117 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:31:15.540314 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536120 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:31:15.540314 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536122 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:31:15.540314 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536125 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:31:15.540314 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536128 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:31:15.540314 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536131 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:31:15.540314 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536133 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:31:15.540314 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536136 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:31:15.540314 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536138 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:31:15.540314 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536141 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:31:15.540314 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536143 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:31:15.540314 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536146 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:31:15.540314 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536149 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:31:15.540314 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536151 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:31:15.540314 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536154 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:31:15.540314 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536156 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:31:15.540314 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536159 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:31:15.540783 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536162 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:31:15.540783 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536164 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:31:15.540783 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536169 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:31:15.540783 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536173 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:31:15.540783 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536176 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:31:15.540783 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536179 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:31:15.540783 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536181 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:31:15.540783 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536184 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:31:15.540783 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536186 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:31:15.540783 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536189 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:31:15.540783 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536191 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:31:15.540783 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536194 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:31:15.540783 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536196 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:31:15.540783 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536199 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:31:15.540783 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536202 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:31:15.540783 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536205 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:31:15.540783 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536208 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:31:15.540783 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536211 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:31:15.540783 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536213 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:31:15.541259 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536216 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:31:15.541259 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536218 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:31:15.541259 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536221 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:31:15.541259 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536224 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:31:15.541259 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536227 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:31:15.541259 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536230 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:31:15.541259 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536232 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:31:15.541259 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536235 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:31:15.541259 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536237 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:31:15.541259 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536240 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:31:15.541259 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536242 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:31:15.541259 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536245 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:31:15.541259 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536247 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:31:15.541259 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536251 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:31:15.541259 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536253 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:31:15.541259 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536256 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:31:15.541259 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536258 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:31:15.541259 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536261 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:31:15.541259 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536263 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:31:15.541259 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536266 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:31:15.541737 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536268 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:31:15.541737 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536271 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:31:15.541737 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536273 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:31:15.541737 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536276 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:31:15.541737 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536278 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:31:15.541737 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536281 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:31:15.541737 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536284 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:31:15.541737 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536286 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:31:15.541737 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536289 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:31:15.541737 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536292 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:31:15.541737 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536295 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:31:15.541737 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536297 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:31:15.541737 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536300 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:31:15.541737 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536302 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:31:15.541737 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536305 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:31:15.541737 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536307 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:31:15.541737 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536309 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:31:15.541737 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536312 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:31:15.541737 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536315 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:31:15.541737 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536317 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:31:15.542205 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.536344 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 12:31:15.542205 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536500 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:31:15.542205 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536505 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:31:15.542205 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536508 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:31:15.542205 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536511 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:31:15.542205 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536514 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:31:15.542205 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536518 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:31:15.542205 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536520 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:31:15.542205 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536523 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:31:15.542205 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536526 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:31:15.542205 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536528 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:31:15.542205 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536531 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:31:15.542205 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536533 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:31:15.542205 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536542 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:31:15.542205 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536547 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:31:15.542205 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536550 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:31:15.542722 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536552 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:31:15.542722 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536555 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:31:15.542722 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536558 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:31:15.542722 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536560 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:31:15.542722 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536563 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:31:15.542722 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536566 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:31:15.542722 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536568 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:31:15.542722 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536571 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:31:15.542722 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536574 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:31:15.542722 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536576 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:31:15.542722 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536579 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:31:15.542722 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536581 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:31:15.542722 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536584 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:31:15.542722 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536586 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:31:15.542722 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536589 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:31:15.542722 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536592 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:31:15.542722 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536594 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:31:15.542722 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536597 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:31:15.542722 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536599 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:31:15.543174 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536602 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:31:15.543174 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536604 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:31:15.543174 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536607 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:31:15.543174 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536609 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:31:15.543174 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536613 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:31:15.543174 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536615 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:31:15.543174 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536618 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:31:15.543174 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536621 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:31:15.543174 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536623 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:31:15.543174 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536626 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:31:15.543174 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536628 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:31:15.543174 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536631 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:31:15.543174 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536633 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:31:15.543174 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536636 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:31:15.543174 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536638 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:31:15.543174 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536641 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:31:15.543174 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536643 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:31:15.543174 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536646 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:31:15.543174 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536649 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:31:15.543174 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536651 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:31:15.543654 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536654 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:31:15.543654 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536656 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:31:15.543654 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536658 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:31:15.543654 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536661 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:31:15.543654 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536663 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:31:15.543654 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536666 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:31:15.543654 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536668 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:31:15.543654 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536671 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:31:15.543654 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536673 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:31:15.543654 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536676 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:31:15.543654 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536680 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:31:15.543654 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536683 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:31:15.543654 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536686 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:31:15.543654 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536689 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:31:15.543654 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536691 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:31:15.543654 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536694 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:31:15.543654 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536697 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:31:15.543654 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536700 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:31:15.543654 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536703 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:31:15.543654 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536705 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:31:15.544169 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536707 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:31:15.544169 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536710 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:31:15.544169 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536713 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:31:15.544169 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536715 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:31:15.544169 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536718 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:31:15.544169 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536720 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:31:15.544169 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536723 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:31:15.544169 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536726 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:31:15.544169 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536728 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:31:15.544169 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536731 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:31:15.544169 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536733 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:31:15.544169 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536736 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:31:15.544169 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536738 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:31:15.544169 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536741 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:31:15.544169 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536744 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:31:15.544169 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536746 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:31:15.544169 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536749 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:31:15.544169 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536751 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:31:15.544169 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536754 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:31:15.544169 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536756 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:31:15.544661 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536759 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:31:15.544661 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536762 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:31:15.544661 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536766 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:31:15.544661 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536769 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:31:15.544661 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536771 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:31:15.544661 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536774 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:31:15.544661 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536776 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:31:15.544661 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536779 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:31:15.544661 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536781 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:31:15.544661 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536784 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:31:15.544661 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536787 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:31:15.544661 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536790 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:31:15.544661 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.536814 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 12:31:15.544661 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536846 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:31:15.544661 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536849 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:31:15.545004 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536852 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:31:15.545004 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536855 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:31:15.545004 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536857 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:31:15.545004 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536860 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:31:15.545004 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536865 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:31:15.545004 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536868 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:31:15.545004 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536871 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:31:15.545004 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536873 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:31:15.545004 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536876 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:31:15.545004 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536879 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:31:15.545004 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536881 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:31:15.545004 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536884 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:31:15.545004 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536887 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:31:15.545004 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536889 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:31:15.545004 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536892 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:31:15.545004 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536895 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:31:15.545004 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536897 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:31:15.545004 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536900 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:31:15.545004 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536902 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:31:15.545455 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536905 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:31:15.545455 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536908 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:31:15.545455 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536910 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:31:15.545455 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536913 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:31:15.545455 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536915 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:31:15.545455 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536917 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:31:15.545455 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536920 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:31:15.545455 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536923 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:31:15.545455 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536925 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:31:15.545455 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536929 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:31:15.545455 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536932 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:31:15.545455 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536934 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:31:15.545455 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536936 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:31:15.545455 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536939 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:31:15.545455 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536941 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:31:15.545455 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536944 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:31:15.545455 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536946 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:31:15.545455 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536950 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:31:15.545455 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536953 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:31:15.545455 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536955 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:31:15.545963 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536958 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:31:15.545963 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536960 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:31:15.545963 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536962 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:31:15.545963 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536965 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:31:15.545963 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536967 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:31:15.545963 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536970 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:31:15.545963 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536972 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:31:15.545963 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536975 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:31:15.545963 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536977 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:31:15.545963 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536980 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:31:15.545963 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536982 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:31:15.545963 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536985 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:31:15.545963 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536987 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:31:15.545963 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536990 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:31:15.545963 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536994 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:31:15.545963 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536997 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:31:15.545963 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.536999 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:31:15.545963 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537002 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:31:15.545963 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537004 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:31:15.545963 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537007 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:31:15.546422 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537009 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:31:15.546422 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537012 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:31:15.546422 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537015 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:31:15.546422 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537019 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:31:15.546422 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537023 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:31:15.546422 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537025 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:31:15.546422 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537028 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:31:15.546422 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537030 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:31:15.546422 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537033 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:31:15.546422 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537037 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:31:15.546422 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537039 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:31:15.546422 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537042 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:31:15.546422 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537044 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:31:15.546422 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537047 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:31:15.546422 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537049 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:31:15.546422 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537052 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:31:15.546422 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537054 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:31:15.546422 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537057 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:31:15.546422 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537059 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:31:15.546422 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537062 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:31:15.546892 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537064 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:31:15.546892 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537067 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:31:15.546892 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537069 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:31:15.546892 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537072 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:31:15.546892 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537075 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:31:15.546892 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537077 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:31:15.546892 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537081 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:31:15.546892 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537083 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:31:15.546892 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537086 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:31:15.546892 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537089 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:31:15.546892 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537092 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:31:15.546892 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537095 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:31:15.546892 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537097 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:31:15.546892 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537100 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:31:15.546892 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537105 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:31:15.546892 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537110 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:31:15.546892 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537113 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:31:15.546892 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537116 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:31:15.546892 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537119 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:31:15.547330 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537122 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:31:15.547330 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537124 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:31:15.547330 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537127 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:31:15.547330 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537130 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:31:15.547330 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537132 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:31:15.547330 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:15.537135 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:31:15.547330 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.537196 2577 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 12:31:15.547330 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.539239 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 12:31:15.547330 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.540138 2577 server.go:1014] "Starting client certificate rotation" Oct 31 12:31:15.547330 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.540239 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 12:31:15.547330 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.540282 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 12:31:15.547330 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.542421 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 12:31:15.547330 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.543614 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 12:31:15.556062 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.556043 2577 log.go:25] "Validated CRI v1 runtime API" Oct 31 12:31:15.561704 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.561684 2577 log.go:25] "Validated CRI v1 image API" Oct 31 12:31:15.562598 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.562569 2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 12:31:15.566098 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.566077 2577 fs.go:135] Filesystem UUIDs: map[1b5eaec0-cd27-4b2e-a677-03c9a4fb30a2:/dev/nvme0n1p4 266238b3-2d5c-426e-835b-cefe6aae45c6:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 31 12:31:15.566185 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.566096 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 12:31:15.573149 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.573025 2577 manager.go:217] Machine: {Timestamp:2026-10-31 12:31:15.571537156 +0000 UTC m=+0.323444234 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3108973 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c2c03d63203b0a644ce9fad41a60c SystemUUID:ec2c2c03-d632-03b0-a644-ce9fad41a60c BootID:94656512-3c53-44de-aad9-07c262579e1a Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e0:ac:7d:57 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e0:ac:7d:57 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:82:4c:9f:e9:93:28 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 12:31:15.573921 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.573905 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 12:31:15.574040 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.574023 2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 12:31:15.576423 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.576405 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 12:31:15.577569 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.577544 2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 12:31:15.577743 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.577575 2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-157.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 12:31:15.577824 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.577748 2577 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 12:31:15.577824 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.577761 2577 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 12:31:15.577905 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.577830 2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 12:31:15.577905 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.577858 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 12:31:15.578985 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.578972 2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 12:31:15.583000 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.582985 2577 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 12:31:15.583136 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.583125 2577 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 12:31:15.583192 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.583169 2577 kubelet.go:493] "Attempting to sync node with API server" Oct 31 12:31:15.583192 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.583183 2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 12:31:15.583279 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.583197 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 12:31:15.583279 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.583216 2577 kubelet.go:405] "Adding apiserver pod source" Oct 31 12:31:15.583279 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.583229 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 12:31:15.583606 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.583593 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 12:31:15.586866 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.586849 2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 12:31:15.593358 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.593344 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 12:31:15.594285 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.594272 2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 12:31:15.594322 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.594304 2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 12:31:15.594484 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.594472 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 12:31:15.594519 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.594497 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 12:31:15.594519 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.594507 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 12:31:15.594519 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.594513 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 12:31:15.594596 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.594520 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 12:31:15.594596 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.594530 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 12:31:15.595221 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.595212 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 12:31:15.595255 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.595222 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 12:31:15.595255 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.595230 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 12:31:15.595255 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.595236 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 12:31:15.595255 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.595250 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 12:31:15.595422 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.595258 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 12:31:15.596941 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.596928 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 12:31:15.596941 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.596939 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 12:31:15.598643 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.598618 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vp7j6" Oct 31 12:31:15.601240 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.601226 2577 server.go:1263] "Started kubelet" Oct 31 12:31:15.601310 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.601285 2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 12:31:15.601448 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.601360 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 12:31:15.601525 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.601511 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 12:31:15.602137 ip-10-0-131-157 systemd[1]: Started Kubernetes Kubelet. Oct 31 12:31:15.603100 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.603088 2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 12:31:15.604638 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.604622 2577 server.go:317] "Adding debug handlers to kubelet server" Oct 31 12:31:15.607455 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:15.607420 2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 12:31:15.609949 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.609933 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 12:31:15.610594 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.610575 2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 12:31:15.610701 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.610664 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vp7j6" Oct 31 12:31:15.610770 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:15.610733 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-157.us-east-2.compute.internal\" not found" Oct 31 12:31:15.611468 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.611453 2577 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 12:31:15.611468 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.611462 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 12:31:15.611468 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.611469 2577 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 12:31:15.611701 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.611618 2577 reconstruct.go:97] "Volume reconstruction finished" Oct 31 12:31:15.611701 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.611653 2577 reconciler.go:29] "Reconciler: start to sync state" Oct 31 12:31:15.612093 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.612071 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 12:31:15.612161 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.612095 2577 factory.go:55] Registering systemd factory Oct 31 12:31:15.612161 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.612107 2577 factory.go:223] Registration of the systemd container factory successfully Oct 31 12:31:15.612937 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.612898 2577 factory.go:153] Registering CRI-O factory Oct 31 12:31:15.613024 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.612939 2577 factory.go:223] Registration of the crio container factory successfully Oct 31 12:31:15.613024 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.612965 2577 factory.go:103] Registering Raw factory Oct 31 12:31:15.613024 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.612991 2577 manager.go:1199] Started watching for new ooms in manager Oct 31 12:31:15.616370 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:15.616344 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 31 12:31:15.616482 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:15.616460 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-157.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 31 12:31:15.616685 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:15.615283 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-157.us-east-2.compute.internal.18e39d28e5ad39fc default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-157.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-157.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 12:31:15.601177084 +0000 UTC m=+0.353084163,LastTimestamp:2026-10-31 12:31:15.601177084 +0000 UTC m=+0.353084163,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-157.us-east-2.compute.internal,}" Oct 31 12:31:15.616983 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:15.616844 2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-157.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 31 12:31:15.617155 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.617137 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-157.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 12:31:15.617666 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.617646 2577 manager.go:319] Starting recovery of all containers Oct 31 12:31:15.620347 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:15.620317 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 31 12:31:15.629624 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.629605 2577 manager.go:324] Recovery completed Oct 31 12:31:15.633712 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.633697 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 12:31:15.635518 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.635496 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 12:31:15.635621 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.635539 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 12:31:15.635621 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.635554 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 12:31:15.636244 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.636226 2577 cpu_manager.go:226] "Starting" policy="none" Oct 31 12:31:15.636244 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.636241 2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 12:31:15.636330 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.636233 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 12:31:15.636330 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.636259 2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 12:31:15.636994 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.636962 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 12:31:15.637046 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.637010 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 12:31:15.637046 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.637028 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 12:31:15.638388 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.638376 2577 policy_none.go:50] "Start" Oct 31 12:31:15.638429 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.638392 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 12:31:15.638429 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.638402 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 12:31:15.640471 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.640457 2577 policy_none.go:44] "Start" Oct 31 12:31:15.691994 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.683694 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 12:31:15.691994 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.684045 2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 12:31:15.691994 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.685329 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 12:31:15.691994 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.685352 2577 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 12:31:15.691994 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.685369 2577 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 12:31:15.691994 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:15.685407 2577 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 12:31:15.691994 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.686928 2577 manager.go:341] "Starting Device Plugin manager" Oct 31 12:31:15.691994 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:15.686952 2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 12:31:15.691994 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.686960 2577 server.go:87] "Starting device plugin registration server" Oct 31 12:31:15.691994 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.687210 2577 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 12:31:15.691994 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.687220 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 12:31:15.691994 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.687522 2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 12:31:15.691994 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.687582 2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 12:31:15.691994 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.687587 2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 12:31:15.691994 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:15.688182 2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 12:31:15.691994 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:15.688231 2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-157.us-east-2.compute.internal\" not found" Oct 31 12:31:15.691994 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.689117 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:31:15.785910 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.785832 2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal"] Oct 31 12:31:15.786030 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.785910 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 12:31:15.786940 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.786924 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 12:31:15.787023 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.786953 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 12:31:15.787023 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.786963 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 12:31:15.787547 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.787531 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 12:31:15.788285 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.788268 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 12:31:15.788361 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.788296 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 12:31:15.788361 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.788306 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 12:31:15.788361 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.788326 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-157.us-east-2.compute.internal" Oct 31 12:31:15.789216 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.789204 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 12:31:15.789374 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.789360 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 12:31:15.789458 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.789390 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 12:31:15.789980 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.789964 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 12:31:15.790029 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.789998 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 12:31:15.790029 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.790010 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 12:31:15.790123 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.790015 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 12:31:15.790123 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.790034 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 12:31:15.790123 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.790091 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 12:31:15.792107 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.792093 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 12:31:15.792159 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.792121 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 12:31:15.792790 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.792775 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 12:31:15.792886 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.792806 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 12:31:15.792886 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.792821 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 12:31:15.797361 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.797346 2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-157.us-east-2.compute.internal" Oct 31 12:31:15.797430 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:15.797367 2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-157.us-east-2.compute.internal\": node \"ip-10-0-131-157.us-east-2.compute.internal\" not found" Oct 31 12:31:15.812691 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.812669 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8a68def89a8e2f4b1d88fecf85bfdb5e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal\" (UID: \"8a68def89a8e2f4b1d88fecf85bfdb5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 12:31:15.812795 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.812695 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a68def89a8e2f4b1d88fecf85bfdb5e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal\" (UID: \"8a68def89a8e2f4b1d88fecf85bfdb5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 12:31:15.812795 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.812712 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/72de75e4a93b82a0133be1d6ed9e0a51-config\") pod \"kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal\" (UID: \"72de75e4a93b82a0133be1d6ed9e0a51\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 12:31:15.817734 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:15.817719 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-157.us-east-2.compute.internal\" not found" Oct 31 12:31:15.823042 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:15.823025 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-157.us-east-2.compute.internal\" not found" node="ip-10-0-131-157.us-east-2.compute.internal" Oct 31 12:31:15.825719 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:15.825696 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-157.us-east-2.compute.internal\" not found" node="ip-10-0-131-157.us-east-2.compute.internal" Oct 31 12:31:15.913425 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.913395 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8a68def89a8e2f4b1d88fecf85bfdb5e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal\" (UID: \"8a68def89a8e2f4b1d88fecf85bfdb5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 12:31:15.913425 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.913425 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a68def89a8e2f4b1d88fecf85bfdb5e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal\" (UID: \"8a68def89a8e2f4b1d88fecf85bfdb5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 12:31:15.913646 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.913460 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/72de75e4a93b82a0133be1d6ed9e0a51-config\") pod \"kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal\" (UID: \"72de75e4a93b82a0133be1d6ed9e0a51\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 12:31:15.913646 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.913513 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/72de75e4a93b82a0133be1d6ed9e0a51-config\") pod \"kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal\" (UID: \"72de75e4a93b82a0133be1d6ed9e0a51\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 12:31:15.913646 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.913520 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8a68def89a8e2f4b1d88fecf85bfdb5e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal\" (UID: \"8a68def89a8e2f4b1d88fecf85bfdb5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 12:31:15.913646 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:15.913520 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a68def89a8e2f4b1d88fecf85bfdb5e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal\" (UID: \"8a68def89a8e2f4b1d88fecf85bfdb5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 12:31:15.918521 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:15.918505 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-157.us-east-2.compute.internal\" not found" Oct 31 12:31:16.019251 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:16.019220 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-157.us-east-2.compute.internal\" not found" Oct 31 12:31:16.120001 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:16.119971 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-157.us-east-2.compute.internal\" not found" Oct 31 12:31:16.128148 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:16.128123 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 12:31:16.131296 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:16.131281 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 12:31:16.221021 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:16.220986 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-157.us-east-2.compute.internal\" not found" Oct 31 12:31:16.321565 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:16.321539 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-157.us-east-2.compute.internal\" not found" Oct 31 12:31:16.422113 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:16.422048 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-157.us-east-2.compute.internal\" not found" Oct 31 12:31:16.522614 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:16.522583 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-157.us-east-2.compute.internal\" not found" Oct 31 12:31:16.539756 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:16.539731 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 12:31:16.539878 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:16.539859 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 12:31:16.622189 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:16.622149 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 12:26:16 +0000 UTC" deadline="2028-08-03 09:27:40.449239802 +0000 UTC" Oct 31 12:31:16.622189 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:16.622181 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15404h56m23.827063463s" Oct 31 12:31:16.623244 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:16.623226 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-157.us-east-2.compute.internal\" not found" Oct 31 12:31:16.724156 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:16.724125 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-157.us-east-2.compute.internal\" not found" Oct 31 12:31:16.758833 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:16.758809 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:31:16.816888 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:16.816846 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a68def89a8e2f4b1d88fecf85bfdb5e.slice/crio-0b35abf8cdb506dcaacb590ad330fb6cbbaee3964124e6b0ca499b097254feb4 WatchSource:0}: Error finding container 0b35abf8cdb506dcaacb590ad330fb6cbbaee3964124e6b0ca499b097254feb4: Status 404 returned error can't find the container with id 0b35abf8cdb506dcaacb590ad330fb6cbbaee3964124e6b0ca499b097254feb4 Oct 31 12:31:16.817168 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:16.817143 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72de75e4a93b82a0133be1d6ed9e0a51.slice/crio-cc809f08fd1b1ec8bba0f6df66aee099d71617ec4e73c31c6b501d2db29a95c7 WatchSource:0}: Error finding container cc809f08fd1b1ec8bba0f6df66aee099d71617ec4e73c31c6b501d2db29a95c7: Status 404 returned error can't find the container with id cc809f08fd1b1ec8bba0f6df66aee099d71617ec4e73c31c6b501d2db29a95c7 Oct 31 12:31:16.822605 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:16.822589 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 12:31:16.824492 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:16.824475 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-157.us-east-2.compute.internal\" not found" Oct 31 12:31:16.925227 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:16.925192 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-157.us-east-2.compute.internal\" not found" Oct 31 12:31:17.003946 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:17.003876 2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:31:17.026189 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:17.026165 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-157.us-east-2.compute.internal\" not found" Oct 31 12:31:17.089388 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:17.089362 2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:31:17.111451 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:17.111406 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 12:31:17.124018 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:17.123992 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 12:31:17.125182 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:17.125165 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 12:31:17.133982 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:17.133963 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 12:31:18.920390 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.920360 2577 apiserver.go:52] "Watching apiserver" Oct 31 12:31:18.929468 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.929438 2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 12:31:18.929788 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.929770 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-bgcqc","openshift-image-registry/node-ca-tk5vl","openshift-multus/network-metrics-daemon-8nbvz","openshift-ovn-kubernetes/ovnkube-node-hghw2","kube-system/global-pull-secret-syncer-wgrhs","openshift-multus/multus-additional-cni-plugins-j7fd8","kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-4wqp6","openshift-dns/node-resolver-5gdlb","openshift-network-operator/iptables-alerter-d7j6l","kube-system/konnectivity-agent-crhzp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal","openshift-multus/multus-dklfq"] Oct 31 12:31:18.932539 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.932515 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-d7j6l" Oct 31 12:31:18.934752 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.934719 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-crhzp" Oct 31 12:31:18.936548 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.936525 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 12:31:18.936666 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.936608 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 12:31:18.936666 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.936606 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 12:31:18.936775 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.936667 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8c8p7\"" Oct 31 12:31:18.937737 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.937721 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:31:18.937842 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:18.937812 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nbvz" podUID="bbf2e0b3-d31b-44ca-934c-3710f4262779" Oct 31 12:31:18.938214 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.938194 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hk7cx\"" Oct 31 12:31:18.938325 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.938306 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 12:31:18.938325 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.938315 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 12:31:18.940484 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.940136 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:18.943798 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.943691 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:31:18.943798 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:18.943773 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgrhs" podUID="ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a" Oct 31 12:31:18.944959 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.944937 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j7fd8" Oct 31 12:31:18.945418 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.945398 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 12:31:18.947302 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.947284 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5gdlb" Oct 31 12:31:18.950732 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.950714 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dklfq" Oct 31 12:31:18.953464 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.953446 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tk5vl" Oct 31 12:31:18.956678 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.955864 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:31:18.956678 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:18.955933 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgcqc" podUID="02c6d8f5-97ff-41e0-8e38-71e1b0c87f54" Oct 31 12:31:18.958315 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.958298 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-cni-bin\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:18.958385 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.958323 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b091d91-47bc-4480-85b2-e23a8139df66-ovn-node-metrics-cert\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:18.958385 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.958339 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:18.958385 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.958354 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b4c0e6ed-9c41-4023-9e84-524635b9f1fb-iptables-alerter-script\") pod \"iptables-alerter-d7j6l\" (UID: \"b4c0e6ed-9c41-4023-9e84-524635b9f1fb\") " pod="openshift-network-operator/iptables-alerter-d7j6l" Oct 31 12:31:18.958385 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.958380 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvd64\" (UniqueName: \"kubernetes.io/projected/b4c0e6ed-9c41-4023-9e84-524635b9f1fb-kube-api-access-tvd64\") pod \"iptables-alerter-d7j6l\" (UID: \"b4c0e6ed-9c41-4023-9e84-524635b9f1fb\") " pod="openshift-network-operator/iptables-alerter-d7j6l" Oct 31 12:31:18.958585 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.958399 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-cni-binary-copy\") pod \"multus-additional-cni-plugins-j7fd8\" (UID: \"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35\") " pod="openshift-multus/multus-additional-cni-plugins-j7fd8" Oct 31 12:31:18.958585 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.958420 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/26913ad4-6613-4f3f-9819-9be238088c0a-hosts-file\") pod \"node-resolver-5gdlb\" (UID: \"26913ad4-6613-4f3f-9819-9be238088c0a\") " pod="openshift-dns/node-resolver-5gdlb" Oct 31 12:31:18.958585 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.958445 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26913ad4-6613-4f3f-9819-9be238088c0a-tmp-dir\") pod \"node-resolver-5gdlb\" (UID: \"26913ad4-6613-4f3f-9819-9be238088c0a\") " pod="openshift-dns/node-resolver-5gdlb" Oct 31 12:31:18.958585 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.958461 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2z5c\" (UniqueName: \"kubernetes.io/projected/bbf2e0b3-d31b-44ca-934c-3710f4262779-kube-api-access-j2z5c\") pod \"network-metrics-daemon-8nbvz\" (UID: \"bbf2e0b3-d31b-44ca-934c-3710f4262779\") " pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:31:18.958585 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.958476 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-etc-openvswitch\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:18.958585 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.958489 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-node-log\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:18.958585 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.958502 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-log-socket\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:18.958585 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.958514 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b4c0e6ed-9c41-4023-9e84-524635b9f1fb-host-slash\") pod \"iptables-alerter-d7j6l\" (UID: \"b4c0e6ed-9c41-4023-9e84-524635b9f1fb\") " pod="openshift-network-operator/iptables-alerter-d7j6l" Oct 31 12:31:18.958585 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.958530 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/43421968-0e5b-4e27-be79-104aae5f72e0-agent-certs\") pod \"konnectivity-agent-crhzp\" (UID: \"43421968-0e5b-4e27-be79-104aae5f72e0\") " pod="kube-system/konnectivity-agent-crhzp" Oct 31 12:31:18.958585 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.958554 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j7fd8\" (UID: \"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35\") " pod="openshift-multus/multus-additional-cni-plugins-j7fd8" Oct 31 12:31:18.958585 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.958577 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-run-ovn\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:18.958585 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.958589 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf2e0b3-d31b-44ca-934c-3710f4262779-metrics-certs\") pod \"network-metrics-daemon-8nbvz\" (UID: \"bbf2e0b3-d31b-44ca-934c-3710f4262779\") " pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:31:18.959175 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.958612 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-kubelet\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:18.959175 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.958627 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-run-openvswitch\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:18.959175 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.958659 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b091d91-47bc-4480-85b2-e23a8139df66-ovnkube-script-lib\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:18.959175 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.958699 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a-kubelet-config\") pod \"global-pull-secret-syncer-wgrhs\" (UID: \"ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a\") " pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:31:18.959175 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.958721 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-system-cni-dir\") pod \"multus-additional-cni-plugins-j7fd8\" (UID: \"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35\") " pod="openshift-multus/multus-additional-cni-plugins-j7fd8" Oct 31 12:31:18.959175 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.958753 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-run-netns\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:18.959175 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.958776 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-run-systemd\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:18.959175 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.958852 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-cni-netd\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:18.959175 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.958885 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b091d91-47bc-4480-85b2-e23a8139df66-ovnkube-config\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:18.959175 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.958920 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:18.959175 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.958935 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n79b\" (UniqueName: \"kubernetes.io/projected/0b091d91-47bc-4480-85b2-e23a8139df66-kube-api-access-4n79b\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:18.959175 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.958966 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a-dbus\") pod \"global-pull-secret-syncer-wgrhs\" (UID: \"ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a\") " pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:31:18.959175 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.959002 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/43421968-0e5b-4e27-be79-104aae5f72e0-konnectivity-ca\") pod \"konnectivity-agent-crhzp\" (UID: \"43421968-0e5b-4e27-be79-104aae5f72e0\") " pod="kube-system/konnectivity-agent-crhzp" Oct 31 12:31:18.959175 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.959027 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-var-lib-openvswitch\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:18.959175 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.959053 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-run-ovn-kubernetes\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:18.959175 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.959077 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b091d91-47bc-4480-85b2-e23a8139df66-env-overrides\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:18.959175 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.959128 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j7fd8\" (UID: \"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35\") " pod="openshift-multus/multus-additional-cni-plugins-j7fd8" Oct 31 12:31:18.959837 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.959153 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j7fd8\" (UID: \"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35\") " pod="openshift-multus/multus-additional-cni-plugins-j7fd8" Oct 31 12:31:18.959837 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.959224 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhnsq\" (UniqueName: \"kubernetes.io/projected/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-kube-api-access-rhnsq\") pod \"multus-additional-cni-plugins-j7fd8\" (UID: \"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35\") " pod="openshift-multus/multus-additional-cni-plugins-j7fd8" Oct 31 12:31:18.959837 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.959278 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-779z6\" (UniqueName: \"kubernetes.io/projected/26913ad4-6613-4f3f-9819-9be238088c0a-kube-api-access-779z6\") pod \"node-resolver-5gdlb\" (UID: \"26913ad4-6613-4f3f-9819-9be238088c0a\") " pod="openshift-dns/node-resolver-5gdlb" Oct 31 12:31:18.959837 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.959338 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-systemd-units\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:18.959837 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.959389 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-slash\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:18.959837 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.959424 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a-original-pull-secret\") pod \"global-pull-secret-syncer-wgrhs\" (UID: \"ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a\") " pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:31:18.959837 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.959466 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-cnibin\") pod \"multus-additional-cni-plugins-j7fd8\" (UID: \"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35\") " pod="openshift-multus/multus-additional-cni-plugins-j7fd8" Oct 31 12:31:18.960780 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.960760 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:31:18.961151 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.961132 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 12:31:18.980765 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.980740 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-pfwjd" Oct 31 12:31:18.992907 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:18.992880 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pfwjd" Oct 31 12:31:19.028407 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.028365 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" event={"ID":"8a68def89a8e2f4b1d88fecf85bfdb5e","Type":"ContainerStarted","Data":"0b35abf8cdb506dcaacb590ad330fb6cbbaee3964124e6b0ca499b097254feb4"} Oct 31 12:31:19.029653 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.029612 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal" event={"ID":"72de75e4a93b82a0133be1d6ed9e0a51","Type":"ContainerStarted","Data":"cc809f08fd1b1ec8bba0f6df66aee099d71617ec4e73c31c6b501d2db29a95c7"} Oct 31 12:31:19.039204 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.037942 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 12:31:19.039204 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.037923 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 12:31:19.039204 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.038894 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 12:31:19.039419 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.039386 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 12:31:19.039602 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.039582 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 12:31:19.039847 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.039822 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 12:31:19.040210 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.040194 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 12:31:19.040378 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.040366 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 12:31:19.040484 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.040459 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 12:31:19.040583 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.040568 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 12:31:19.042270 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.042251 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 12:31:19.042517 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.042501 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zfd9q\"" Oct 31 12:31:19.042582 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.042566 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 12:31:19.042627 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.042593 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 12:31:19.042707 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.042687 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-w8vnc\"" Oct 31 12:31:19.042757 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.042718 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 12:31:19.042804 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.042767 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7phhl\"" Oct 31 12:31:19.042851 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.042827 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 12:31:19.042895 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.042886 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 12:31:19.042942 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.042918 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6k457\"" Oct 31 12:31:19.042990 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.042971 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fmbl2\"" Oct 31 12:31:19.043035 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.042994 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 12:31:19.043035 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.043022 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zzvpb\"" Oct 31 12:31:19.043125 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.043074 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 12:31:19.043125 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.043093 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 12:31:19.043216 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.043132 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bkb2g\"" Oct 31 12:31:19.043216 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.043170 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 12:31:19.043216 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.043197 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 12:31:19.043216 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.043216 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 12:31:19.047757 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.047597 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 12:31:19.059796 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.059767 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/24e17633-30a9-4410-b000-2fbb97202951-sys-fs\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:31:19.059917 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.059815 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-var-lib-openvswitch\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.059917 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.059837 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b091d91-47bc-4480-85b2-e23a8139df66-env-overrides\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.059917 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.059895 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j7fd8\" (UID: \"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35\") " pod="openshift-multus/multus-additional-cni-plugins-j7fd8" Oct 31 12:31:19.060068 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.059979 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-var-lib-openvswitch\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.060366 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.060349 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b091d91-47bc-4480-85b2-e23a8139df66-env-overrides\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.060432 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.060399 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-run\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.060432 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.060417 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j7fd8\" (UID: \"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35\") " pod="openshift-multus/multus-additional-cni-plugins-j7fd8" Oct 31 12:31:19.060432 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.060426 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-tuned\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.060571 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.060443 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-systemd-units\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.060571 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.060458 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-slash\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.060571 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.060490 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-systemd-units\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.060571 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.060513 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a-original-pull-secret\") pod \"global-pull-secret-syncer-wgrhs\" (UID: \"ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a\") " pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:31:19.060571 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.060527 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-cnibin\") pod \"multus-additional-cni-plugins-j7fd8\" (UID: \"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35\") " pod="openshift-multus/multus-additional-cni-plugins-j7fd8" Oct 31 12:31:19.060826 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.060580 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-system-cni-dir\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.060826 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.060604 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-slash\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.060826 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.060604 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-etc-kubernetes\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.060826 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.060619 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-cnibin\") pod \"multus-additional-cni-plugins-j7fd8\" (UID: \"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35\") " pod="openshift-multus/multus-additional-cni-plugins-j7fd8" Oct 31 12:31:19.060826 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.060660 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-cni-bin\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.060826 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:19.060705 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 12:31:19.060826 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.060735 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-systemd\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.060826 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.060765 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-cni-bin\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.060826 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.060801 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5hf6\" (UniqueName: \"kubernetes.io/projected/5d3565a5-d07a-4dc0-abd1-e697573baf22-kube-api-access-h5hf6\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.060826 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.060826 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/14dc1e6d-70e1-4ba5-bffb-d9299433756d-serviceca\") pod \"node-ca-tk5vl\" (UID: \"14dc1e6d-70e1-4ba5-bffb-d9299433756d\") " pod="openshift-image-registry/node-ca-tk5vl" Oct 31 12:31:19.061223 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:19.060905 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a-original-pull-secret podName:ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a nodeName:}" failed. No retries permitted until 2026-10-31 12:31:19.560885571 +0000 UTC m=+2.977590025 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a-original-pull-secret") pod "global-pull-secret-syncer-wgrhs" (UID: "ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a") : object "kube-system"/"original-pull-secret" not registered Oct 31 12:31:19.061223 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.060922 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-modprobe-d\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.061223 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.060960 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-etc-openvswitch\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.061223 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.060992 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/43421968-0e5b-4e27-be79-104aae5f72e0-agent-certs\") pod \"konnectivity-agent-crhzp\" (UID: \"43421968-0e5b-4e27-be79-104aae5f72e0\") " pod="kube-system/konnectivity-agent-crhzp" Oct 31 12:31:19.061223 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.061036 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/24e17633-30a9-4410-b000-2fbb97202951-etc-selinux\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:31:19.061223 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.061061 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-kubelet\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.061223 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.061086 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-run-openvswitch\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.061223 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.061110 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-host-run-multus-certs\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.061223 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.061158 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-kubernetes\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.061223 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.061162 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-kubelet\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.061223 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.061186 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/24e17633-30a9-4410-b000-2fbb97202951-socket-dir\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:31:19.061223 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.061192 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-etc-openvswitch\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.061223 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.061210 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-run-openvswitch\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.061806 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.061236 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/24e17633-30a9-4410-b000-2fbb97202951-registration-dir\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:31:19.061806 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.061276 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-system-cni-dir\") pod \"multus-additional-cni-plugins-j7fd8\" (UID: \"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35\") " pod="openshift-multus/multus-additional-cni-plugins-j7fd8" Oct 31 12:31:19.061806 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.061311 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-system-cni-dir\") pod \"multus-additional-cni-plugins-j7fd8\" (UID: \"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35\") " pod="openshift-multus/multus-additional-cni-plugins-j7fd8" Oct 31 12:31:19.061806 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.061316 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhppb\" (UniqueName: \"kubernetes.io/projected/24e17633-30a9-4410-b000-2fbb97202951-kube-api-access-zhppb\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:31:19.061806 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.061360 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-run-netns\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.061806 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.061389 2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 12:31:19.061806 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.061420 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-run-systemd\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.061806 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.061443 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-cni-netd\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.061806 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.061395 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-run-netns\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.061806 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.061476 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-cni-netd\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.061806 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.061530 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b091d91-47bc-4480-85b2-e23a8139df66-ovnkube-config\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.061806 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.061555 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-run-systemd\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.061806 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.061592 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/43421968-0e5b-4e27-be79-104aae5f72e0-konnectivity-ca\") pod \"konnectivity-agent-crhzp\" (UID: \"43421968-0e5b-4e27-be79-104aae5f72e0\") " pod="kube-system/konnectivity-agent-crhzp" Oct 31 12:31:19.061806 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.061685 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-multus-socket-dir-parent\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.061806 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.061788 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-hostroot\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.062333 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.061866 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-run-ovn-kubernetes\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.062333 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.061901 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhnsq\" (UniqueName: \"kubernetes.io/projected/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-kube-api-access-rhnsq\") pod \"multus-additional-cni-plugins-j7fd8\" (UID: \"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35\") " pod="openshift-multus/multus-additional-cni-plugins-j7fd8" Oct 31 12:31:19.062333 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.061954 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-run-ovn-kubernetes\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.062333 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.061958 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-779z6\" (UniqueName: \"kubernetes.io/projected/26913ad4-6613-4f3f-9819-9be238088c0a-kube-api-access-779z6\") pod \"node-resolver-5gdlb\" (UID: \"26913ad4-6613-4f3f-9819-9be238088c0a\") " pod="openshift-dns/node-resolver-5gdlb" Oct 31 12:31:19.062333 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.062088 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bd53d085-6474-45d6-bd59-825fea180a79-multus-daemon-config\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.062333 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.062118 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgs94\" (UniqueName: \"kubernetes.io/projected/bd53d085-6474-45d6-bd59-825fea180a79-kube-api-access-zgs94\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.062333 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.062164 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-host\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.062333 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.062189 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd53d085-6474-45d6-bd59-825fea180a79-cni-binary-copy\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.062333 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.062209 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d3565a5-d07a-4dc0-abd1-e697573baf22-tmp\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.062333 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.062238 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph9jm\" (UniqueName: \"kubernetes.io/projected/02c6d8f5-97ff-41e0-8e38-71e1b0c87f54-kube-api-access-ph9jm\") pod \"network-check-target-bgcqc\" (UID: \"02c6d8f5-97ff-41e0-8e38-71e1b0c87f54\") " pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:31:19.062333 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.062263 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b091d91-47bc-4480-85b2-e23a8139df66-ovn-node-metrics-cert\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.062333 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.062283 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.062333 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.062307 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b4c0e6ed-9c41-4023-9e84-524635b9f1fb-iptables-alerter-script\") pod \"iptables-alerter-d7j6l\" (UID: \"b4c0e6ed-9c41-4023-9e84-524635b9f1fb\") " pod="openshift-network-operator/iptables-alerter-d7j6l" Oct 31 12:31:19.062333 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.062328 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvd64\" (UniqueName: \"kubernetes.io/projected/b4c0e6ed-9c41-4023-9e84-524635b9f1fb-kube-api-access-tvd64\") pod \"iptables-alerter-d7j6l\" (UID: \"b4c0e6ed-9c41-4023-9e84-524635b9f1fb\") " pod="openshift-network-operator/iptables-alerter-d7j6l" Oct 31 12:31:19.062922 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.062348 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-cni-binary-copy\") pod \"multus-additional-cni-plugins-j7fd8\" (UID: \"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35\") " pod="openshift-multus/multus-additional-cni-plugins-j7fd8" Oct 31 12:31:19.062922 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.062409 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/26913ad4-6613-4f3f-9819-9be238088c0a-hosts-file\") pod \"node-resolver-5gdlb\" (UID: \"26913ad4-6613-4f3f-9819-9be238088c0a\") " pod="openshift-dns/node-resolver-5gdlb" Oct 31 12:31:19.062922 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.062438 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26913ad4-6613-4f3f-9819-9be238088c0a-tmp-dir\") pod \"node-resolver-5gdlb\" (UID: \"26913ad4-6613-4f3f-9819-9be238088c0a\") " pod="openshift-dns/node-resolver-5gdlb" Oct 31 12:31:19.062922 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.062460 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-host-run-netns\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.062922 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.062480 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-host-var-lib-cni-bin\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.062922 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.062503 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2z5c\" (UniqueName: \"kubernetes.io/projected/bbf2e0b3-d31b-44ca-934c-3710f4262779-kube-api-access-j2z5c\") pod \"network-metrics-daemon-8nbvz\" (UID: \"bbf2e0b3-d31b-44ca-934c-3710f4262779\") " pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:31:19.062922 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.062524 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-node-log\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.062922 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.062546 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-log-socket\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.062922 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.062565 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b4c0e6ed-9c41-4023-9e84-524635b9f1fb-host-slash\") pod \"iptables-alerter-d7j6l\" (UID: \"b4c0e6ed-9c41-4023-9e84-524635b9f1fb\") " pod="openshift-network-operator/iptables-alerter-d7j6l" Oct 31 12:31:19.062922 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.062585 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j7fd8\" (UID: \"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35\") " pod="openshift-multus/multus-additional-cni-plugins-j7fd8" Oct 31 12:31:19.062922 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.062606 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-multus-cni-dir\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.062922 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.062627 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-host-var-lib-cni-multus\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.062922 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.062708 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-log-socket\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.062922 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.062826 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b4c0e6ed-9c41-4023-9e84-524635b9f1fb-host-slash\") pod \"iptables-alerter-d7j6l\" (UID: \"b4c0e6ed-9c41-4023-9e84-524635b9f1fb\") " pod="openshift-network-operator/iptables-alerter-d7j6l" Oct 31 12:31:19.062922 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.062839 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-node-log\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.062922 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.062877 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/26913ad4-6613-4f3f-9819-9be238088c0a-hosts-file\") pod \"node-resolver-5gdlb\" (UID: \"26913ad4-6613-4f3f-9819-9be238088c0a\") " pod="openshift-dns/node-resolver-5gdlb" Oct 31 12:31:19.062922 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.062880 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-run-ovn\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.063562 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.062900 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf2e0b3-d31b-44ca-934c-3710f4262779-metrics-certs\") pod \"network-metrics-daemon-8nbvz\" (UID: \"bbf2e0b3-d31b-44ca-934c-3710f4262779\") " pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:31:19.063562 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.062916 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b091d91-47bc-4480-85b2-e23a8139df66-ovnkube-script-lib\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.063562 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.062948 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.063562 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.063275 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b091d91-47bc-4480-85b2-e23a8139df66-ovnkube-config\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.063562 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.063278 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26913ad4-6613-4f3f-9819-9be238088c0a-tmp-dir\") pod \"node-resolver-5gdlb\" (UID: \"26913ad4-6613-4f3f-9819-9be238088c0a\") " pod="openshift-dns/node-resolver-5gdlb" Oct 31 12:31:19.063562 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.063310 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/43421968-0e5b-4e27-be79-104aae5f72e0-konnectivity-ca\") pod \"konnectivity-agent-crhzp\" (UID: \"43421968-0e5b-4e27-be79-104aae5f72e0\") " pod="kube-system/konnectivity-agent-crhzp" Oct 31 12:31:19.063562 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.063328 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-cnibin\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.063562 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.063352 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-run-ovn\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.063562 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:19.063395 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:31:19.063562 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.063434 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-host-run-k8s-cni-cncf-io\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.063562 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.063475 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b091d91-47bc-4480-85b2-e23a8139df66-ovnkube-script-lib\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.063562 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.063496 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-cni-binary-copy\") pod \"multus-additional-cni-plugins-j7fd8\" (UID: \"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35\") " pod="openshift-multus/multus-additional-cni-plugins-j7fd8" Oct 31 12:31:19.063562 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:19.063528 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbf2e0b3-d31b-44ca-934c-3710f4262779-metrics-certs podName:bbf2e0b3-d31b-44ca-934c-3710f4262779 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:19.563508048 +0000 UTC m=+2.980212498 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bbf2e0b3-d31b-44ca-934c-3710f4262779-metrics-certs") pod "network-metrics-daemon-8nbvz" (UID: "bbf2e0b3-d31b-44ca-934c-3710f4262779") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:31:19.064105 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.063823 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-multus-conf-dir\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.064105 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.063868 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14dc1e6d-70e1-4ba5-bffb-d9299433756d-host\") pod \"node-ca-tk5vl\" (UID: \"14dc1e6d-70e1-4ba5-bffb-d9299433756d\") " pod="openshift-image-registry/node-ca-tk5vl" Oct 31 12:31:19.064105 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.063904 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-sysctl-d\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.064105 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.063938 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a-kubelet-config\") pod \"global-pull-secret-syncer-wgrhs\" (UID: \"ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a\") " pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:31:19.064105 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.063955 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j7fd8\" (UID: \"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35\") " pod="openshift-multus/multus-additional-cni-plugins-j7fd8" Oct 31 12:31:19.064105 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.063960 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-host-var-lib-kubelet\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.064105 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.063995 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b4c0e6ed-9c41-4023-9e84-524635b9f1fb-iptables-alerter-script\") pod \"iptables-alerter-d7j6l\" (UID: \"b4c0e6ed-9c41-4023-9e84-524635b9f1fb\") " pod="openshift-network-operator/iptables-alerter-d7j6l" Oct 31 12:31:19.064105 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.064003 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-sysconfig\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.064105 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.064011 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a-kubelet-config\") pod \"global-pull-secret-syncer-wgrhs\" (UID: \"ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a\") " pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:31:19.064105 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.064027 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-sys\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.064105 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.064054 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-var-lib-kubelet\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.064105 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.064086 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/24e17633-30a9-4410-b000-2fbb97202951-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:31:19.064586 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.064117 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n79b\" (UniqueName: \"kubernetes.io/projected/0b091d91-47bc-4480-85b2-e23a8139df66-kube-api-access-4n79b\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.064586 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.064153 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a-dbus\") pod \"global-pull-secret-syncer-wgrhs\" (UID: \"ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a\") " pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:31:19.064586 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.064177 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j7fd8\" (UID: \"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35\") " pod="openshift-multus/multus-additional-cni-plugins-j7fd8" Oct 31 12:31:19.064586 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.064202 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd2xl\" (UniqueName: \"kubernetes.io/projected/14dc1e6d-70e1-4ba5-bffb-d9299433756d-kube-api-access-sd2xl\") pod \"node-ca-tk5vl\" (UID: \"14dc1e6d-70e1-4ba5-bffb-d9299433756d\") " pod="openshift-image-registry/node-ca-tk5vl" Oct 31 12:31:19.064586 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.064224 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-sysctl-conf\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.064586 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.064252 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-lib-modules\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.064586 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.064273 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/24e17633-30a9-4410-b000-2fbb97202951-device-dir\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:31:19.064586 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.064311 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a-dbus\") pod \"global-pull-secret-syncer-wgrhs\" (UID: \"ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a\") " pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:31:19.064586 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.064484 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j7fd8\" (UID: \"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35\") " pod="openshift-multus/multus-additional-cni-plugins-j7fd8" Oct 31 12:31:19.066319 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.066299 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b091d91-47bc-4480-85b2-e23a8139df66-ovn-node-metrics-cert\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.066448 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.066429 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/43421968-0e5b-4e27-be79-104aae5f72e0-agent-certs\") pod \"konnectivity-agent-crhzp\" (UID: \"43421968-0e5b-4e27-be79-104aae5f72e0\") " pod="kube-system/konnectivity-agent-crhzp" Oct 31 12:31:19.074613 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.074561 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-779z6\" (UniqueName: \"kubernetes.io/projected/26913ad4-6613-4f3f-9819-9be238088c0a-kube-api-access-779z6\") pod \"node-resolver-5gdlb\" (UID: \"26913ad4-6613-4f3f-9819-9be238088c0a\") " pod="openshift-dns/node-resolver-5gdlb" Oct 31 12:31:19.076312 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.076292 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n79b\" (UniqueName: \"kubernetes.io/projected/0b091d91-47bc-4480-85b2-e23a8139df66-kube-api-access-4n79b\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.076385 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.076359 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2z5c\" (UniqueName: \"kubernetes.io/projected/bbf2e0b3-d31b-44ca-934c-3710f4262779-kube-api-access-j2z5c\") pod \"network-metrics-daemon-8nbvz\" (UID: \"bbf2e0b3-d31b-44ca-934c-3710f4262779\") " pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:31:19.076917 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.076899 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhnsq\" (UniqueName: \"kubernetes.io/projected/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-kube-api-access-rhnsq\") pod \"multus-additional-cni-plugins-j7fd8\" (UID: \"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35\") " pod="openshift-multus/multus-additional-cni-plugins-j7fd8" Oct 31 12:31:19.077474 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.077456 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvd64\" (UniqueName: \"kubernetes.io/projected/b4c0e6ed-9c41-4023-9e84-524635b9f1fb-kube-api-access-tvd64\") pod \"iptables-alerter-d7j6l\" (UID: \"b4c0e6ed-9c41-4023-9e84-524635b9f1fb\") " pod="openshift-network-operator/iptables-alerter-d7j6l" Oct 31 12:31:19.165322 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.165289 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bd53d085-6474-45d6-bd59-825fea180a79-multus-daemon-config\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.165322 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.165322 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgs94\" (UniqueName: \"kubernetes.io/projected/bd53d085-6474-45d6-bd59-825fea180a79-kube-api-access-zgs94\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.165556 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.165345 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-host\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.165556 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.165461 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd53d085-6474-45d6-bd59-825fea180a79-cni-binary-copy\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.165556 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.165488 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d3565a5-d07a-4dc0-abd1-e697573baf22-tmp\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.165556 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.165513 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph9jm\" (UniqueName: \"kubernetes.io/projected/02c6d8f5-97ff-41e0-8e38-71e1b0c87f54-kube-api-access-ph9jm\") pod \"network-check-target-bgcqc\" (UID: \"02c6d8f5-97ff-41e0-8e38-71e1b0c87f54\") " pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:31:19.165556 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.165543 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-host-run-netns\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.165832 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.165578 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-host-var-lib-cni-bin\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.165832 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.165610 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-multus-cni-dir\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.165832 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.165628 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-host-var-lib-cni-multus\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.165832 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.165680 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-cnibin\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.165832 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.165701 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-host-run-k8s-cni-cncf-io\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.165832 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.165723 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-multus-conf-dir\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.165832 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.165746 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14dc1e6d-70e1-4ba5-bffb-d9299433756d-host\") pod \"node-ca-tk5vl\" (UID: \"14dc1e6d-70e1-4ba5-bffb-d9299433756d\") " pod="openshift-image-registry/node-ca-tk5vl" Oct 31 12:31:19.165832 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.165768 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-sysctl-d\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.165832 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.165793 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-host-var-lib-kubelet\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.165832 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.165822 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-sysconfig\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.166275 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.165852 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-sys\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.166275 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.165876 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-var-lib-kubelet\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.166275 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.165923 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/24e17633-30a9-4410-b000-2fbb97202951-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:31:19.166275 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.165960 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd2xl\" (UniqueName: \"kubernetes.io/projected/14dc1e6d-70e1-4ba5-bffb-d9299433756d-kube-api-access-sd2xl\") pod \"node-ca-tk5vl\" (UID: \"14dc1e6d-70e1-4ba5-bffb-d9299433756d\") " pod="openshift-image-registry/node-ca-tk5vl" Oct 31 12:31:19.166275 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.165983 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-sysctl-conf\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.166275 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.165998 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bd53d085-6474-45d6-bd59-825fea180a79-multus-daemon-config\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.166275 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.166005 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-lib-modules\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.166275 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.166066 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/24e17633-30a9-4410-b000-2fbb97202951-device-dir\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:31:19.166275 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.166095 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/24e17633-30a9-4410-b000-2fbb97202951-sys-fs\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:31:19.166275 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.166123 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-run\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.166275 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.166150 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-tuned\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.166275 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.166161 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-lib-modules\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.166275 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.166211 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd53d085-6474-45d6-bd59-825fea180a79-cni-binary-copy\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.166275 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.166209 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-system-cni-dir\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.166275 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.166256 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-etc-kubernetes\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.166956 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.166289 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-systemd\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.166956 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.166296 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-host\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.166956 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.166368 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5hf6\" (UniqueName: \"kubernetes.io/projected/5d3565a5-d07a-4dc0-abd1-e697573baf22-kube-api-access-h5hf6\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.166956 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.166392 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14dc1e6d-70e1-4ba5-bffb-d9299433756d-host\") pod \"node-ca-tk5vl\" (UID: \"14dc1e6d-70e1-4ba5-bffb-d9299433756d\") " pod="openshift-image-registry/node-ca-tk5vl" Oct 31 12:31:19.166956 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.166520 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-sysctl-d\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.166956 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.166562 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-host-var-lib-cni-multus\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.166956 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.166833 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-host-run-netns\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.166956 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.166878 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-host-var-lib-cni-bin\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.166956 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.166934 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-multus-cni-dir\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.167395 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.166973 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/24e17633-30a9-4410-b000-2fbb97202951-sys-fs\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:31:19.167395 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.167023 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-host-var-lib-kubelet\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.167395 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.167028 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-sys\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.167395 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.166978 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/24e17633-30a9-4410-b000-2fbb97202951-device-dir\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:31:19.167395 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.167066 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-run\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.167395 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.167111 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-sysconfig\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.167395 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.167124 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-var-lib-kubelet\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.167395 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.167139 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-host-run-k8s-cni-cncf-io\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.167395 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.167162 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/24e17633-30a9-4410-b000-2fbb97202951-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:31:19.167395 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.167184 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-cnibin\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.167395 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.167192 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-multus-conf-dir\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.167395 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.166395 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/14dc1e6d-70e1-4ba5-bffb-d9299433756d-serviceca\") pod \"node-ca-tk5vl\" (UID: \"14dc1e6d-70e1-4ba5-bffb-d9299433756d\") " pod="openshift-image-registry/node-ca-tk5vl" Oct 31 12:31:19.167395 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.167247 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-system-cni-dir\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.167395 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.167283 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-modprobe-d\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.167395 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.167312 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-sysctl-conf\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.167395 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.167321 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/24e17633-30a9-4410-b000-2fbb97202951-etc-selinux\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:31:19.167395 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.167366 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-modprobe-d\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.167395 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.167210 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-systemd\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.168230 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.167366 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-host-run-multus-certs\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.168230 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.167406 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/24e17633-30a9-4410-b000-2fbb97202951-etc-selinux\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:31:19.168230 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.167407 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-host-run-multus-certs\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.168230 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.167421 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-etc-kubernetes\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.168230 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.167432 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-kubernetes\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.168230 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.167457 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/24e17633-30a9-4410-b000-2fbb97202951-socket-dir\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:31:19.168230 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.167485 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/24e17633-30a9-4410-b000-2fbb97202951-registration-dir\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:31:19.168230 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.167518 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-kubernetes\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.168230 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.167526 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhppb\" (UniqueName: \"kubernetes.io/projected/24e17633-30a9-4410-b000-2fbb97202951-kube-api-access-zhppb\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:31:19.168230 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.167559 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-multus-socket-dir-parent\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.168230 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.167567 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/24e17633-30a9-4410-b000-2fbb97202951-registration-dir\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:31:19.168230 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.167583 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-hostroot\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.168230 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.167691 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-multus-socket-dir-parent\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.168230 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.167708 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-hostroot\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.168230 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.167840 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/14dc1e6d-70e1-4ba5-bffb-d9299433756d-serviceca\") pod \"node-ca-tk5vl\" (UID: \"14dc1e6d-70e1-4ba5-bffb-d9299433756d\") " pod="openshift-image-registry/node-ca-tk5vl" Oct 31 12:31:19.168230 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.167929 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/24e17633-30a9-4410-b000-2fbb97202951-socket-dir\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:31:19.169249 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.169228 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d3565a5-d07a-4dc0-abd1-e697573baf22-tmp\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.170914 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.170891 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-tuned\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.178217 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:19.178195 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 12:31:19.178335 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:19.178220 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 12:31:19.178335 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:19.178234 2577 projected.go:196] Error preparing data for projected volume kube-api-access-ph9jm for pod openshift-network-diagnostics/network-check-target-bgcqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:31:19.178335 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:19.178306 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02c6d8f5-97ff-41e0-8e38-71e1b0c87f54-kube-api-access-ph9jm podName:02c6d8f5-97ff-41e0-8e38-71e1b0c87f54 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:19.678286777 +0000 UTC m=+3.094991233 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ph9jm" (UniqueName: "kubernetes.io/projected/02c6d8f5-97ff-41e0-8e38-71e1b0c87f54-kube-api-access-ph9jm") pod "network-check-target-bgcqc" (UID: "02c6d8f5-97ff-41e0-8e38-71e1b0c87f54") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:31:19.180909 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.180884 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgs94\" (UniqueName: \"kubernetes.io/projected/bd53d085-6474-45d6-bd59-825fea180a79-kube-api-access-zgs94\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.181013 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.180954 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5hf6\" (UniqueName: \"kubernetes.io/projected/5d3565a5-d07a-4dc0-abd1-e697573baf22-kube-api-access-h5hf6\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.181343 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.181326 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhppb\" (UniqueName: \"kubernetes.io/projected/24e17633-30a9-4410-b000-2fbb97202951-kube-api-access-zhppb\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:31:19.181568 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.181548 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd2xl\" (UniqueName: \"kubernetes.io/projected/14dc1e6d-70e1-4ba5-bffb-d9299433756d-kube-api-access-sd2xl\") pod \"node-ca-tk5vl\" (UID: \"14dc1e6d-70e1-4ba5-bffb-d9299433756d\") " pod="openshift-image-registry/node-ca-tk5vl" Oct 31 12:31:19.246777 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.246738 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-d7j6l" Oct 31 12:31:19.255292 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.255265 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-crhzp" Oct 31 12:31:19.258473 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:19.258445 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4c0e6ed_9c41_4023_9e84_524635b9f1fb.slice/crio-2a2f94ca992e3771f5713cbd3ada4bc4cab041afa0a1eb45530fe3ee42469c15 WatchSource:0}: Error finding container 2a2f94ca992e3771f5713cbd3ada4bc4cab041afa0a1eb45530fe3ee42469c15: Status 404 returned error can't find the container with id 2a2f94ca992e3771f5713cbd3ada4bc4cab041afa0a1eb45530fe3ee42469c15 Oct 31 12:31:19.264478 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:19.264454 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43421968_0e5b_4e27_be79_104aae5f72e0.slice/crio-3504f1ff87a9423da711361f7da6db57c5bd88c69326687759db34467ddcfb1b WatchSource:0}: Error finding container 3504f1ff87a9423da711361f7da6db57c5bd88c69326687759db34467ddcfb1b: Status 404 returned error can't find the container with id 3504f1ff87a9423da711361f7da6db57c5bd88c69326687759db34467ddcfb1b Oct 31 12:31:19.286924 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.286733 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5gdlb" Oct 31 12:31:19.289071 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.289048 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:19.291429 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.291405 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dklfq" Oct 31 12:31:19.296720 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:19.296698 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26913ad4_6613_4f3f_9819_9be238088c0a.slice/crio-3b8f659ef7a491710aa5a36a9d4f2c3b2469a212d3bb308827483dda8be8b556 WatchSource:0}: Error finding container 3b8f659ef7a491710aa5a36a9d4f2c3b2469a212d3bb308827483dda8be8b556: Status 404 returned error can't find the container with id 3b8f659ef7a491710aa5a36a9d4f2c3b2469a212d3bb308827483dda8be8b556 Oct 31 12:31:19.299608 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:19.299581 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b091d91_47bc_4480_85b2_e23a8139df66.slice/crio-d80f43156c9865079efe4848eefacddbdee81a7f18ec7222d77c5723944c269a WatchSource:0}: Error finding container d80f43156c9865079efe4848eefacddbdee81a7f18ec7222d77c5723944c269a: Status 404 returned error can't find the container with id d80f43156c9865079efe4848eefacddbdee81a7f18ec7222d77c5723944c269a Oct 31 12:31:19.300601 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.300583 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j7fd8" Oct 31 12:31:19.303859 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:19.303026 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd53d085_6474_45d6_bd59_825fea180a79.slice/crio-a32690afe6d52b867aceffb58e48753674527d54678cbb3cdbb2b441b7c99e8c WatchSource:0}: Error finding container a32690afe6d52b867aceffb58e48753674527d54678cbb3cdbb2b441b7c99e8c: Status 404 returned error can't find the container with id a32690afe6d52b867aceffb58e48753674527d54678cbb3cdbb2b441b7c99e8c Oct 31 12:31:19.303859 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.303115 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tk5vl" Oct 31 12:31:19.309555 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.309531 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:31:19.311934 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:19.311685 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c75b6d0_3fc9_4e2e_aba2_f464a48cab35.slice/crio-2371284ce749382e13be426c9d83dd544dfdcc771669afc0a19abb73b59a2845 WatchSource:0}: Error finding container 2371284ce749382e13be426c9d83dd544dfdcc771669afc0a19abb73b59a2845: Status 404 returned error can't find the container with id 2371284ce749382e13be426c9d83dd544dfdcc771669afc0a19abb73b59a2845 Oct 31 12:31:19.314243 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:19.314216 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14dc1e6d_70e1_4ba5_bffb_d9299433756d.slice/crio-af8a3287d760c49bab4cd6f000a6ccf86a22d13d3c19f124b2e2654ed0084904 WatchSource:0}: Error finding container af8a3287d760c49bab4cd6f000a6ccf86a22d13d3c19f124b2e2654ed0084904: Status 404 returned error can't find the container with id af8a3287d760c49bab4cd6f000a6ccf86a22d13d3c19f124b2e2654ed0084904 Oct 31 12:31:19.318218 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.318073 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:31:19.322342 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:19.322292 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d3565a5_d07a_4dc0_abd1_e697573baf22.slice/crio-255a03156c1e7f8f432700c836775758fc904bf00fb80ebd67d8800ec97c3a71 WatchSource:0}: Error finding container 255a03156c1e7f8f432700c836775758fc904bf00fb80ebd67d8800ec97c3a71: Status 404 returned error can't find the container with id 255a03156c1e7f8f432700c836775758fc904bf00fb80ebd67d8800ec97c3a71 Oct 31 12:31:19.332695 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:31:19.332670 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24e17633_30a9_4410_b000_2fbb97202951.slice/crio-da22f1b02aab199a2fd5d6f85284e273cd2aed5d3d2946debb29989dc43335d2 WatchSource:0}: Error finding container da22f1b02aab199a2fd5d6f85284e273cd2aed5d3d2946debb29989dc43335d2: Status 404 returned error can't find the container with id da22f1b02aab199a2fd5d6f85284e273cd2aed5d3d2946debb29989dc43335d2 Oct 31 12:31:19.475340 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.475255 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:31:19.571022 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.570983 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf2e0b3-d31b-44ca-934c-3710f4262779-metrics-certs\") pod \"network-metrics-daemon-8nbvz\" (UID: \"bbf2e0b3-d31b-44ca-934c-3710f4262779\") " pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:31:19.571200 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.571043 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a-original-pull-secret\") pod \"global-pull-secret-syncer-wgrhs\" (UID: \"ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a\") " pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:31:19.571265 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:19.571210 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 12:31:19.571317 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:19.571273 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a-original-pull-secret podName:ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a nodeName:}" failed. No retries permitted until 2026-10-31 12:31:20.571255415 +0000 UTC m=+3.987959872 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a-original-pull-secret") pod "global-pull-secret-syncer-wgrhs" (UID: "ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a") : object "kube-system"/"original-pull-secret" not registered Oct 31 12:31:19.571689 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:19.571671 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:31:19.571780 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:19.571725 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbf2e0b3-d31b-44ca-934c-3710f4262779-metrics-certs podName:bbf2e0b3-d31b-44ca-934c-3710f4262779 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:20.571710496 +0000 UTC m=+3.988414952 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bbf2e0b3-d31b-44ca-934c-3710f4262779-metrics-certs") pod "network-metrics-daemon-8nbvz" (UID: "bbf2e0b3-d31b-44ca-934c-3710f4262779") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:31:19.773547 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.773454 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph9jm\" (UniqueName: \"kubernetes.io/projected/02c6d8f5-97ff-41e0-8e38-71e1b0c87f54-kube-api-access-ph9jm\") pod \"network-check-target-bgcqc\" (UID: \"02c6d8f5-97ff-41e0-8e38-71e1b0c87f54\") " pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:31:19.773704 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:19.773669 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 12:31:19.773704 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:19.773693 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 12:31:19.773785 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:19.773705 2577 projected.go:196] Error preparing data for projected volume kube-api-access-ph9jm for pod openshift-network-diagnostics/network-check-target-bgcqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:31:19.773785 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:19.773765 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02c6d8f5-97ff-41e0-8e38-71e1b0c87f54-kube-api-access-ph9jm podName:02c6d8f5-97ff-41e0-8e38-71e1b0c87f54 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:20.773745917 +0000 UTC m=+4.190450376 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ph9jm" (UniqueName: "kubernetes.io/projected/02c6d8f5-97ff-41e0-8e38-71e1b0c87f54-kube-api-access-ph9jm") pod "network-check-target-bgcqc" (UID: "02c6d8f5-97ff-41e0-8e38-71e1b0c87f54") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:31:19.996515 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.996442 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 12:26:18 +0000 UTC" deadline="2028-06-01 17:37:09.63343016 +0000 UTC" Oct 31 12:31:19.996515 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:19.996511 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13901h5m49.636922699s" Oct 31 12:31:20.024036 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:20.023958 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:31:20.024184 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:20.024090 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgcqc" podUID="02c6d8f5-97ff-41e0-8e38-71e1b0c87f54" Oct 31 12:31:20.024518 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:20.024497 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:31:20.024667 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:20.024626 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgrhs" podUID="ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a" Oct 31 12:31:20.035561 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:20.035483 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" event={"ID":"5d3565a5-d07a-4dc0-abd1-e697573baf22","Type":"ContainerStarted","Data":"255a03156c1e7f8f432700c836775758fc904bf00fb80ebd67d8800ec97c3a71"} Oct 31 12:31:20.038808 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:20.038751 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tk5vl" event={"ID":"14dc1e6d-70e1-4ba5-bffb-d9299433756d","Type":"ContainerStarted","Data":"af8a3287d760c49bab4cd6f000a6ccf86a22d13d3c19f124b2e2654ed0084904"} Oct 31 12:31:20.042027 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:20.041961 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j7fd8" event={"ID":"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35","Type":"ContainerStarted","Data":"2371284ce749382e13be426c9d83dd544dfdcc771669afc0a19abb73b59a2845"} Oct 31 12:31:20.050878 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:20.050850 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" event={"ID":"0b091d91-47bc-4480-85b2-e23a8139df66","Type":"ContainerStarted","Data":"d80f43156c9865079efe4848eefacddbdee81a7f18ec7222d77c5723944c269a"} Oct 31 12:31:20.054722 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:20.054695 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5gdlb" event={"ID":"26913ad4-6613-4f3f-9819-9be238088c0a","Type":"ContainerStarted","Data":"3b8f659ef7a491710aa5a36a9d4f2c3b2469a212d3bb308827483dda8be8b556"} Oct 31 12:31:20.057703 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:20.057679 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-crhzp" event={"ID":"43421968-0e5b-4e27-be79-104aae5f72e0","Type":"ContainerStarted","Data":"3504f1ff87a9423da711361f7da6db57c5bd88c69326687759db34467ddcfb1b"} Oct 31 12:31:20.060384 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:20.059898 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" event={"ID":"24e17633-30a9-4410-b000-2fbb97202951","Type":"ContainerStarted","Data":"da22f1b02aab199a2fd5d6f85284e273cd2aed5d3d2946debb29989dc43335d2"} Oct 31 12:31:20.064578 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:20.064553 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dklfq" event={"ID":"bd53d085-6474-45d6-bd59-825fea180a79","Type":"ContainerStarted","Data":"a32690afe6d52b867aceffb58e48753674527d54678cbb3cdbb2b441b7c99e8c"} Oct 31 12:31:20.069520 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:20.069492 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-d7j6l" event={"ID":"b4c0e6ed-9c41-4023-9e84-524635b9f1fb","Type":"ContainerStarted","Data":"2a2f94ca992e3771f5713cbd3ada4bc4cab041afa0a1eb45530fe3ee42469c15"} Oct 31 12:31:20.580147 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:20.579452 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf2e0b3-d31b-44ca-934c-3710f4262779-metrics-certs\") pod \"network-metrics-daemon-8nbvz\" (UID: \"bbf2e0b3-d31b-44ca-934c-3710f4262779\") " pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:31:20.580147 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:20.579733 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a-original-pull-secret\") pod \"global-pull-secret-syncer-wgrhs\" (UID: \"ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a\") " pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:31:20.580147 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:20.579604 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:31:20.580147 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:20.579939 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbf2e0b3-d31b-44ca-934c-3710f4262779-metrics-certs podName:bbf2e0b3-d31b-44ca-934c-3710f4262779 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:22.57992166 +0000 UTC m=+5.996626116 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bbf2e0b3-d31b-44ca-934c-3710f4262779-metrics-certs") pod "network-metrics-daemon-8nbvz" (UID: "bbf2e0b3-d31b-44ca-934c-3710f4262779") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:31:20.580147 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:20.579883 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 12:31:20.580147 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:20.579971 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a-original-pull-secret podName:ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a nodeName:}" failed. No retries permitted until 2026-10-31 12:31:22.579965367 +0000 UTC m=+5.996669804 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a-original-pull-secret") pod "global-pull-secret-syncer-wgrhs" (UID: "ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a") : object "kube-system"/"original-pull-secret" not registered Oct 31 12:31:20.781162 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:20.781133 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph9jm\" (UniqueName: \"kubernetes.io/projected/02c6d8f5-97ff-41e0-8e38-71e1b0c87f54-kube-api-access-ph9jm\") pod \"network-check-target-bgcqc\" (UID: \"02c6d8f5-97ff-41e0-8e38-71e1b0c87f54\") " pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:31:20.781351 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:20.781337 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 12:31:20.781413 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:20.781358 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 12:31:20.781413 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:20.781372 2577 projected.go:196] Error preparing data for projected volume kube-api-access-ph9jm for pod openshift-network-diagnostics/network-check-target-bgcqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:31:20.781503 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:20.781426 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02c6d8f5-97ff-41e0-8e38-71e1b0c87f54-kube-api-access-ph9jm podName:02c6d8f5-97ff-41e0-8e38-71e1b0c87f54 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:22.781409568 +0000 UTC m=+6.198114010 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ph9jm" (UniqueName: "kubernetes.io/projected/02c6d8f5-97ff-41e0-8e38-71e1b0c87f54-kube-api-access-ph9jm") pod "network-check-target-bgcqc" (UID: "02c6d8f5-97ff-41e0-8e38-71e1b0c87f54") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:31:20.997672 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:20.997595 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 12:26:18 +0000 UTC" deadline="2028-05-04 03:06:42.024032122 +0000 UTC" Oct 31 12:31:20.997672 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:20.997674 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13214h35m21.026365073s" Oct 31 12:31:21.023561 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:21.023530 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:31:21.023867 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:21.023691 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nbvz" podUID="bbf2e0b3-d31b-44ca-934c-3710f4262779" Oct 31 12:31:21.097417 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:21.096919 2577 generic.go:356] "Generic (PLEG): container finished" podID="8a68def89a8e2f4b1d88fecf85bfdb5e" containerID="50f6ee2338bf62416e92e9e2bfe9d4c8bdd96dc04ed17c57a22d6fb2fdfad98f" exitCode=0 Oct 31 12:31:21.097417 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:21.097036 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" event={"ID":"8a68def89a8e2f4b1d88fecf85bfdb5e","Type":"ContainerDied","Data":"50f6ee2338bf62416e92e9e2bfe9d4c8bdd96dc04ed17c57a22d6fb2fdfad98f"} Oct 31 12:31:21.107554 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:21.107519 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal" event={"ID":"72de75e4a93b82a0133be1d6ed9e0a51","Type":"ContainerStarted","Data":"eae2685bf8a0bc14141e2122daca3e98f4a2d00dcf5a141c15c4d232cbb482f8"} Oct 31 12:31:21.133330 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:21.133268 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal" podStartSLOduration=3.133248313 podStartE2EDuration="3.133248313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:21.133026664 +0000 UTC m=+4.549731124" watchObservedRunningTime="2026-10-31 12:31:21.133248313 +0000 UTC m=+4.549952786" Oct 31 12:31:22.021682 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:22.021650 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:31:22.022326 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:22.021775 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgcqc" podUID="02c6d8f5-97ff-41e0-8e38-71e1b0c87f54" Oct 31 12:31:22.022326 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:22.022167 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:31:22.022326 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:22.022257 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgrhs" podUID="ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a" Oct 31 12:31:22.120122 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:22.120032 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" event={"ID":"8a68def89a8e2f4b1d88fecf85bfdb5e","Type":"ContainerStarted","Data":"d295a68fcf3ba939e56428b90189cdde34a8f545d27bec40113d5649d0189664"} Oct 31 12:31:22.139110 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:22.139061 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" podStartSLOduration=4.13904916 podStartE2EDuration="4.13904916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:22.138853055 +0000 UTC m=+5.555557515" watchObservedRunningTime="2026-10-31 12:31:22.13904916 +0000 UTC m=+5.555753619" Oct 31 12:31:22.597164 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:22.597082 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a-original-pull-secret\") pod \"global-pull-secret-syncer-wgrhs\" (UID: \"ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a\") " pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:31:22.597331 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:22.597180 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf2e0b3-d31b-44ca-934c-3710f4262779-metrics-certs\") pod \"network-metrics-daemon-8nbvz\" (UID: \"bbf2e0b3-d31b-44ca-934c-3710f4262779\") " pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:31:22.597331 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:22.597292 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:31:22.597331 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:22.597292 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 12:31:22.597482 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:22.597352 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a-original-pull-secret podName:ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a nodeName:}" failed. No retries permitted until 2026-10-31 12:31:26.597336376 +0000 UTC m=+10.014040814 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a-original-pull-secret") pod "global-pull-secret-syncer-wgrhs" (UID: "ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a") : object "kube-system"/"original-pull-secret" not registered Oct 31 12:31:22.597482 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:22.597373 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbf2e0b3-d31b-44ca-934c-3710f4262779-metrics-certs podName:bbf2e0b3-d31b-44ca-934c-3710f4262779 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:26.597363451 +0000 UTC m=+10.014067889 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bbf2e0b3-d31b-44ca-934c-3710f4262779-metrics-certs") pod "network-metrics-daemon-8nbvz" (UID: "bbf2e0b3-d31b-44ca-934c-3710f4262779") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:31:22.801784 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:22.798969 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph9jm\" (UniqueName: \"kubernetes.io/projected/02c6d8f5-97ff-41e0-8e38-71e1b0c87f54-kube-api-access-ph9jm\") pod \"network-check-target-bgcqc\" (UID: \"02c6d8f5-97ff-41e0-8e38-71e1b0c87f54\") " pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:31:22.801784 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:22.799157 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 12:31:22.801784 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:22.799178 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 12:31:22.801784 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:22.799194 2577 projected.go:196] Error preparing data for projected volume kube-api-access-ph9jm for pod openshift-network-diagnostics/network-check-target-bgcqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:31:22.801784 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:22.799272 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02c6d8f5-97ff-41e0-8e38-71e1b0c87f54-kube-api-access-ph9jm podName:02c6d8f5-97ff-41e0-8e38-71e1b0c87f54 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:26.799254192 +0000 UTC m=+10.215958645 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ph9jm" (UniqueName: "kubernetes.io/projected/02c6d8f5-97ff-41e0-8e38-71e1b0c87f54-kube-api-access-ph9jm") pod "network-check-target-bgcqc" (UID: "02c6d8f5-97ff-41e0-8e38-71e1b0c87f54") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:31:23.022189 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:23.022155 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:31:23.022630 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:23.022312 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nbvz" podUID="bbf2e0b3-d31b-44ca-934c-3710f4262779" Oct 31 12:31:23.125143 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:23.125104 2577 generic.go:356] "Generic (PLEG): container finished" podID="8c75b6d0-3fc9-4e2e-aba2-f464a48cab35" containerID="39b31960d2335ff6536593a02304d9fb9762569b0418f0c120601d3fff86cae1" exitCode=0 Oct 31 12:31:23.126241 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:23.126215 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j7fd8" event={"ID":"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35","Type":"ContainerDied","Data":"39b31960d2335ff6536593a02304d9fb9762569b0418f0c120601d3fff86cae1"} Oct 31 12:31:24.020982 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:24.020947 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:31:24.021159 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:24.021068 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgcqc" podUID="02c6d8f5-97ff-41e0-8e38-71e1b0c87f54" Oct 31 12:31:24.021434 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:24.021411 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:31:24.021542 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:24.021521 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgrhs" podUID="ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a" Oct 31 12:31:24.134925 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:24.134231 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-crhzp" event={"ID":"43421968-0e5b-4e27-be79-104aae5f72e0","Type":"ContainerStarted","Data":"5de0e6807f76c30060e6f9c4d3049c4c5f3737514e9d5de6747a9498eedd9161"} Oct 31 12:31:25.020880 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:25.020851 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:31:25.021004 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:25.020984 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nbvz" podUID="bbf2e0b3-d31b-44ca-934c-3710f4262779" Oct 31 12:31:25.143131 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:25.143045 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tk5vl" event={"ID":"14dc1e6d-70e1-4ba5-bffb-d9299433756d","Type":"ContainerStarted","Data":"ee99b2bc9fed80defb6d2fc6c4a5b88d72247764030dec39537dfb75b9d25f28"} Oct 31 12:31:25.161033 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:25.160976 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-crhzp" podStartSLOduration=4.179784432 podStartE2EDuration="8.160957113s" totalImagesPullingTime="3.981172681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:31:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:24.152676012 +0000 UTC m=+7.569380470" watchObservedRunningTime="2026-10-31 12:31:25.160957113 +0000 UTC m=+8.577661574" Oct 31 12:31:26.021611 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:26.021536 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:31:26.021781 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:26.021689 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgcqc" podUID="02c6d8f5-97ff-41e0-8e38-71e1b0c87f54" Oct 31 12:31:26.022146 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:26.022123 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:31:26.022248 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:26.022227 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgrhs" podUID="ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a" Oct 31 12:31:26.148946 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:26.148909 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" event={"ID":"24e17633-30a9-4410-b000-2fbb97202951","Type":"ContainerStarted","Data":"b941ca88dd670ff140be1e58b4b3f290292216ac9b4c5d87b0e5f7f739bc9831"} Oct 31 12:31:26.278219 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:26.278153 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-crhzp" Oct 31 12:31:26.634008 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:26.633545 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf2e0b3-d31b-44ca-934c-3710f4262779-metrics-certs\") pod \"network-metrics-daemon-8nbvz\" (UID: \"bbf2e0b3-d31b-44ca-934c-3710f4262779\") " pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:31:26.634008 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:26.633610 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a-original-pull-secret\") pod \"global-pull-secret-syncer-wgrhs\" (UID: \"ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a\") " pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:31:26.634008 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:26.633783 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 12:31:26.634008 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:26.633835 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a-original-pull-secret podName:ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a nodeName:}" failed. No retries permitted until 2026-10-31 12:31:34.633821198 +0000 UTC m=+18.050525649 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a-original-pull-secret") pod "global-pull-secret-syncer-wgrhs" (UID: "ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a") : object "kube-system"/"original-pull-secret" not registered Oct 31 12:31:26.634314 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:26.634125 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:31:26.634314 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:26.634158 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbf2e0b3-d31b-44ca-934c-3710f4262779-metrics-certs podName:bbf2e0b3-d31b-44ca-934c-3710f4262779 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:34.634146574 +0000 UTC m=+18.050851024 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bbf2e0b3-d31b-44ca-934c-3710f4262779-metrics-certs") pod "network-metrics-daemon-8nbvz" (UID: "bbf2e0b3-d31b-44ca-934c-3710f4262779") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:31:26.816621 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:26.816593 2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 12:31:26.835123 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:26.835094 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph9jm\" (UniqueName: \"kubernetes.io/projected/02c6d8f5-97ff-41e0-8e38-71e1b0c87f54-kube-api-access-ph9jm\") pod \"network-check-target-bgcqc\" (UID: \"02c6d8f5-97ff-41e0-8e38-71e1b0c87f54\") " pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:31:26.835261 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:26.835228 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 12:31:26.835261 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:26.835248 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 12:31:26.835261 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:26.835259 2577 projected.go:196] Error preparing data for projected volume kube-api-access-ph9jm for pod openshift-network-diagnostics/network-check-target-bgcqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:31:26.835427 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:26.835352 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02c6d8f5-97ff-41e0-8e38-71e1b0c87f54-kube-api-access-ph9jm podName:02c6d8f5-97ff-41e0-8e38-71e1b0c87f54 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:34.835336253 +0000 UTC m=+18.252040690 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ph9jm" (UniqueName: "kubernetes.io/projected/02c6d8f5-97ff-41e0-8e38-71e1b0c87f54-kube-api-access-ph9jm") pod "network-check-target-bgcqc" (UID: "02c6d8f5-97ff-41e0-8e38-71e1b0c87f54") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:31:26.955436 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:26.955410 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-crhzp" Oct 31 12:31:26.956187 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:26.956163 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-crhzp" Oct 31 12:31:26.973293 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:26.973238 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tk5vl" podStartSLOduration=5.071847852 podStartE2EDuration="9.973220486s" totalImagesPullingTime="4.901372634s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:31:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:25.162166769 +0000 UTC m=+8.578871228" watchObservedRunningTime="2026-10-31 12:31:26.973220486 +0000 UTC m=+10.389924947" Oct 31 12:31:27.022380 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:27.022349 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:31:27.022509 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:27.022472 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nbvz" podUID="bbf2e0b3-d31b-44ca-934c-3710f4262779" Oct 31 12:31:27.027057 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:27.026509 2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T12:31:26.816616564Z","UUID":"a376b7c0-6508-41a4-96ff-d13b27cb6a65","Handler":null,"Name":"","Endpoint":""} Oct 31 12:31:27.028162 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:27.028126 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 12:31:27.028162 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:27.028157 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 12:31:27.151690 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:27.151602 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5gdlb" event={"ID":"26913ad4-6613-4f3f-9819-9be238088c0a","Type":"ContainerStarted","Data":"f0b551ea77b30df8c67e543f25a44cef1bcf0c8a8372f540781a492e1a07fff5"} Oct 31 12:31:27.154232 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:27.154201 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" event={"ID":"24e17633-30a9-4410-b000-2fbb97202951","Type":"ContainerStarted","Data":"8c30f3dc9439b8516509a972e345875fa8ac0d6ff190959d6a500c53a059fa3b"} Oct 31 12:31:27.155989 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:27.155963 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dklfq" event={"ID":"bd53d085-6474-45d6-bd59-825fea180a79","Type":"ContainerStarted","Data":"3db32464b9292ee2b7fa0c2a0d19716120ee45a51f754061e28d4da9ea816efb"} Oct 31 12:31:27.156603 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:27.156585 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-crhzp" Oct 31 12:31:27.171809 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:27.171762 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-5gdlb" podStartSLOduration=2.482668614 podStartE2EDuration="10.17174382s" totalImagesPullingTime="7.689075206s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:31:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:27.171303202 +0000 UTC m=+10.588007663" watchObservedRunningTime="2026-10-31 12:31:27.17174382 +0000 UTC m=+10.588448274" Oct 31 12:31:27.211805 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:27.211755 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-dklfq" podStartSLOduration=3.12612955 podStartE2EDuration="10.211739998s" totalImagesPullingTime="7.085610448s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:31:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:27.211344707 +0000 UTC m=+10.628049167" watchObservedRunningTime="2026-10-31 12:31:27.211739998 +0000 UTC m=+10.628444451" Oct 31 12:31:28.021824 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:28.021568 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:31:28.021962 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:28.021925 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgcqc" podUID="02c6d8f5-97ff-41e0-8e38-71e1b0c87f54" Oct 31 12:31:28.021962 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:28.021591 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:31:28.022055 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:28.022024 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgrhs" podUID="ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a" Oct 31 12:31:28.161053 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:28.161015 2577 generic.go:356] "Generic (PLEG): container finished" podID="8c75b6d0-3fc9-4e2e-aba2-f464a48cab35" containerID="dc9ba54ab45f379e93ca2b9ef1c767af2ee2e6e9b08039345a6fbb992b80ff00" exitCode=0 Oct 31 12:31:28.161476 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:28.161057 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j7fd8" event={"ID":"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35","Type":"ContainerDied","Data":"dc9ba54ab45f379e93ca2b9ef1c767af2ee2e6e9b08039345a6fbb992b80ff00"} Oct 31 12:31:28.164557 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:28.164078 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" event={"ID":"24e17633-30a9-4410-b000-2fbb97202951","Type":"ContainerStarted","Data":"025ea30685414871ad37daaea1dc2a472395814003407ec8ac5e91333c99a23e"} Oct 31 12:31:28.210322 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:28.210271 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" podStartSLOduration=3.007890616 podStartE2EDuration="11.210256172s" totalImagesPullingTime="8.202365556s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:31:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:28.209835165 +0000 UTC m=+11.626539624" watchObservedRunningTime="2026-10-31 12:31:28.210256172 +0000 UTC m=+11.626960631" Oct 31 12:31:29.021344 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:29.021315 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:31:29.021484 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:29.021460 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nbvz" podUID="bbf2e0b3-d31b-44ca-934c-3710f4262779" Oct 31 12:31:29.169317 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:29.169281 2577 generic.go:356] "Generic (PLEG): container finished" podID="8c75b6d0-3fc9-4e2e-aba2-f464a48cab35" containerID="9aa4f45abe694c91bf2614b64265805783a11ca8d562997fb51744b32f27ae94" exitCode=0 Oct 31 12:31:29.169784 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:29.169373 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j7fd8" event={"ID":"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35","Type":"ContainerDied","Data":"9aa4f45abe694c91bf2614b64265805783a11ca8d562997fb51744b32f27ae94"} Oct 31 12:31:29.172313 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:29.171992 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-d7j6l" event={"ID":"b4c0e6ed-9c41-4023-9e84-524635b9f1fb","Type":"ContainerStarted","Data":"d96702e303f893b6eb6b91126447ac263f171688245a051ead7bd1e92648f1a5"} Oct 31 12:31:29.215437 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:29.214916 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-d7j6l" podStartSLOduration=4.61533748 podStartE2EDuration="12.214900267s" totalImagesPullingTime="7.599562787s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:31:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:29.214427474 +0000 UTC m=+12.631131933" watchObservedRunningTime="2026-10-31 12:31:29.214900267 +0000 UTC m=+12.631604726" Oct 31 12:31:30.021022 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:30.020991 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:31:30.021159 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:30.021003 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:31:30.021159 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:30.021086 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgcqc" podUID="02c6d8f5-97ff-41e0-8e38-71e1b0c87f54" Oct 31 12:31:30.021238 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:30.021174 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgrhs" podUID="ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a" Oct 31 12:31:30.175677 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:30.175629 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" event={"ID":"5d3565a5-d07a-4dc0-abd1-e697573baf22","Type":"ContainerStarted","Data":"99229c4c01f0024d4678a8f808fee22c0ddce06508007cb1270d752f04617525"} Oct 31 12:31:30.195201 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:30.195158 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" podStartSLOduration=3.058622 podStartE2EDuration="13.195144704s" totalImagesPullingTime="10.136522704s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:31:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:30.194876229 +0000 UTC m=+13.611580692" watchObservedRunningTime="2026-10-31 12:31:30.195144704 +0000 UTC m=+13.611849163" Oct 31 12:31:31.021496 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:31.021467 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:31:31.021703 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:31.021677 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nbvz" podUID="bbf2e0b3-d31b-44ca-934c-3710f4262779" Oct 31 12:31:31.180243 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:31.180203 2577 generic.go:356] "Generic (PLEG): container finished" podID="8c75b6d0-3fc9-4e2e-aba2-f464a48cab35" containerID="da49b8aafa449ef6ed4ba8a2af64cc32422fbec861ce60e20b8feb07b6cb4993" exitCode=0 Oct 31 12:31:31.180885 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:31.180276 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j7fd8" event={"ID":"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35","Type":"ContainerDied","Data":"da49b8aafa449ef6ed4ba8a2af64cc32422fbec861ce60e20b8feb07b6cb4993"} Oct 31 12:31:32.021600 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:32.021405 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:31:32.021807 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:32.021466 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:31:32.021807 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:32.021714 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgrhs" podUID="ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a" Oct 31 12:31:32.021807 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:32.021771 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgcqc" podUID="02c6d8f5-97ff-41e0-8e38-71e1b0c87f54" Oct 31 12:31:33.024093 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:33.024068 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:31:33.024475 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:33.024186 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nbvz" podUID="bbf2e0b3-d31b-44ca-934c-3710f4262779" Oct 31 12:31:33.187274 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:33.187238 2577 generic.go:356] "Generic (PLEG): container finished" podID="8c75b6d0-3fc9-4e2e-aba2-f464a48cab35" containerID="e540611cac188a7b5c4e6d95d8f3d5a6a39ca151cb1a75c651ce67cd5473ec33" exitCode=0 Oct 31 12:31:33.187441 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:33.187302 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j7fd8" event={"ID":"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35","Type":"ContainerDied","Data":"e540611cac188a7b5c4e6d95d8f3d5a6a39ca151cb1a75c651ce67cd5473ec33"} Oct 31 12:31:34.021202 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:34.021159 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:31:34.021202 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:34.021203 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:31:34.021453 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:34.021293 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgcqc" podUID="02c6d8f5-97ff-41e0-8e38-71e1b0c87f54" Oct 31 12:31:34.021453 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:34.021434 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgrhs" podUID="ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a" Oct 31 12:31:34.193217 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:34.193181 2577 generic.go:356] "Generic (PLEG): container finished" podID="8c75b6d0-3fc9-4e2e-aba2-f464a48cab35" containerID="c3f14f59145ed299d794b96da8d04940f13a270c0db13c7b72220643aafbe741" exitCode=0 Oct 31 12:31:34.193729 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:34.193225 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j7fd8" event={"ID":"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35","Type":"ContainerDied","Data":"c3f14f59145ed299d794b96da8d04940f13a270c0db13c7b72220643aafbe741"} Oct 31 12:31:34.697050 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:34.697007 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf2e0b3-d31b-44ca-934c-3710f4262779-metrics-certs\") pod \"network-metrics-daemon-8nbvz\" (UID: \"bbf2e0b3-d31b-44ca-934c-3710f4262779\") " pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:31:34.697201 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:34.697059 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a-original-pull-secret\") pod \"global-pull-secret-syncer-wgrhs\" (UID: \"ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a\") " pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:31:34.697201 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:34.697177 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 12:31:34.697201 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:34.697177 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:31:34.697344 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:34.697240 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a-original-pull-secret podName:ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a nodeName:}" failed. No retries permitted until 2026-10-31 12:31:50.697227005 +0000 UTC m=+34.113931461 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a-original-pull-secret") pod "global-pull-secret-syncer-wgrhs" (UID: "ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a") : object "kube-system"/"original-pull-secret" not registered Oct 31 12:31:34.697344 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:34.697256 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbf2e0b3-d31b-44ca-934c-3710f4262779-metrics-certs podName:bbf2e0b3-d31b-44ca-934c-3710f4262779 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:50.697249253 +0000 UTC m=+34.113953691 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bbf2e0b3-d31b-44ca-934c-3710f4262779-metrics-certs") pod "network-metrics-daemon-8nbvz" (UID: "bbf2e0b3-d31b-44ca-934c-3710f4262779") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:31:34.898242 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:34.898205 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph9jm\" (UniqueName: \"kubernetes.io/projected/02c6d8f5-97ff-41e0-8e38-71e1b0c87f54-kube-api-access-ph9jm\") pod \"network-check-target-bgcqc\" (UID: \"02c6d8f5-97ff-41e0-8e38-71e1b0c87f54\") " pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:31:34.898419 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:34.898395 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 12:31:34.898460 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:34.898424 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 12:31:34.898460 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:34.898439 2577 projected.go:196] Error preparing data for projected volume kube-api-access-ph9jm for pod openshift-network-diagnostics/network-check-target-bgcqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:31:34.898545 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:34.898504 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02c6d8f5-97ff-41e0-8e38-71e1b0c87f54-kube-api-access-ph9jm podName:02c6d8f5-97ff-41e0-8e38-71e1b0c87f54 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:50.89848522 +0000 UTC m=+34.315189657 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ph9jm" (UniqueName: "kubernetes.io/projected/02c6d8f5-97ff-41e0-8e38-71e1b0c87f54-kube-api-access-ph9jm") pod "network-check-target-bgcqc" (UID: "02c6d8f5-97ff-41e0-8e38-71e1b0c87f54") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:31:35.021452 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:35.021361 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:31:35.021613 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:35.021514 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nbvz" podUID="bbf2e0b3-d31b-44ca-934c-3710f4262779" Oct 31 12:31:35.199546 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:35.199507 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j7fd8" event={"ID":"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35","Type":"ContainerStarted","Data":"401403d0ecbf96fde1ebc9192e169987436918f575e3c141430d31765b910e5c"} Oct 31 12:31:35.227266 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:35.227189 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-j7fd8" podStartSLOduration=7.254147793 podStartE2EDuration="18.227170558s" totalImagesPullingTime="10.973022765s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:31:17 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:35.226498737 +0000 UTC m=+18.643203199" watchObservedRunningTime="2026-10-31 12:31:35.227170558 +0000 UTC m=+18.643875025" Oct 31 12:31:36.021865 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:36.021831 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:31:36.022012 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:36.021866 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:31:36.022012 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:36.021950 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgrhs" podUID="ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a" Oct 31 12:31:36.022135 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:36.022024 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgcqc" podUID="02c6d8f5-97ff-41e0-8e38-71e1b0c87f54" Oct 31 12:31:37.021877 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:37.021714 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:31:37.022259 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:37.021982 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nbvz" podUID="bbf2e0b3-d31b-44ca-934c-3710f4262779" Oct 31 12:31:37.206859 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:37.206821 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" event={"ID":"0b091d91-47bc-4480-85b2-e23a8139df66","Type":"ContainerStarted","Data":"011f1c256bafcd3026ad1e27ea9a06f841c0c68bb0cfebd35a2f036899a8c1f3"} Oct 31 12:31:37.207051 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:37.206991 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" event={"ID":"0b091d91-47bc-4480-85b2-e23a8139df66","Type":"ContainerStarted","Data":"a9342fcc1039419fc72dd1e2ef0d1ffa988aa3651e037c3a0668c4a65fa42a89"} Oct 31 12:31:37.207107 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:37.207084 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" event={"ID":"0b091d91-47bc-4480-85b2-e23a8139df66","Type":"ContainerStarted","Data":"7a9f68e111c894a3344ef592293f4e75d81f15dced1c1ff6861ddae6a4b2a7cc"} Oct 31 12:31:37.207211 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:37.207196 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" event={"ID":"0b091d91-47bc-4480-85b2-e23a8139df66","Type":"ContainerStarted","Data":"f7709255e7aef812a0bce6ce7276ea8047a856d437ad9dc0de87518ce9df3b38"} Oct 31 12:31:37.207279 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:37.207266 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" event={"ID":"0b091d91-47bc-4480-85b2-e23a8139df66","Type":"ContainerStarted","Data":"77401f78f7dbbbe6019fe656a7d1186973589f5db095d1ef4321fb2a7b551576"} Oct 31 12:31:37.207349 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:37.207337 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" event={"ID":"0b091d91-47bc-4480-85b2-e23a8139df66","Type":"ContainerStarted","Data":"ef44a9ffa8ec4303019608decf4643b44bc651d598070bfc69aee2ae4c2f2e99"} Oct 31 12:31:38.021612 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:38.021576 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:31:38.021612 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:38.021609 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:31:38.021834 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:38.021713 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgrhs" podUID="ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a" Oct 31 12:31:38.021897 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:38.021826 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgcqc" podUID="02c6d8f5-97ff-41e0-8e38-71e1b0c87f54" Oct 31 12:31:39.021770 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:39.021735 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:31:39.021985 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:39.021883 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nbvz" podUID="bbf2e0b3-d31b-44ca-934c-3710f4262779" Oct 31 12:31:39.215590 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:39.215554 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" event={"ID":"0b091d91-47bc-4480-85b2-e23a8139df66","Type":"ContainerStarted","Data":"6a11360b0f5bf5e26eaa0f8f17af7d2809b083a778a494896450b00d0d580b5c"} Oct 31 12:31:40.021768 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:40.021733 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:31:40.022000 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:40.021734 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:31:40.022000 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:40.021845 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgcqc" podUID="02c6d8f5-97ff-41e0-8e38-71e1b0c87f54" Oct 31 12:31:40.022000 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:40.021915 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgrhs" podUID="ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a" Oct 31 12:31:41.021602 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:41.021563 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:31:41.021811 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:41.021720 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nbvz" podUID="bbf2e0b3-d31b-44ca-934c-3710f4262779" Oct 31 12:31:41.626681 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:41.626654 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5gdlb_26913ad4-6613-4f3f-9819-9be238088c0a/dns-node-resolver/0.log" Oct 31 12:31:42.021762 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:42.021712 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:31:42.021762 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:42.021746 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:31:42.021969 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:42.021831 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgrhs" podUID="ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a" Oct 31 12:31:42.021969 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:42.021941 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgcqc" podUID="02c6d8f5-97ff-41e0-8e38-71e1b0c87f54" Oct 31 12:31:42.225652 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:42.225608 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" event={"ID":"0b091d91-47bc-4480-85b2-e23a8139df66","Type":"ContainerStarted","Data":"c4c8b9c6061ce325a787672347aa2e2cd892302c2f24df17a7cceb689f6a3f51"} Oct 31 12:31:42.225894 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:42.225872 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:42.241808 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:42.241782 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:42.259344 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:42.259294 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" podStartSLOduration=7.828031638 podStartE2EDuration="25.259282377s" totalImagesPullingTime="17.431250739s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:31:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:42.258979612 +0000 UTC m=+25.675684071" watchObservedRunningTime="2026-10-31 12:31:42.259282377 +0000 UTC m=+25.675986836" Oct 31 12:31:42.984424 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:42.984258 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tk5vl_14dc1e6d-70e1-4ba5-bffb-d9299433756d/node-ca/0.log" Oct 31 12:31:43.021552 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:43.021524 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:31:43.021736 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:43.021680 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nbvz" podUID="bbf2e0b3-d31b-44ca-934c-3710f4262779" Oct 31 12:31:43.037752 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:43.037679 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8nbvz"] Oct 31 12:31:43.041194 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:43.041169 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bgcqc"] Oct 31 12:31:43.042730 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:43.041828 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:31:43.042730 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:43.041936 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wgrhs"] Oct 31 12:31:43.042730 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:43.041953 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgcqc" podUID="02c6d8f5-97ff-41e0-8e38-71e1b0c87f54" Oct 31 12:31:43.042730 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:43.042003 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:31:43.042730 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:43.042082 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgrhs" podUID="ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a" Oct 31 12:31:43.227129 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:43.227104 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:31:43.227264 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:43.227207 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nbvz" podUID="bbf2e0b3-d31b-44ca-934c-3710f4262779" Oct 31 12:31:43.227264 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:43.227231 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:43.227554 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:43.227298 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:43.244060 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:43.244035 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:31:45.021466 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:45.021434 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:31:45.021466 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:45.021468 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:31:45.022047 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:45.021433 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:31:45.022047 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:45.021555 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgrhs" podUID="ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a" Oct 31 12:31:45.022047 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:45.021634 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nbvz" podUID="bbf2e0b3-d31b-44ca-934c-3710f4262779" Oct 31 12:31:45.022047 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:45.021706 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgcqc" podUID="02c6d8f5-97ff-41e0-8e38-71e1b0c87f54" Oct 31 12:31:47.022220 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:47.022183 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:31:47.022664 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:47.022280 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgcqc" podUID="02c6d8f5-97ff-41e0-8e38-71e1b0c87f54" Oct 31 12:31:47.022664 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:47.022360 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:31:47.022664 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:47.022449 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:31:47.022664 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:47.022471 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgrhs" podUID="ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a" Oct 31 12:31:47.022664 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:47.022530 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nbvz" podUID="bbf2e0b3-d31b-44ca-934c-3710f4262779" Oct 31 12:31:49.021104 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:49.021074 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:31:49.021519 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:49.021140 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:31:49.021519 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:49.021136 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:31:49.021519 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:49.021226 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nbvz" podUID="bbf2e0b3-d31b-44ca-934c-3710f4262779" Oct 31 12:31:49.021519 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:49.021342 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgcqc" podUID="02c6d8f5-97ff-41e0-8e38-71e1b0c87f54" Oct 31 12:31:49.021519 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:49.021419 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgrhs" podUID="ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a" Oct 31 12:31:50.704030 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:50.704000 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a-original-pull-secret\") pod \"global-pull-secret-syncer-wgrhs\" (UID: \"ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a\") " pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:31:50.704518 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:50.704055 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf2e0b3-d31b-44ca-934c-3710f4262779-metrics-certs\") pod \"network-metrics-daemon-8nbvz\" (UID: \"bbf2e0b3-d31b-44ca-934c-3710f4262779\") " pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:31:50.704518 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:50.704138 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:31:50.704518 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:50.704139 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 12:31:50.704518 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:50.704188 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbf2e0b3-d31b-44ca-934c-3710f4262779-metrics-certs podName:bbf2e0b3-d31b-44ca-934c-3710f4262779 nodeName:}" failed. No retries permitted until 2026-10-31 12:32:22.704175481 +0000 UTC m=+66.120879921 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bbf2e0b3-d31b-44ca-934c-3710f4262779-metrics-certs") pod "network-metrics-daemon-8nbvz" (UID: "bbf2e0b3-d31b-44ca-934c-3710f4262779") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:31:50.704518 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:50.704200 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a-original-pull-secret podName:ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a nodeName:}" failed. No retries permitted until 2026-10-31 12:32:22.704194095 +0000 UTC m=+66.120898531 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a-original-pull-secret") pod "global-pull-secret-syncer-wgrhs" (UID: "ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a") : object "kube-system"/"original-pull-secret" not registered Oct 31 12:31:50.904955 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:50.904917 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph9jm\" (UniqueName: \"kubernetes.io/projected/02c6d8f5-97ff-41e0-8e38-71e1b0c87f54-kube-api-access-ph9jm\") pod \"network-check-target-bgcqc\" (UID: \"02c6d8f5-97ff-41e0-8e38-71e1b0c87f54\") " pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:31:50.905117 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:50.905088 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 12:31:50.905117 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:50.905113 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 12:31:50.905189 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:50.905125 2577 projected.go:196] Error preparing data for projected volume kube-api-access-ph9jm for pod openshift-network-diagnostics/network-check-target-bgcqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:31:50.905189 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:50.905180 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02c6d8f5-97ff-41e0-8e38-71e1b0c87f54-kube-api-access-ph9jm podName:02c6d8f5-97ff-41e0-8e38-71e1b0c87f54 nodeName:}" failed. No retries permitted until 2026-10-31 12:32:22.90516622 +0000 UTC m=+66.321870657 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ph9jm" (UniqueName: "kubernetes.io/projected/02c6d8f5-97ff-41e0-8e38-71e1b0c87f54-kube-api-access-ph9jm") pod "network-check-target-bgcqc" (UID: "02c6d8f5-97ff-41e0-8e38-71e1b0c87f54") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:31:51.021822 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:51.021737 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:31:51.021981 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:51.021738 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:31:51.021981 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:51.021855 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgcqc" podUID="02c6d8f5-97ff-41e0-8e38-71e1b0c87f54" Oct 31 12:31:51.021981 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:51.021952 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgrhs" podUID="ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a" Oct 31 12:31:51.021981 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:51.021752 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:31:51.022166 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:51.022068 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nbvz" podUID="bbf2e0b3-d31b-44ca-934c-3710f4262779" Oct 31 12:31:53.021439 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:53.021400 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:31:53.021869 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:53.021456 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:31:53.021869 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:53.021472 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:31:53.021869 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:53.021562 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nbvz" podUID="bbf2e0b3-d31b-44ca-934c-3710f4262779" Oct 31 12:31:53.021869 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:53.021670 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgcqc" podUID="02c6d8f5-97ff-41e0-8e38-71e1b0c87f54" Oct 31 12:31:53.021869 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:53.021739 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgrhs" podUID="ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a" Oct 31 12:31:55.021574 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:55.021545 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:31:55.022051 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:55.021546 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:31:55.022051 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:55.021691 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgcqc" podUID="02c6d8f5-97ff-41e0-8e38-71e1b0c87f54" Oct 31 12:31:55.022051 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:55.021545 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:31:55.022051 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:55.021753 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nbvz" podUID="bbf2e0b3-d31b-44ca-934c-3710f4262779" Oct 31 12:31:55.022051 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:55.021806 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgrhs" podUID="ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a" Oct 31 12:31:57.022663 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:57.022608 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:31:57.023076 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:57.022747 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:31:57.023076 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:57.022778 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nbvz" podUID="bbf2e0b3-d31b-44ca-934c-3710f4262779" Oct 31 12:31:57.023076 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:57.022791 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:31:57.023076 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:57.022836 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgrhs" podUID="ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a" Oct 31 12:31:57.023076 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:57.022872 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgcqc" podUID="02c6d8f5-97ff-41e0-8e38-71e1b0c87f54" Oct 31 12:31:59.020789 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:59.020752 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:31:59.021226 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:59.020765 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:31:59.021226 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:59.020875 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nbvz" podUID="bbf2e0b3-d31b-44ca-934c-3710f4262779" Oct 31 12:31:59.021226 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:31:59.020765 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:31:59.021226 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:59.020952 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgcqc" podUID="02c6d8f5-97ff-41e0-8e38-71e1b0c87f54" Oct 31 12:31:59.021226 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:31:59.021033 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgrhs" podUID="ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a" Oct 31 12:32:01.021474 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:01.021435 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:32:01.021474 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:01.021465 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:32:01.022184 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:01.021465 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:32:01.022184 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:32:01.021550 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgcqc" podUID="02c6d8f5-97ff-41e0-8e38-71e1b0c87f54" Oct 31 12:32:01.022184 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:32:01.021655 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgrhs" podUID="ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a" Oct 31 12:32:01.022184 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:32:01.021709 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nbvz" podUID="bbf2e0b3-d31b-44ca-934c-3710f4262779" Oct 31 12:32:03.021951 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:03.021774 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:32:03.022384 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:03.021778 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:32:03.022384 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:32:03.022025 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nbvz" podUID="bbf2e0b3-d31b-44ca-934c-3710f4262779" Oct 31 12:32:03.022384 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:03.021807 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:32:03.022384 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:32:03.022100 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgrhs" podUID="ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a" Oct 31 12:32:03.022384 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:32:03.022160 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgcqc" podUID="02c6d8f5-97ff-41e0-8e38-71e1b0c87f54" Oct 31 12:32:05.021184 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:05.021147 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:32:05.021700 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:05.021160 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:32:05.021700 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:32:05.021257 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgrhs" podUID="ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a" Oct 31 12:32:05.021700 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:05.021270 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:32:05.021700 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:32:05.021393 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nbvz" podUID="bbf2e0b3-d31b-44ca-934c-3710f4262779" Oct 31 12:32:05.021700 ip-10-0-131-157 kubenswrapper[2577]: E1031 12:32:05.021442 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgcqc" podUID="02c6d8f5-97ff-41e0-8e38-71e1b0c87f54" Oct 31 12:32:06.366589 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.366558 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeReady" Oct 31 12:32:06.367110 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.366704 2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 12:32:06.452044 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.452011 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pwtgm"] Oct 31 12:32:06.453671 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.453656 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pwtgm" Oct 31 12:32:06.455345 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.455322 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bdwh8"] Oct 31 12:32:06.456787 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.456760 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bdwh8" Oct 31 12:32:06.459250 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.459232 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 12:32:06.459611 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.459597 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 12:32:06.459775 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.459759 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-mrn76\"" Oct 31 12:32:06.459832 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.459762 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 12:32:06.466922 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.466902 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pwtgm"] Oct 31 12:32:06.476419 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.476398 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bdwh8"] Oct 31 12:32:06.478034 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.478016 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9t68r"] Oct 31 12:32:06.479785 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.479768 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9t68r" Oct 31 12:32:06.494917 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.494890 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9t68r"] Oct 31 12:32:06.559751 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.559710 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 12:32:06.559927 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.559836 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 12:32:06.559927 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.559836 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-52dg7\"" Oct 31 12:32:06.559927 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.559871 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 12:32:06.560043 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.559954 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 12:32:06.560043 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.560012 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nthx6\"" Oct 31 12:32:06.560110 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.560053 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 12:32:06.560143 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.560124 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 12:32:06.619688 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.619615 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/97cf3735-d794-4628-9048-252f949317b9-crio-socket\") pod \"insights-runtime-extractor-9t68r\" (UID: \"97cf3735-d794-4628-9048-252f949317b9\") " pod="openshift-insights/insights-runtime-extractor-9t68r" Oct 31 12:32:06.619688 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.619663 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f4a19150-beb0-4289-91ee-b1668968e6ca-config-volume\") pod \"dns-default-bdwh8\" (UID: \"f4a19150-beb0-4289-91ee-b1668968e6ca\") " pod="openshift-dns/dns-default-bdwh8" Oct 31 12:32:06.619806 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.619758 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdlgx\" (UniqueName: \"kubernetes.io/projected/45870e34-5e37-443a-826b-81c9c031051c-kube-api-access-kdlgx\") pod \"ingress-canary-pwtgm\" (UID: \"45870e34-5e37-443a-826b-81c9c031051c\") " pod="openshift-ingress-canary/ingress-canary-pwtgm" Oct 31 12:32:06.619806 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.619789 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4a19150-beb0-4289-91ee-b1668968e6ca-metrics-tls\") pod \"dns-default-bdwh8\" (UID: \"f4a19150-beb0-4289-91ee-b1668968e6ca\") " pod="openshift-dns/dns-default-bdwh8" Oct 31 12:32:06.619875 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.619840 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64hd7\" (UniqueName: \"kubernetes.io/projected/f4a19150-beb0-4289-91ee-b1668968e6ca-kube-api-access-64hd7\") pod \"dns-default-bdwh8\" (UID: \"f4a19150-beb0-4289-91ee-b1668968e6ca\") " pod="openshift-dns/dns-default-bdwh8" Oct 31 12:32:06.619875 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.619856 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/97cf3735-d794-4628-9048-252f949317b9-data-volume\") pod \"insights-runtime-extractor-9t68r\" (UID: \"97cf3735-d794-4628-9048-252f949317b9\") " pod="openshift-insights/insights-runtime-extractor-9t68r" Oct 31 12:32:06.619947 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.619879 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/97cf3735-d794-4628-9048-252f949317b9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9t68r\" (UID: \"97cf3735-d794-4628-9048-252f949317b9\") " pod="openshift-insights/insights-runtime-extractor-9t68r" Oct 31 12:32:06.619947 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.619894 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4a19150-beb0-4289-91ee-b1668968e6ca-tmp-dir\") pod \"dns-default-bdwh8\" (UID: \"f4a19150-beb0-4289-91ee-b1668968e6ca\") " pod="openshift-dns/dns-default-bdwh8" Oct 31 12:32:06.619947 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.619909 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97cf3735-d794-4628-9048-252f949317b9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9t68r\" (UID: \"97cf3735-d794-4628-9048-252f949317b9\") " pod="openshift-insights/insights-runtime-extractor-9t68r" Oct 31 12:32:06.620037 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.620013 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/45870e34-5e37-443a-826b-81c9c031051c-cert\") pod \"ingress-canary-pwtgm\" (UID: \"45870e34-5e37-443a-826b-81c9c031051c\") " pod="openshift-ingress-canary/ingress-canary-pwtgm" Oct 31 12:32:06.620078 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.620053 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xpnl\" (UniqueName: \"kubernetes.io/projected/97cf3735-d794-4628-9048-252f949317b9-kube-api-access-2xpnl\") pod \"insights-runtime-extractor-9t68r\" (UID: \"97cf3735-d794-4628-9048-252f949317b9\") " pod="openshift-insights/insights-runtime-extractor-9t68r" Oct 31 12:32:06.721086 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.721047 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdlgx\" (UniqueName: \"kubernetes.io/projected/45870e34-5e37-443a-826b-81c9c031051c-kube-api-access-kdlgx\") pod \"ingress-canary-pwtgm\" (UID: \"45870e34-5e37-443a-826b-81c9c031051c\") " pod="openshift-ingress-canary/ingress-canary-pwtgm" Oct 31 12:32:06.721086 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.721084 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4a19150-beb0-4289-91ee-b1668968e6ca-metrics-tls\") pod \"dns-default-bdwh8\" (UID: \"f4a19150-beb0-4289-91ee-b1668968e6ca\") " pod="openshift-dns/dns-default-bdwh8" Oct 31 12:32:06.721349 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.721108 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64hd7\" (UniqueName: \"kubernetes.io/projected/f4a19150-beb0-4289-91ee-b1668968e6ca-kube-api-access-64hd7\") pod \"dns-default-bdwh8\" (UID: \"f4a19150-beb0-4289-91ee-b1668968e6ca\") " pod="openshift-dns/dns-default-bdwh8" Oct 31 12:32:06.721349 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.721124 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/97cf3735-d794-4628-9048-252f949317b9-data-volume\") pod \"insights-runtime-extractor-9t68r\" (UID: \"97cf3735-d794-4628-9048-252f949317b9\") " pod="openshift-insights/insights-runtime-extractor-9t68r" Oct 31 12:32:06.721349 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.721195 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/97cf3735-d794-4628-9048-252f949317b9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9t68r\" (UID: \"97cf3735-d794-4628-9048-252f949317b9\") " pod="openshift-insights/insights-runtime-extractor-9t68r" Oct 31 12:32:06.721349 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.721233 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4a19150-beb0-4289-91ee-b1668968e6ca-tmp-dir\") pod \"dns-default-bdwh8\" (UID: \"f4a19150-beb0-4289-91ee-b1668968e6ca\") " pod="openshift-dns/dns-default-bdwh8" Oct 31 12:32:06.721349 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.721253 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97cf3735-d794-4628-9048-252f949317b9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9t68r\" (UID: \"97cf3735-d794-4628-9048-252f949317b9\") " pod="openshift-insights/insights-runtime-extractor-9t68r" Oct 31 12:32:06.721349 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.721283 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/45870e34-5e37-443a-826b-81c9c031051c-cert\") pod \"ingress-canary-pwtgm\" (UID: \"45870e34-5e37-443a-826b-81c9c031051c\") " pod="openshift-ingress-canary/ingress-canary-pwtgm" Oct 31 12:32:06.721349 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.721297 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xpnl\" (UniqueName: \"kubernetes.io/projected/97cf3735-d794-4628-9048-252f949317b9-kube-api-access-2xpnl\") pod \"insights-runtime-extractor-9t68r\" (UID: \"97cf3735-d794-4628-9048-252f949317b9\") " pod="openshift-insights/insights-runtime-extractor-9t68r" Oct 31 12:32:06.721703 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.721481 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/97cf3735-d794-4628-9048-252f949317b9-crio-socket\") pod \"insights-runtime-extractor-9t68r\" (UID: \"97cf3735-d794-4628-9048-252f949317b9\") " pod="openshift-insights/insights-runtime-extractor-9t68r" Oct 31 12:32:06.721703 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.721498 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/97cf3735-d794-4628-9048-252f949317b9-data-volume\") pod \"insights-runtime-extractor-9t68r\" (UID: \"97cf3735-d794-4628-9048-252f949317b9\") " pod="openshift-insights/insights-runtime-extractor-9t68r" Oct 31 12:32:06.721703 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.721510 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f4a19150-beb0-4289-91ee-b1668968e6ca-config-volume\") pod \"dns-default-bdwh8\" (UID: \"f4a19150-beb0-4289-91ee-b1668968e6ca\") " pod="openshift-dns/dns-default-bdwh8" Oct 31 12:32:06.721703 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.721596 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4a19150-beb0-4289-91ee-b1668968e6ca-tmp-dir\") pod \"dns-default-bdwh8\" (UID: \"f4a19150-beb0-4289-91ee-b1668968e6ca\") " pod="openshift-dns/dns-default-bdwh8" Oct 31 12:32:06.721889 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.721727 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/97cf3735-d794-4628-9048-252f949317b9-crio-socket\") pod \"insights-runtime-extractor-9t68r\" (UID: \"97cf3735-d794-4628-9048-252f949317b9\") " pod="openshift-insights/insights-runtime-extractor-9t68r" Oct 31 12:32:06.721935 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.721925 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/97cf3735-d794-4628-9048-252f949317b9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9t68r\" (UID: \"97cf3735-d794-4628-9048-252f949317b9\") " pod="openshift-insights/insights-runtime-extractor-9t68r" Oct 31 12:32:06.722138 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.722118 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f4a19150-beb0-4289-91ee-b1668968e6ca-config-volume\") pod \"dns-default-bdwh8\" (UID: \"f4a19150-beb0-4289-91ee-b1668968e6ca\") " pod="openshift-dns/dns-default-bdwh8" Oct 31 12:32:06.725246 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.725224 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97cf3735-d794-4628-9048-252f949317b9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9t68r\" (UID: \"97cf3735-d794-4628-9048-252f949317b9\") " pod="openshift-insights/insights-runtime-extractor-9t68r" Oct 31 12:32:06.725324 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.725250 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4a19150-beb0-4289-91ee-b1668968e6ca-metrics-tls\") pod \"dns-default-bdwh8\" (UID: \"f4a19150-beb0-4289-91ee-b1668968e6ca\") " pod="openshift-dns/dns-default-bdwh8" Oct 31 12:32:06.725359 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.725331 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/45870e34-5e37-443a-826b-81c9c031051c-cert\") pod \"ingress-canary-pwtgm\" (UID: \"45870e34-5e37-443a-826b-81c9c031051c\") " pod="openshift-ingress-canary/ingress-canary-pwtgm" Oct 31 12:32:06.735129 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.735103 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdlgx\" (UniqueName: \"kubernetes.io/projected/45870e34-5e37-443a-826b-81c9c031051c-kube-api-access-kdlgx\") pod \"ingress-canary-pwtgm\" (UID: \"45870e34-5e37-443a-826b-81c9c031051c\") " pod="openshift-ingress-canary/ingress-canary-pwtgm" Oct 31 12:32:06.735286 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.735145 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64hd7\" (UniqueName: \"kubernetes.io/projected/f4a19150-beb0-4289-91ee-b1668968e6ca-kube-api-access-64hd7\") pod \"dns-default-bdwh8\" (UID: \"f4a19150-beb0-4289-91ee-b1668968e6ca\") " pod="openshift-dns/dns-default-bdwh8" Oct 31 12:32:06.735361 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.735160 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xpnl\" (UniqueName: \"kubernetes.io/projected/97cf3735-d794-4628-9048-252f949317b9-kube-api-access-2xpnl\") pod \"insights-runtime-extractor-9t68r\" (UID: \"97cf3735-d794-4628-9048-252f949317b9\") " pod="openshift-insights/insights-runtime-extractor-9t68r" Oct 31 12:32:06.765452 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.765414 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pwtgm" Oct 31 12:32:06.771829 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.771810 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bdwh8" Oct 31 12:32:06.790700 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:06.790627 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9t68r" Oct 31 12:32:07.022686 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:07.022625 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:32:07.022848 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:07.022746 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:32:07.022895 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:07.022881 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:32:07.026917 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:07.026894 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 12:32:07.028029 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:07.028011 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 12:32:07.028310 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:07.028290 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l77g8\"" Oct 31 12:32:07.028493 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:07.028464 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 12:32:07.028493 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:07.028486 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-l5d6v\"" Oct 31 12:32:07.028663 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:07.028520 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zbx6v\"" Oct 31 12:32:07.028663 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:07.028557 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 12:32:07.055214 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:07.055185 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bdwh8"] Oct 31 12:32:07.058072 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:07.058042 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pwtgm"] Oct 31 12:32:07.058917 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:32:07.058884 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4a19150_beb0_4289_91ee_b1668968e6ca.slice/crio-4242a5b80f9bf08c64de825adaa8d447534331d8362cdedc8dbf2fc82d20f911 WatchSource:0}: Error finding container 4242a5b80f9bf08c64de825adaa8d447534331d8362cdedc8dbf2fc82d20f911: Status 404 returned error can't find the container with id 4242a5b80f9bf08c64de825adaa8d447534331d8362cdedc8dbf2fc82d20f911 Oct 31 12:32:07.061176 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:32:07.061151 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45870e34_5e37_443a_826b_81c9c031051c.slice/crio-e6eb922f4a0cd05a8f419575ee3f34f4ac887c9b83ebc2138465e7b10d647228 WatchSource:0}: Error finding container e6eb922f4a0cd05a8f419575ee3f34f4ac887c9b83ebc2138465e7b10d647228: Status 404 returned error can't find the container with id e6eb922f4a0cd05a8f419575ee3f34f4ac887c9b83ebc2138465e7b10d647228 Oct 31 12:32:07.078447 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:07.078426 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9t68r"] Oct 31 12:32:07.081221 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:32:07.081200 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97cf3735_d794_4628_9048_252f949317b9.slice/crio-7f9fad3cf08ec8fb0a264377931414fafbf489a06b13f1a62d1867691c252545 WatchSource:0}: Error finding container 7f9fad3cf08ec8fb0a264377931414fafbf489a06b13f1a62d1867691c252545: Status 404 returned error can't find the container with id 7f9fad3cf08ec8fb0a264377931414fafbf489a06b13f1a62d1867691c252545 Oct 31 12:32:07.279040 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:07.278951 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pwtgm" event={"ID":"45870e34-5e37-443a-826b-81c9c031051c","Type":"ContainerStarted","Data":"e6eb922f4a0cd05a8f419575ee3f34f4ac887c9b83ebc2138465e7b10d647228"} Oct 31 12:32:07.280516 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:07.280488 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9t68r" event={"ID":"97cf3735-d794-4628-9048-252f949317b9","Type":"ContainerStarted","Data":"70039bbc947f37589b490567ec4e30e85e05b2f674fab2741f8c3c7ed0dc3149"} Oct 31 12:32:07.280625 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:07.280540 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9t68r" event={"ID":"97cf3735-d794-4628-9048-252f949317b9","Type":"ContainerStarted","Data":"7f9fad3cf08ec8fb0a264377931414fafbf489a06b13f1a62d1867691c252545"} Oct 31 12:32:07.281537 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:07.281508 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bdwh8" event={"ID":"f4a19150-beb0-4289-91ee-b1668968e6ca","Type":"ContainerStarted","Data":"4242a5b80f9bf08c64de825adaa8d447534331d8362cdedc8dbf2fc82d20f911"} Oct 31 12:32:08.286013 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:08.285978 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9t68r" event={"ID":"97cf3735-d794-4628-9048-252f949317b9","Type":"ContainerStarted","Data":"84097832cb002b45fd61c3b456066ffc1a84f696d4c5cd0a12d07c096a967ec5"} Oct 31 12:32:09.091003 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:09.090973 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-57b586f5c7-hbnwz"] Oct 31 12:32:09.093855 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:09.093833 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:09.102447 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:09.102391 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 31 12:32:09.103511 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:09.103486 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 31 12:32:09.109093 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:09.109067 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57b586f5c7-hbnwz"] Oct 31 12:32:09.198830 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:09.198800 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 31 12:32:09.198972 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:09.198942 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 31 12:32:09.199026 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:09.199017 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 31 12:32:09.199239 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:09.199221 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 31 12:32:09.199318 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:09.199261 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 31 12:32:09.199371 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:09.199320 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-m7qs2\"" Oct 31 12:32:09.241835 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:09.241807 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c91a35c-7933-4399-b7ff-4c1226d72d20-tmp\") pod \"console-57b586f5c7-hbnwz\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:09.242020 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:09.241850 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c91a35c-7933-4399-b7ff-4c1226d72d20-console-serving-cert\") pod \"console-57b586f5c7-hbnwz\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:09.242020 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:09.241879 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c91a35c-7933-4399-b7ff-4c1226d72d20-service-ca\") pod \"console-57b586f5c7-hbnwz\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:09.242020 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:09.241904 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c91a35c-7933-4399-b7ff-4c1226d72d20-oauth-serving-cert\") pod \"console-57b586f5c7-hbnwz\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:09.242020 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:09.241961 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c91a35c-7933-4399-b7ff-4c1226d72d20-console-oauth-config\") pod \"console-57b586f5c7-hbnwz\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:09.242020 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:09.241988 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8c91a35c-7933-4399-b7ff-4c1226d72d20-console-config\") pod \"console-57b586f5c7-hbnwz\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:09.242020 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:09.242012 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls87f\" (UniqueName: \"kubernetes.io/projected/8c91a35c-7933-4399-b7ff-4c1226d72d20-kube-api-access-ls87f\") pod \"console-57b586f5c7-hbnwz\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:09.292555 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:09.292515 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bdwh8" event={"ID":"f4a19150-beb0-4289-91ee-b1668968e6ca","Type":"ContainerStarted","Data":"cae0e49c86ce8cd8266ee04eb067380afca32940a5406bcbc92b36c24f236544"} Oct 31 12:32:09.345074 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:09.343236 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c91a35c-7933-4399-b7ff-4c1226d72d20-console-oauth-config\") pod \"console-57b586f5c7-hbnwz\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:09.345074 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:09.343261 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8c91a35c-7933-4399-b7ff-4c1226d72d20-console-config\") pod \"console-57b586f5c7-hbnwz\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:09.345074 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:09.343278 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls87f\" (UniqueName: \"kubernetes.io/projected/8c91a35c-7933-4399-b7ff-4c1226d72d20-kube-api-access-ls87f\") pod \"console-57b586f5c7-hbnwz\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:09.345074 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:09.343322 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c91a35c-7933-4399-b7ff-4c1226d72d20-tmp\") pod \"console-57b586f5c7-hbnwz\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:09.345074 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:09.343339 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c91a35c-7933-4399-b7ff-4c1226d72d20-console-serving-cert\") pod \"console-57b586f5c7-hbnwz\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:09.345074 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:09.343353 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c91a35c-7933-4399-b7ff-4c1226d72d20-service-ca\") pod \"console-57b586f5c7-hbnwz\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:09.345074 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:09.343369 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c91a35c-7933-4399-b7ff-4c1226d72d20-oauth-serving-cert\") pod \"console-57b586f5c7-hbnwz\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:09.345074 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:09.343900 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c91a35c-7933-4399-b7ff-4c1226d72d20-oauth-serving-cert\") pod \"console-57b586f5c7-hbnwz\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:09.345074 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:09.344723 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8c91a35c-7933-4399-b7ff-4c1226d72d20-console-config\") pod \"console-57b586f5c7-hbnwz\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:09.346012 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:09.345539 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c91a35c-7933-4399-b7ff-4c1226d72d20-service-ca\") pod \"console-57b586f5c7-hbnwz\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:09.346012 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:09.345833 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c91a35c-7933-4399-b7ff-4c1226d72d20-tmp\") pod \"console-57b586f5c7-hbnwz\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:09.348031 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:09.347977 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c91a35c-7933-4399-b7ff-4c1226d72d20-console-oauth-config\") pod \"console-57b586f5c7-hbnwz\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:09.348474 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:09.348435 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c91a35c-7933-4399-b7ff-4c1226d72d20-console-serving-cert\") pod \"console-57b586f5c7-hbnwz\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:09.392248 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:09.392215 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls87f\" (UniqueName: \"kubernetes.io/projected/8c91a35c-7933-4399-b7ff-4c1226d72d20-kube-api-access-ls87f\") pod \"console-57b586f5c7-hbnwz\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:09.408085 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:09.407850 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:09.693566 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:09.693536 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57b586f5c7-hbnwz"] Oct 31 12:32:09.697992 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:32:09.697962 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c91a35c_7933_4399_b7ff_4c1226d72d20.slice/crio-a633f11a094758484648b16d5585aebc43099d3f40a08aa8223d97224bf41d0e WatchSource:0}: Error finding container a633f11a094758484648b16d5585aebc43099d3f40a08aa8223d97224bf41d0e: Status 404 returned error can't find the container with id a633f11a094758484648b16d5585aebc43099d3f40a08aa8223d97224bf41d0e Oct 31 12:32:10.043940 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:10.043867 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-227cc"] Oct 31 12:32:10.047352 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:10.047324 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-227cc" Oct 31 12:32:10.052780 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:10.052755 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 31 12:32:10.053067 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:10.052955 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 31 12:32:10.053067 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:10.052973 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-ghpjn\"" Oct 31 12:32:10.053067 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:10.052985 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 12:32:10.053787 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:10.053768 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 12:32:10.053787 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:10.053774 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 12:32:10.057668 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:10.057272 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-227cc"] Oct 31 12:32:10.149507 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:10.149460 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ddebd541-4d2f-4861-9dd0-0ee66b03aaf2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-227cc\" (UID: \"ddebd541-4d2f-4861-9dd0-0ee66b03aaf2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-227cc" Oct 31 12:32:10.149701 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:10.149545 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddebd541-4d2f-4861-9dd0-0ee66b03aaf2-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-227cc\" (UID: \"ddebd541-4d2f-4861-9dd0-0ee66b03aaf2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-227cc" Oct 31 12:32:10.149753 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:10.149714 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8shn\" (UniqueName: \"kubernetes.io/projected/ddebd541-4d2f-4861-9dd0-0ee66b03aaf2-kube-api-access-m8shn\") pod \"prometheus-operator-9cd89b74c-227cc\" (UID: \"ddebd541-4d2f-4861-9dd0-0ee66b03aaf2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-227cc" Oct 31 12:32:10.149809 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:10.149778 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ddebd541-4d2f-4861-9dd0-0ee66b03aaf2-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-227cc\" (UID: \"ddebd541-4d2f-4861-9dd0-0ee66b03aaf2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-227cc" Oct 31 12:32:10.250840 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:10.250801 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ddebd541-4d2f-4861-9dd0-0ee66b03aaf2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-227cc\" (UID: \"ddebd541-4d2f-4861-9dd0-0ee66b03aaf2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-227cc" Oct 31 12:32:10.251006 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:10.250899 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddebd541-4d2f-4861-9dd0-0ee66b03aaf2-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-227cc\" (UID: \"ddebd541-4d2f-4861-9dd0-0ee66b03aaf2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-227cc" Oct 31 12:32:10.251006 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:10.250934 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8shn\" (UniqueName: \"kubernetes.io/projected/ddebd541-4d2f-4861-9dd0-0ee66b03aaf2-kube-api-access-m8shn\") pod \"prometheus-operator-9cd89b74c-227cc\" (UID: \"ddebd541-4d2f-4861-9dd0-0ee66b03aaf2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-227cc" Oct 31 12:32:10.251006 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:10.250955 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ddebd541-4d2f-4861-9dd0-0ee66b03aaf2-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-227cc\" (UID: \"ddebd541-4d2f-4861-9dd0-0ee66b03aaf2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-227cc" Oct 31 12:32:10.252137 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:10.252085 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddebd541-4d2f-4861-9dd0-0ee66b03aaf2-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-227cc\" (UID: \"ddebd541-4d2f-4861-9dd0-0ee66b03aaf2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-227cc" Oct 31 12:32:10.254231 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:10.254205 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ddebd541-4d2f-4861-9dd0-0ee66b03aaf2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-227cc\" (UID: \"ddebd541-4d2f-4861-9dd0-0ee66b03aaf2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-227cc" Oct 31 12:32:10.254231 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:10.254225 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ddebd541-4d2f-4861-9dd0-0ee66b03aaf2-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-227cc\" (UID: \"ddebd541-4d2f-4861-9dd0-0ee66b03aaf2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-227cc" Oct 31 12:32:10.264679 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:10.264635 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8shn\" (UniqueName: \"kubernetes.io/projected/ddebd541-4d2f-4861-9dd0-0ee66b03aaf2-kube-api-access-m8shn\") pod \"prometheus-operator-9cd89b74c-227cc\" (UID: \"ddebd541-4d2f-4861-9dd0-0ee66b03aaf2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-227cc" Oct 31 12:32:10.298182 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:10.298077 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bdwh8" event={"ID":"f4a19150-beb0-4289-91ee-b1668968e6ca","Type":"ContainerStarted","Data":"cc8f3abfef86d55fddd7fa359e1319177359e6c5e268f01b12ac4126ea57ebdc"} Oct 31 12:32:10.298607 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:10.298237 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bdwh8" Oct 31 12:32:10.301010 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:10.300983 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9t68r" event={"ID":"97cf3735-d794-4628-9048-252f949317b9","Type":"ContainerStarted","Data":"c824a8a41e3f2fce163b83e3080333284e67f807d3702ad9806fa15044f4f6bd"} Oct 31 12:32:10.302295 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:10.302266 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57b586f5c7-hbnwz" event={"ID":"8c91a35c-7933-4399-b7ff-4c1226d72d20","Type":"ContainerStarted","Data":"a633f11a094758484648b16d5585aebc43099d3f40a08aa8223d97224bf41d0e"} Oct 31 12:32:10.324207 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:10.324156 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-bdwh8" podStartSLOduration=2.244793864 podStartE2EDuration="4.324141883s" totalImagesPullingTime="2.079348019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:32:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:10.323003959 +0000 UTC m=+53.739708430" watchObservedRunningTime="2026-10-31 12:32:10.324141883 +0000 UTC m=+53.740846339" Oct 31 12:32:10.344474 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:10.344417 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9t68r" podStartSLOduration=1.729253047 podStartE2EDuration="4.344399606s" totalImagesPullingTime="2.615146559s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:32:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:10.343541995 +0000 UTC m=+53.760246455" watchObservedRunningTime="2026-10-31 12:32:10.344399606 +0000 UTC m=+53.761104070" Oct 31 12:32:10.366897 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:10.366866 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-227cc" Oct 31 12:32:10.426608 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:10.426515 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5gdlb_26913ad4-6613-4f3f-9819-9be238088c0a/dns-node-resolver/0.log" Oct 31 12:32:10.667122 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:10.667088 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-227cc"] Oct 31 12:32:11.306442 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:11.306352 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-227cc" event={"ID":"ddebd541-4d2f-4861-9dd0-0ee66b03aaf2","Type":"ContainerStarted","Data":"86e20428c0c192b9f1bdf2e62481619c797e84aeffec2ec17676284c65292fe7"} Oct 31 12:32:11.308144 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:11.308091 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pwtgm" event={"ID":"45870e34-5e37-443a-826b-81c9c031051c","Type":"ContainerStarted","Data":"7bfe97ce87bf7f8ad1581392e2af20e55f4570c194e6832060ca46287fbeb7b2"} Oct 31 12:32:11.330084 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:11.330030 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pwtgm" podStartSLOduration=1.468318954 podStartE2EDuration="5.330015907s" totalImagesPullingTime="3.861696953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:32:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:11.329913197 +0000 UTC m=+54.746617655" watchObservedRunningTime="2026-10-31 12:32:11.330015907 +0000 UTC m=+54.746720367" Oct 31 12:32:12.039282 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:12.039255 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tk5vl_14dc1e6d-70e1-4ba5-bffb-d9299433756d/node-ca/0.log" Oct 31 12:32:12.313982 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:12.313713 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-227cc" event={"ID":"ddebd541-4d2f-4861-9dd0-0ee66b03aaf2","Type":"ContainerStarted","Data":"4e8a4d6a050daed2df0807898b546d02f360eef472dfeae9b967f99d1ece19a3"} Oct 31 12:32:12.313982 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:12.313798 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-227cc" event={"ID":"ddebd541-4d2f-4861-9dd0-0ee66b03aaf2","Type":"ContainerStarted","Data":"2fb444c6e9ab856ed20feb7cd74248f336def860695582101d58fb5bb3fa79cf"} Oct 31 12:32:12.333220 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:12.333164 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-227cc" podStartSLOduration=1.038989695 podStartE2EDuration="2.33314949s" totalImagesPullingTime="1.294159795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:32:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:12.332671125 +0000 UTC m=+55.749375584" watchObservedRunningTime="2026-10-31 12:32:12.33314949 +0000 UTC m=+55.749853970" Oct 31 12:32:12.625815 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:12.625738 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pwtgm_45870e34-5e37-443a-826b-81c9c031051c/serve-healthcheck-canary/0.log" Oct 31 12:32:13.318500 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:13.318463 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57b586f5c7-hbnwz" event={"ID":"8c91a35c-7933-4399-b7ff-4c1226d72d20","Type":"ContainerStarted","Data":"51509f6a0062fbd3559a7d18bd83266e30fc7b243607755ee7b545ba47b32ff9"} Oct 31 12:32:13.340382 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:13.340332 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-57b586f5c7-hbnwz" podStartSLOduration=0.890975 podStartE2EDuration="4.340317373s" totalImagesPullingTime="3.449342373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:32:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:13.338830163 +0000 UTC m=+56.755534622" watchObservedRunningTime="2026-10-31 12:32:13.340317373 +0000 UTC m=+56.757021832" Oct 31 12:32:14.445216 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:14.445180 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gqmhl"] Oct 31 12:32:14.448392 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:14.448374 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:14.487167 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:14.487140 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 12:32:14.487309 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:14.487275 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zhwtd\"" Oct 31 12:32:14.487309 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:14.487273 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 12:32:14.537997 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:14.537967 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 12:32:14.585629 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:14.585579 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4b6af1c5-6045-4e4c-aa87-2473d19bd000-root\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:14.585629 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:14.585635 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b6af1c5-6045-4e4c-aa87-2473d19bd000-sys\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:14.585840 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:14.585725 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4b6af1c5-6045-4e4c-aa87-2473d19bd000-node-exporter-accelerators-collector-config\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:14.585840 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:14.585766 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4b6af1c5-6045-4e4c-aa87-2473d19bd000-node-exporter-wtmp\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:14.585840 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:14.585783 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4b6af1c5-6045-4e4c-aa87-2473d19bd000-node-exporter-textfile\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:14.585840 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:14.585827 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b6af1c5-6045-4e4c-aa87-2473d19bd000-metrics-client-ca\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:14.585955 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:14.585928 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4b6af1c5-6045-4e4c-aa87-2473d19bd000-node-exporter-tls\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:14.585955 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:14.585951 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b6af1c5-6045-4e4c-aa87-2473d19bd000-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:14.586038 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:14.585975 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rprnd\" (UniqueName: \"kubernetes.io/projected/4b6af1c5-6045-4e4c-aa87-2473d19bd000-kube-api-access-rprnd\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:14.686952 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:14.686910 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b6af1c5-6045-4e4c-aa87-2473d19bd000-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:14.686952 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:14.686956 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rprnd\" (UniqueName: \"kubernetes.io/projected/4b6af1c5-6045-4e4c-aa87-2473d19bd000-kube-api-access-rprnd\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:14.687199 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:14.687105 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4b6af1c5-6045-4e4c-aa87-2473d19bd000-root\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:14.687199 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:14.687146 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b6af1c5-6045-4e4c-aa87-2473d19bd000-sys\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:14.687199 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:14.687162 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4b6af1c5-6045-4e4c-aa87-2473d19bd000-node-exporter-accelerators-collector-config\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:14.687199 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:14.687185 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4b6af1c5-6045-4e4c-aa87-2473d19bd000-node-exporter-wtmp\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:14.687391 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:14.687208 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4b6af1c5-6045-4e4c-aa87-2473d19bd000-node-exporter-textfile\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:14.687391 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:14.687210 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4b6af1c5-6045-4e4c-aa87-2473d19bd000-root\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:14.687391 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:14.687229 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b6af1c5-6045-4e4c-aa87-2473d19bd000-metrics-client-ca\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:14.687391 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:14.687233 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b6af1c5-6045-4e4c-aa87-2473d19bd000-sys\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:14.687391 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:14.687313 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4b6af1c5-6045-4e4c-aa87-2473d19bd000-node-exporter-tls\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:14.687391 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:14.687371 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4b6af1c5-6045-4e4c-aa87-2473d19bd000-node-exporter-wtmp\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:14.687586 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:14.687533 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4b6af1c5-6045-4e4c-aa87-2473d19bd000-node-exporter-textfile\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:14.687909 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:14.687885 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4b6af1c5-6045-4e4c-aa87-2473d19bd000-node-exporter-accelerators-collector-config\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:14.688113 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:14.688090 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b6af1c5-6045-4e4c-aa87-2473d19bd000-metrics-client-ca\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:14.689508 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:14.689487 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b6af1c5-6045-4e4c-aa87-2473d19bd000-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:14.689612 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:14.689597 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4b6af1c5-6045-4e4c-aa87-2473d19bd000-node-exporter-tls\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:14.699073 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:14.699025 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rprnd\" (UniqueName: \"kubernetes.io/projected/4b6af1c5-6045-4e4c-aa87-2473d19bd000-kube-api-access-rprnd\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:14.764161 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:14.764128 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:14.775173 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:32:14.775147 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b6af1c5_6045_4e4c_aa87_2473d19bd000.slice/crio-f763f3a95d574aac7ca59fd0a9b3957bb56261dce7e237013b93f0c6cd662978 WatchSource:0}: Error finding container f763f3a95d574aac7ca59fd0a9b3957bb56261dce7e237013b93f0c6cd662978: Status 404 returned error can't find the container with id f763f3a95d574aac7ca59fd0a9b3957bb56261dce7e237013b93f0c6cd662978 Oct 31 12:32:15.244195 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:15.244166 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:15.326067 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:15.326030 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gqmhl" event={"ID":"4b6af1c5-6045-4e4c-aa87-2473d19bd000","Type":"ContainerStarted","Data":"f763f3a95d574aac7ca59fd0a9b3957bb56261dce7e237013b93f0c6cd662978"} Oct 31 12:32:16.331079 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:16.331044 2577 generic.go:356] "Generic (PLEG): container finished" podID="4b6af1c5-6045-4e4c-aa87-2473d19bd000" containerID="98658f29d2c7d3a00b3307cbb1941b286c18083cc24ffdb1212124d9ecd17777" exitCode=0 Oct 31 12:32:16.331435 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:16.331104 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gqmhl" event={"ID":"4b6af1c5-6045-4e4c-aa87-2473d19bd000","Type":"ContainerDied","Data":"98658f29d2c7d3a00b3307cbb1941b286c18083cc24ffdb1212124d9ecd17777"} Oct 31 12:32:17.336203 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:17.336163 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gqmhl" event={"ID":"4b6af1c5-6045-4e4c-aa87-2473d19bd000","Type":"ContainerStarted","Data":"4e2b1fd92540a894604c7fc505af2877d076c926b76ed99f0dfab6ae44145b98"} Oct 31 12:32:17.336701 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:17.336213 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gqmhl" event={"ID":"4b6af1c5-6045-4e4c-aa87-2473d19bd000","Type":"ContainerStarted","Data":"e7c9b10d242b6bb6fa47222f5172f9fbabdfe1cb59996170f1e7dc27e84a17b4"} Oct 31 12:32:17.362868 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:17.362812 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gqmhl" podStartSLOduration=2.362376017 podStartE2EDuration="3.362799482s" totalImagesPullingTime="922.942399ms" totalInitContainerRuntime="77.481066ms" isStatefulPod=true podCreationTimestamp="2026-10-31 12:32:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:17.361272559 +0000 UTC m=+60.777977018" watchObservedRunningTime="2026-10-31 12:32:17.362799482 +0000 UTC m=+60.779503941" Oct 31 12:32:18.932523 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:18.932488 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-847b6d5877-sxkd8"] Oct 31 12:32:18.936743 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:18.936720 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:18.946480 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:18.946456 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 12:32:18.946480 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:18.946456 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 31 12:32:18.950378 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:18.950358 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-847b6d5877-sxkd8"] Oct 31 12:32:18.965996 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:18.965969 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 31 12:32:18.967670 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:18.966527 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 31 12:32:18.967670 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:18.966564 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-vttbs\"" Oct 31 12:32:18.967670 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:18.966671 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-6k99scgtm5t5r\"" Oct 31 12:32:19.125937 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:19.125899 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/25a02fd6-8ab0-4cca-94cd-d4082f874170-metrics-server-audit-profiles\") pod \"metrics-server-847b6d5877-sxkd8\" (UID: \"25a02fd6-8ab0-4cca-94cd-d4082f874170\") " pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:19.126126 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:19.125956 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/25a02fd6-8ab0-4cca-94cd-d4082f874170-client-ca-bundle\") pod \"metrics-server-847b6d5877-sxkd8\" (UID: \"25a02fd6-8ab0-4cca-94cd-d4082f874170\") " pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:19.126126 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:19.126029 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/25a02fd6-8ab0-4cca-94cd-d4082f874170-secret-metrics-server-tls\") pod \"metrics-server-847b6d5877-sxkd8\" (UID: \"25a02fd6-8ab0-4cca-94cd-d4082f874170\") " pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:19.126126 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:19.126105 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25a02fd6-8ab0-4cca-94cd-d4082f874170-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-847b6d5877-sxkd8\" (UID: \"25a02fd6-8ab0-4cca-94cd-d4082f874170\") " pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:19.126237 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:19.126163 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srn9m\" (UniqueName: \"kubernetes.io/projected/25a02fd6-8ab0-4cca-94cd-d4082f874170-kube-api-access-srn9m\") pod \"metrics-server-847b6d5877-sxkd8\" (UID: \"25a02fd6-8ab0-4cca-94cd-d4082f874170\") " pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:19.126237 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:19.126204 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/25a02fd6-8ab0-4cca-94cd-d4082f874170-secret-metrics-server-client-certs\") pod \"metrics-server-847b6d5877-sxkd8\" (UID: \"25a02fd6-8ab0-4cca-94cd-d4082f874170\") " pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:19.126237 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:19.126227 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/25a02fd6-8ab0-4cca-94cd-d4082f874170-audit-log\") pod \"metrics-server-847b6d5877-sxkd8\" (UID: \"25a02fd6-8ab0-4cca-94cd-d4082f874170\") " pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:19.227383 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:19.227298 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/25a02fd6-8ab0-4cca-94cd-d4082f874170-metrics-server-audit-profiles\") pod \"metrics-server-847b6d5877-sxkd8\" (UID: \"25a02fd6-8ab0-4cca-94cd-d4082f874170\") " pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:19.227383 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:19.227345 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/25a02fd6-8ab0-4cca-94cd-d4082f874170-client-ca-bundle\") pod \"metrics-server-847b6d5877-sxkd8\" (UID: \"25a02fd6-8ab0-4cca-94cd-d4082f874170\") " pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:19.227383 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:19.227364 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/25a02fd6-8ab0-4cca-94cd-d4082f874170-secret-metrics-server-tls\") pod \"metrics-server-847b6d5877-sxkd8\" (UID: \"25a02fd6-8ab0-4cca-94cd-d4082f874170\") " pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:19.227604 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:19.227417 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25a02fd6-8ab0-4cca-94cd-d4082f874170-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-847b6d5877-sxkd8\" (UID: \"25a02fd6-8ab0-4cca-94cd-d4082f874170\") " pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:19.227604 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:19.227439 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srn9m\" (UniqueName: \"kubernetes.io/projected/25a02fd6-8ab0-4cca-94cd-d4082f874170-kube-api-access-srn9m\") pod \"metrics-server-847b6d5877-sxkd8\" (UID: \"25a02fd6-8ab0-4cca-94cd-d4082f874170\") " pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:19.227604 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:19.227478 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/25a02fd6-8ab0-4cca-94cd-d4082f874170-secret-metrics-server-client-certs\") pod \"metrics-server-847b6d5877-sxkd8\" (UID: \"25a02fd6-8ab0-4cca-94cd-d4082f874170\") " pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:19.227604 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:19.227506 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/25a02fd6-8ab0-4cca-94cd-d4082f874170-audit-log\") pod \"metrics-server-847b6d5877-sxkd8\" (UID: \"25a02fd6-8ab0-4cca-94cd-d4082f874170\") " pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:19.228055 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:19.228026 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/25a02fd6-8ab0-4cca-94cd-d4082f874170-audit-log\") pod \"metrics-server-847b6d5877-sxkd8\" (UID: \"25a02fd6-8ab0-4cca-94cd-d4082f874170\") " pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:19.228148 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:19.228119 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25a02fd6-8ab0-4cca-94cd-d4082f874170-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-847b6d5877-sxkd8\" (UID: \"25a02fd6-8ab0-4cca-94cd-d4082f874170\") " pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:19.228764 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:19.228747 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/25a02fd6-8ab0-4cca-94cd-d4082f874170-metrics-server-audit-profiles\") pod \"metrics-server-847b6d5877-sxkd8\" (UID: \"25a02fd6-8ab0-4cca-94cd-d4082f874170\") " pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:19.230096 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:19.230071 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/25a02fd6-8ab0-4cca-94cd-d4082f874170-secret-metrics-server-client-certs\") pod \"metrics-server-847b6d5877-sxkd8\" (UID: \"25a02fd6-8ab0-4cca-94cd-d4082f874170\") " pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:19.230399 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:19.230382 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/25a02fd6-8ab0-4cca-94cd-d4082f874170-client-ca-bundle\") pod \"metrics-server-847b6d5877-sxkd8\" (UID: \"25a02fd6-8ab0-4cca-94cd-d4082f874170\") " pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:19.230458 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:19.230421 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/25a02fd6-8ab0-4cca-94cd-d4082f874170-secret-metrics-server-tls\") pod \"metrics-server-847b6d5877-sxkd8\" (UID: \"25a02fd6-8ab0-4cca-94cd-d4082f874170\") " pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:19.237199 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:19.237177 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-zbjvn"] Oct 31 12:32:19.241217 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:19.241201 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-zbjvn" Oct 31 12:32:19.243821 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:19.243801 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srn9m\" (UniqueName: \"kubernetes.io/projected/25a02fd6-8ab0-4cca-94cd-d4082f874170-kube-api-access-srn9m\") pod \"metrics-server-847b6d5877-sxkd8\" (UID: \"25a02fd6-8ab0-4cca-94cd-d4082f874170\") " pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:19.247692 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:19.247630 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:19.248674 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:19.248656 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 31 12:32:19.248764 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:19.248692 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-5jsz7\"" Oct 31 12:32:19.260558 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:19.260533 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-zbjvn"] Oct 31 12:32:19.408760 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:19.408724 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:19.408760 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:19.408773 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:19.413568 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:19.413546 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:19.429022 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:19.428994 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3015bfb0-80c8-4c01-9053-85e5ff2318fb-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-zbjvn\" (UID: \"3015bfb0-80c8-4c01-9053-85e5ff2318fb\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zbjvn" Oct 31 12:32:19.429151 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:19.429024 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d92p\" (UniqueName: \"kubernetes.io/projected/3015bfb0-80c8-4c01-9053-85e5ff2318fb-kube-api-access-5d92p\") pod \"monitoring-plugin-77f4f76594-zbjvn\" (UID: \"3015bfb0-80c8-4c01-9053-85e5ff2318fb\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zbjvn" Oct 31 12:32:19.526315 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:19.526283 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-847b6d5877-sxkd8"] Oct 31 12:32:19.527342 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:32:19.527300 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25a02fd6_8ab0_4cca_94cd_d4082f874170.slice/crio-98c00b91a1a67a918bfc7bdefc29cf5f033528396d20cfa2be189cc550975729 WatchSource:0}: Error finding container 98c00b91a1a67a918bfc7bdefc29cf5f033528396d20cfa2be189cc550975729: Status 404 returned error can't find the container with id 98c00b91a1a67a918bfc7bdefc29cf5f033528396d20cfa2be189cc550975729 Oct 31 12:32:19.529766 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:19.529746 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3015bfb0-80c8-4c01-9053-85e5ff2318fb-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-zbjvn\" (UID: \"3015bfb0-80c8-4c01-9053-85e5ff2318fb\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zbjvn" Oct 31 12:32:19.529860 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:19.529786 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d92p\" (UniqueName: \"kubernetes.io/projected/3015bfb0-80c8-4c01-9053-85e5ff2318fb-kube-api-access-5d92p\") pod \"monitoring-plugin-77f4f76594-zbjvn\" (UID: \"3015bfb0-80c8-4c01-9053-85e5ff2318fb\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zbjvn" Oct 31 12:32:19.532246 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:19.532229 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3015bfb0-80c8-4c01-9053-85e5ff2318fb-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-zbjvn\" (UID: \"3015bfb0-80c8-4c01-9053-85e5ff2318fb\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zbjvn" Oct 31 12:32:19.543662 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:19.543625 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d92p\" (UniqueName: \"kubernetes.io/projected/3015bfb0-80c8-4c01-9053-85e5ff2318fb-kube-api-access-5d92p\") pod \"monitoring-plugin-77f4f76594-zbjvn\" (UID: \"3015bfb0-80c8-4c01-9053-85e5ff2318fb\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zbjvn" Oct 31 12:32:19.557526 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:19.557508 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-zbjvn" Oct 31 12:32:19.831120 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:19.831048 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-zbjvn"] Oct 31 12:32:19.834078 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:32:19.834053 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3015bfb0_80c8_4c01_9053_85e5ff2318fb.slice/crio-0a1787850d6b84588d61e49e4b5d640845f0c97caebe94b71026f54cd02be1a4 WatchSource:0}: Error finding container 0a1787850d6b84588d61e49e4b5d640845f0c97caebe94b71026f54cd02be1a4: Status 404 returned error can't find the container with id 0a1787850d6b84588d61e49e4b5d640845f0c97caebe94b71026f54cd02be1a4 Oct 31 12:32:20.311268 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:20.311238 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bdwh8" Oct 31 12:32:20.352589 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:20.352501 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-zbjvn" event={"ID":"3015bfb0-80c8-4c01-9053-85e5ff2318fb","Type":"ContainerStarted","Data":"0a1787850d6b84588d61e49e4b5d640845f0c97caebe94b71026f54cd02be1a4"} Oct 31 12:32:20.359001 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:20.355080 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" event={"ID":"25a02fd6-8ab0-4cca-94cd-d4082f874170","Type":"ContainerStarted","Data":"98c00b91a1a67a918bfc7bdefc29cf5f033528396d20cfa2be189cc550975729"} Oct 31 12:32:20.360624 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:20.360435 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:21.359327 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:21.359284 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" event={"ID":"25a02fd6-8ab0-4cca-94cd-d4082f874170","Type":"ContainerStarted","Data":"52643e92cafacae60ac74379b087a40a84e3660404bb4eddecac128cb516217a"} Oct 31 12:32:21.394387 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:21.394313 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" podStartSLOduration=1.724688832 podStartE2EDuration="3.394293792s" totalImagesPullingTime="1.66960496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:32:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:21.391237899 +0000 UTC m=+64.807942373" watchObservedRunningTime="2026-10-31 12:32:21.394293792 +0000 UTC m=+64.810998251" Oct 31 12:32:22.364505 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:22.364414 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-zbjvn" event={"ID":"3015bfb0-80c8-4c01-9053-85e5ff2318fb","Type":"ContainerStarted","Data":"68001961b6f702a0ea9ace5d941f373d964bbd7287239096f8a87f2e3510527a"} Oct 31 12:32:22.364951 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:22.364760 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-zbjvn" Oct 31 12:32:22.369605 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:22.369582 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-zbjvn" Oct 31 12:32:22.388222 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:22.388176 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-zbjvn" podStartSLOduration=1.120654741 podStartE2EDuration="3.388163766s" totalImagesPullingTime="2.267509025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:32:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:22.387685233 +0000 UTC m=+65.804389684" watchObservedRunningTime="2026-10-31 12:32:22.388163766 +0000 UTC m=+65.804868224" Oct 31 12:32:22.757551 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:22.757510 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf2e0b3-d31b-44ca-934c-3710f4262779-metrics-certs\") pod \"network-metrics-daemon-8nbvz\" (UID: \"bbf2e0b3-d31b-44ca-934c-3710f4262779\") " pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:32:22.757551 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:22.757554 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a-original-pull-secret\") pod \"global-pull-secret-syncer-wgrhs\" (UID: \"ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a\") " pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:32:22.761139 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:22.761121 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 12:32:22.762397 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:22.762383 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 12:32:22.770903 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:22.770878 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf2e0b3-d31b-44ca-934c-3710f4262779-metrics-certs\") pod \"network-metrics-daemon-8nbvz\" (UID: \"bbf2e0b3-d31b-44ca-934c-3710f4262779\") " pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:32:22.771004 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:22.770991 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a-original-pull-secret\") pod \"global-pull-secret-syncer-wgrhs\" (UID: \"ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a\") " pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:32:22.938784 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:22.938752 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l77g8\"" Oct 31 12:32:22.945983 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:22.945957 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:32:22.949989 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:22.949973 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zbx6v\"" Oct 31 12:32:22.957793 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:22.957760 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:32:22.959138 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:22.959115 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph9jm\" (UniqueName: \"kubernetes.io/projected/02c6d8f5-97ff-41e0-8e38-71e1b0c87f54-kube-api-access-ph9jm\") pod \"network-check-target-bgcqc\" (UID: \"02c6d8f5-97ff-41e0-8e38-71e1b0c87f54\") " pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:32:22.962177 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:22.962140 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 12:32:22.972108 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:22.972088 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 12:32:22.982743 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:22.982716 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph9jm\" (UniqueName: \"kubernetes.io/projected/02c6d8f5-97ff-41e0-8e38-71e1b0c87f54-kube-api-access-ph9jm\") pod \"network-check-target-bgcqc\" (UID: \"02c6d8f5-97ff-41e0-8e38-71e1b0c87f54\") " pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:32:23.219241 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:23.219211 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8nbvz"] Oct 31 12:32:23.222404 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:32:23.222373 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf2e0b3_d31b_44ca_934c_3710f4262779.slice/crio-9af65c148112ebb9dd1268dbe8c909387f2ca55b28da2b69649c5428e5be0e76 WatchSource:0}: Error finding container 9af65c148112ebb9dd1268dbe8c909387f2ca55b28da2b69649c5428e5be0e76: Status 404 returned error can't find the container with id 9af65c148112ebb9dd1268dbe8c909387f2ca55b28da2b69649c5428e5be0e76 Oct 31 12:32:23.238915 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:23.238888 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wgrhs"] Oct 31 12:32:23.241731 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:32:23.241701 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded7130e5_e8ea_4ade_97ba_2c7b7880aa7a.slice/crio-b391fcf82f11c31b74103e72e0102b9d81913ae13eb810d71671a5954464aad3 WatchSource:0}: Error finding container b391fcf82f11c31b74103e72e0102b9d81913ae13eb810d71671a5954464aad3: Status 404 returned error can't find the container with id b391fcf82f11c31b74103e72e0102b9d81913ae13eb810d71671a5954464aad3 Oct 31 12:32:23.244424 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:23.244342 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-l5d6v\"" Oct 31 12:32:23.251834 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:23.251815 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:32:23.368757 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:23.368729 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wgrhs" event={"ID":"ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a","Type":"ContainerStarted","Data":"b391fcf82f11c31b74103e72e0102b9d81913ae13eb810d71671a5954464aad3"} Oct 31 12:32:23.369969 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:23.369947 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8nbvz" event={"ID":"bbf2e0b3-d31b-44ca-934c-3710f4262779","Type":"ContainerStarted","Data":"9af65c148112ebb9dd1268dbe8c909387f2ca55b28da2b69649c5428e5be0e76"} Oct 31 12:32:23.523722 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:23.523596 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bgcqc"] Oct 31 12:32:23.526448 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:32:23.526418 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c6d8f5_97ff_41e0_8e38_71e1b0c87f54.slice/crio-2449fb987accd7f8129f3ad9d4ad0d8aeb202aea937d539aca589a09d925778e WatchSource:0}: Error finding container 2449fb987accd7f8129f3ad9d4ad0d8aeb202aea937d539aca589a09d925778e: Status 404 returned error can't find the container with id 2449fb987accd7f8129f3ad9d4ad0d8aeb202aea937d539aca589a09d925778e Oct 31 12:32:24.375344 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:24.375154 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bgcqc" event={"ID":"02c6d8f5-97ff-41e0-8e38-71e1b0c87f54","Type":"ContainerStarted","Data":"2449fb987accd7f8129f3ad9d4ad0d8aeb202aea937d539aca589a09d925778e"} Oct 31 12:32:25.382590 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:25.382550 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8nbvz" event={"ID":"bbf2e0b3-d31b-44ca-934c-3710f4262779","Type":"ContainerStarted","Data":"d01a171d1bf2a75c916e2fa16dfd417e1500b1e8a1b65bfb408d82b3d2dcd84b"} Oct 31 12:32:25.383014 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:25.382608 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8nbvz" event={"ID":"bbf2e0b3-d31b-44ca-934c-3710f4262779","Type":"ContainerStarted","Data":"ac315e8f345e6211d29fa5f0ca192ca80b5fc9c25a7751d03c48e53406748b82"} Oct 31 12:32:25.404754 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:25.404705 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8nbvz" podStartSLOduration=66.927594077 podStartE2EDuration="1m8.40469134s" totalImagesPullingTime="1.477097263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:31:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:25.404205788 +0000 UTC m=+68.820910260" watchObservedRunningTime="2026-10-31 12:32:25.40469134 +0000 UTC m=+68.821395800" Oct 31 12:32:26.568934 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:26.568903 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b4df7846b-2pjqf"] Oct 31 12:32:26.572312 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:26.572277 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:26.592474 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:26.592445 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b4df7846b-2pjqf"] Oct 31 12:32:26.597299 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:26.597269 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 31 12:32:26.686278 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:26.686239 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpx9x\" (UniqueName: \"kubernetes.io/projected/0f343e95-e397-49aa-b023-98337d3be3ff-kube-api-access-kpx9x\") pod \"console-5b4df7846b-2pjqf\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:26.686447 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:26.686287 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0f343e95-e397-49aa-b023-98337d3be3ff-oauth-serving-cert\") pod \"console-5b4df7846b-2pjqf\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:26.686447 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:26.686312 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f343e95-e397-49aa-b023-98337d3be3ff-console-serving-cert\") pod \"console-5b4df7846b-2pjqf\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:26.686447 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:26.686425 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0f343e95-e397-49aa-b023-98337d3be3ff-console-config\") pod \"console-5b4df7846b-2pjqf\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:26.686599 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:26.686453 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0f343e95-e397-49aa-b023-98337d3be3ff-console-oauth-config\") pod \"console-5b4df7846b-2pjqf\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:26.686599 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:26.686486 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f343e95-e397-49aa-b023-98337d3be3ff-trusted-ca-bundle\") pod \"console-5b4df7846b-2pjqf\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:26.686707 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:26.686599 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0f343e95-e397-49aa-b023-98337d3be3ff-service-ca\") pod \"console-5b4df7846b-2pjqf\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:26.686707 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:26.686633 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f343e95-e397-49aa-b023-98337d3be3ff-tmp\") pod \"console-5b4df7846b-2pjqf\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:26.790248 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:26.790209 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f343e95-e397-49aa-b023-98337d3be3ff-console-serving-cert\") pod \"console-5b4df7846b-2pjqf\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:26.790402 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:26.790266 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0f343e95-e397-49aa-b023-98337d3be3ff-console-config\") pod \"console-5b4df7846b-2pjqf\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:26.790402 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:26.790299 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0f343e95-e397-49aa-b023-98337d3be3ff-console-oauth-config\") pod \"console-5b4df7846b-2pjqf\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:26.790402 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:26.790334 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f343e95-e397-49aa-b023-98337d3be3ff-trusted-ca-bundle\") pod \"console-5b4df7846b-2pjqf\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:26.790402 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:26.790373 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0f343e95-e397-49aa-b023-98337d3be3ff-service-ca\") pod \"console-5b4df7846b-2pjqf\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:26.790402 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:26.790396 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f343e95-e397-49aa-b023-98337d3be3ff-tmp\") pod \"console-5b4df7846b-2pjqf\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:26.790618 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:26.790426 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpx9x\" (UniqueName: \"kubernetes.io/projected/0f343e95-e397-49aa-b023-98337d3be3ff-kube-api-access-kpx9x\") pod \"console-5b4df7846b-2pjqf\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:26.790618 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:26.790456 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0f343e95-e397-49aa-b023-98337d3be3ff-oauth-serving-cert\") pod \"console-5b4df7846b-2pjqf\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:26.791174 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:26.791149 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0f343e95-e397-49aa-b023-98337d3be3ff-oauth-serving-cert\") pod \"console-5b4df7846b-2pjqf\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:26.793121 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:26.793092 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0f343e95-e397-49aa-b023-98337d3be3ff-console-config\") pod \"console-5b4df7846b-2pjqf\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:26.794469 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:26.794446 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f343e95-e397-49aa-b023-98337d3be3ff-trusted-ca-bundle\") pod \"console-5b4df7846b-2pjqf\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:26.795165 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:26.794858 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0f343e95-e397-49aa-b023-98337d3be3ff-service-ca\") pod \"console-5b4df7846b-2pjqf\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:26.795165 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:26.794917 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f343e95-e397-49aa-b023-98337d3be3ff-tmp\") pod \"console-5b4df7846b-2pjqf\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:26.795365 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:26.795340 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f343e95-e397-49aa-b023-98337d3be3ff-console-serving-cert\") pod \"console-5b4df7846b-2pjqf\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:26.803841 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:26.803806 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0f343e95-e397-49aa-b023-98337d3be3ff-console-oauth-config\") pod \"console-5b4df7846b-2pjqf\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:26.808340 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:26.808287 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpx9x\" (UniqueName: \"kubernetes.io/projected/0f343e95-e397-49aa-b023-98337d3be3ff-kube-api-access-kpx9x\") pod \"console-5b4df7846b-2pjqf\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:26.892708 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:26.888849 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:27.193345 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:27.193241 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b4df7846b-2pjqf"] Oct 31 12:32:27.208828 ip-10-0-131-157 kubenswrapper[2577]: W1031 12:32:27.208671 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f343e95_e397_49aa_b023_98337d3be3ff.slice/crio-4334c12d573521f95b214022be018ae2961732b1ee6e21a1d7c07ef1b08b3dce WatchSource:0}: Error finding container 4334c12d573521f95b214022be018ae2961732b1ee6e21a1d7c07ef1b08b3dce: Status 404 returned error can't find the container with id 4334c12d573521f95b214022be018ae2961732b1ee6e21a1d7c07ef1b08b3dce Oct 31 12:32:27.393985 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:27.393932 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b4df7846b-2pjqf" event={"ID":"0f343e95-e397-49aa-b023-98337d3be3ff","Type":"ContainerStarted","Data":"7f97a460b9fdafec97b3b99dc1dc181965efb1f6ea45c7ca086f42e3aa0bf8c0"} Oct 31 12:32:27.393985 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:27.393996 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b4df7846b-2pjqf" event={"ID":"0f343e95-e397-49aa-b023-98337d3be3ff","Type":"ContainerStarted","Data":"4334c12d573521f95b214022be018ae2961732b1ee6e21a1d7c07ef1b08b3dce"} Oct 31 12:32:27.417171 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:27.417112 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5b4df7846b-2pjqf" podStartSLOduration=1.4170947 podStartE2EDuration="1.4170947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:27.415484728 +0000 UTC m=+70.832189188" watchObservedRunningTime="2026-10-31 12:32:27.4170947 +0000 UTC m=+70.833799159" Oct 31 12:32:28.399249 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:28.399209 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bgcqc" event={"ID":"02c6d8f5-97ff-41e0-8e38-71e1b0c87f54","Type":"ContainerStarted","Data":"ea4594f246be291c6391893ecfeb53ac74875b98360e8a9073229463951d4a44"} Oct 31 12:32:28.417228 ip-10-0-131-157 kubenswrapper[2577]: I1031 12:32:28.417120 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bgcqc" podStartSLOduration=66.778730608 podStartE2EDuration="1m11.417100762s" totalImagesPullingTime="4.638370154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:31:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:28.416977685 +0000 UTC m=+71.833682145" watchObservedRunningTime="2026-10-31 12:32:28.417100762 +0000 UTC m=+71.833805223" Oct 31 12:32:29.229953 ip-10-0-131-157 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 12:32:29.246075 ip-10-0-131-157 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 12:32:29.246250 ip-10-0-131-157 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 12:32:29.246299 ip-10-0-131-157 systemd[1]: kubelet.service: Consumed 3.826s CPU time. Oct 31 12:32:29.259447 ip-10-0-131-157 systemd[1]: Starting Kubernetes Kubelet... Oct 31 12:32:29.322892 ip-10-0-131-157 kubenswrapper[5693]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 12:32:29.322892 ip-10-0-131-157 kubenswrapper[5693]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 12:32:29.322892 ip-10-0-131-157 kubenswrapper[5693]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 12:32:29.322892 ip-10-0-131-157 kubenswrapper[5693]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 12:32:29.324974 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.324887 5693 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:32:29.324974 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.324973 5693 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:32:29.324974 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.324976 5693 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:32:29.325080 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.324981 5693 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:32:29.325080 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.324984 5693 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:32:29.325080 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.324987 5693 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:32:29.325080 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.324990 5693 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:32:29.325080 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.324992 5693 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:32:29.325080 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.324995 5693 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:32:29.325080 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.324997 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:32:29.325080 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325000 5693 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:32:29.325080 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325003 5693 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:32:29.325080 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325005 5693 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:32:29.325080 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325008 5693 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:32:29.325080 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325010 5693 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:32:29.325080 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325013 5693 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:32:29.325080 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325015 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:32:29.325080 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325018 5693 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:32:29.325080 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325021 5693 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:32:29.325080 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325024 5693 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:32:29.325080 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325026 5693 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:32:29.325080 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325033 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:32:29.325080 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325035 5693 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:32:29.325560 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325038 5693 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:32:29.325560 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325041 5693 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:32:29.325560 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325044 5693 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:32:29.325560 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325046 5693 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:32:29.325560 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325049 5693 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:32:29.325560 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325052 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:32:29.325560 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325054 5693 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:32:29.325560 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325057 5693 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:32:29.325560 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325059 5693 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:32:29.325560 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325062 5693 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:32:29.325560 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325064 5693 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:32:29.325560 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325067 5693 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:32:29.325560 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325069 5693 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:32:29.325560 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325072 5693 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:32:29.325560 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325075 5693 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:32:29.325560 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325077 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:32:29.325560 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325080 5693 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:32:29.325560 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325082 5693 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:32:29.325560 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325085 5693 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:32:29.325560 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325088 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:32:29.326062 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325094 5693 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:32:29.326062 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325096 5693 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:32:29.326062 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325099 5693 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:32:29.326062 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325101 5693 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:32:29.326062 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325104 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:32:29.326062 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325106 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:32:29.326062 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325109 5693 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:32:29.326062 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325112 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:32:29.326062 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325114 5693 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:32:29.326062 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325117 5693 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:32:29.326062 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325119 5693 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:32:29.326062 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325122 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:32:29.326062 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325125 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:32:29.326062 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325128 5693 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:32:29.326062 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325131 5693 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:32:29.326062 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325133 5693 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:32:29.326062 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325135 5693 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:32:29.326062 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325138 5693 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:32:29.326062 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325140 5693 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:32:29.326532 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325143 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:32:29.326532 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325145 5693 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:32:29.326532 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325148 5693 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:32:29.326532 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325151 5693 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:32:29.326532 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325153 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:32:29.326532 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325156 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:32:29.326532 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325159 5693 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:32:29.326532 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325161 5693 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:32:29.326532 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325164 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:32:29.326532 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325166 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:32:29.326532 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325168 5693 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:32:29.326532 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325171 5693 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:32:29.326532 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325173 5693 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:32:29.326532 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325177 5693 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:32:29.326532 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325181 5693 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:32:29.326532 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325183 5693 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:32:29.326532 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325186 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:32:29.326532 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325189 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:32:29.326532 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325191 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:32:29.327066 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325195 5693 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:32:29.327066 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325200 5693 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:32:29.327066 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325204 5693 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:32:29.327066 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325208 5693 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:32:29.327066 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325212 5693 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:32:29.327066 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325215 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:32:29.327066 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325217 5693 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:32:29.327066 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325220 5693 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:32:29.327066 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325223 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:32:29.327066 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325225 5693 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:32:29.327066 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325228 5693 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:32:29.327066 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325230 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:32:29.327066 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325233 5693 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:32:29.327066 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325236 5693 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:32:29.327066 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325238 5693 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:32:29.327066 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325241 5693 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:32:29.327066 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325244 5693 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:32:29.327066 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325247 5693 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:32:29.327066 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325249 5693 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:32:29.327535 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325252 5693 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:32:29.327535 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325254 5693 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:32:29.327535 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325257 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:32:29.327535 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325260 5693 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:32:29.327535 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325263 5693 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:32:29.327535 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.325265 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:32:29.327535 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326055 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:32:29.327535 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326065 5693 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:32:29.327535 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326068 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:32:29.327535 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326071 5693 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:32:29.327535 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326073 5693 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:32:29.327535 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326077 5693 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:32:29.327535 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326079 5693 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:32:29.327535 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326082 5693 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:32:29.327535 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326084 5693 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:32:29.327535 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326087 5693 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:32:29.327535 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326090 5693 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:32:29.327535 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326092 5693 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:32:29.327535 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326095 5693 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:32:29.328022 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326097 5693 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:32:29.328022 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326101 5693 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:32:29.328022 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326104 5693 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:32:29.328022 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326106 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:32:29.328022 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326109 5693 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:32:29.328022 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326112 5693 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:32:29.328022 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326114 5693 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:32:29.328022 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326117 5693 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:32:29.328022 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326120 5693 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:32:29.328022 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326122 5693 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:32:29.328022 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326125 5693 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:32:29.328022 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326127 5693 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:32:29.328022 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326130 5693 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:32:29.328022 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326133 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:32:29.328022 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326136 5693 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:32:29.328022 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326139 5693 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:32:29.328022 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326141 5693 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:32:29.328022 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326144 5693 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:32:29.328022 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326146 5693 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:32:29.328022 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326149 5693 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:32:29.328022 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326153 5693 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:32:29.328527 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326155 5693 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:32:29.328527 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326158 5693 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:32:29.328527 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326160 5693 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:32:29.328527 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326163 5693 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:32:29.328527 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326166 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:32:29.328527 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326168 5693 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:32:29.328527 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326170 5693 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:32:29.328527 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326173 5693 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:32:29.328527 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326176 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:32:29.328527 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326178 5693 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:32:29.328527 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326181 5693 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:32:29.328527 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326183 5693 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:32:29.328527 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326186 5693 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:32:29.328527 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326188 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:32:29.328527 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326191 5693 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:32:29.328527 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326193 5693 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:32:29.328527 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326196 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:32:29.328527 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326199 5693 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:32:29.328527 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326201 5693 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:32:29.329028 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326204 5693 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:32:29.329028 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326206 5693 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:32:29.329028 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326209 5693 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:32:29.329028 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326212 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:32:29.329028 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326214 5693 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:32:29.329028 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326223 5693 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:32:29.329028 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326226 5693 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:32:29.329028 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326229 5693 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:32:29.329028 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326232 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:32:29.329028 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326234 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:32:29.329028 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326237 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:32:29.329028 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326239 5693 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:32:29.329028 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326243 5693 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:32:29.329028 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326245 5693 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:32:29.329028 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326248 5693 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:32:29.329028 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326251 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:32:29.329028 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326253 5693 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:32:29.329028 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326256 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:32:29.329028 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326258 5693 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:32:29.329028 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326261 5693 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:32:29.329507 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326263 5693 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:32:29.329507 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326265 5693 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:32:29.329507 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326270 5693 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:32:29.329507 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326273 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:32:29.329507 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326277 5693 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:32:29.329507 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326280 5693 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:32:29.329507 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326283 5693 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:32:29.329507 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326286 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:32:29.329507 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326290 5693 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:32:29.329507 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326294 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:32:29.329507 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326297 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:32:29.329507 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326300 5693 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:32:29.329507 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326303 5693 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:32:29.329507 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326306 5693 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:32:29.329507 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326308 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:32:29.329507 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326310 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:32:29.329507 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326313 5693 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:32:29.329507 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326315 5693 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:32:29.329961 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326320 5693 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:32:29.329961 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326324 5693 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:32:29.329961 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326326 5693 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:32:29.329961 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326329 5693 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:32:29.329961 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326331 5693 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:32:29.329961 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326333 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:32:29.329961 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326338 5693 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:32:29.329961 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326340 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:32:29.329961 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326343 5693 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:32:29.329961 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326345 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:32:29.329961 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326347 5693 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:32:29.329961 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326350 5693 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:32:29.329961 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326352 5693 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:32:29.329961 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326355 5693 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:32:29.329961 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326357 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:32:29.329961 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326860 5693 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:32:29.329961 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326867 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:32:29.329961 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326870 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:32:29.329961 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326873 5693 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:32:29.330419 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326876 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:32:29.330419 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326879 5693 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:32:29.330419 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326882 5693 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:32:29.330419 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326884 5693 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:32:29.330419 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326887 5693 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:32:29.330419 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326890 5693 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:32:29.330419 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326892 5693 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:32:29.330419 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326895 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:32:29.330419 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326897 5693 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:32:29.330419 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326900 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:32:29.330419 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326902 5693 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:32:29.330419 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326905 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:32:29.330419 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326907 5693 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:32:29.330419 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326910 5693 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:32:29.330419 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326914 5693 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:32:29.330419 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326918 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:32:29.330419 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326921 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:32:29.330419 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326923 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:32:29.330419 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326926 5693 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:32:29.330907 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326929 5693 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:32:29.330907 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326932 5693 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:32:29.330907 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326934 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:32:29.330907 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326937 5693 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:32:29.330907 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326939 5693 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:32:29.330907 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326941 5693 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:32:29.330907 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326944 5693 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:32:29.330907 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326946 5693 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:32:29.330907 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326949 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:32:29.330907 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326951 5693 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:32:29.330907 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326954 5693 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:32:29.330907 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326956 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:32:29.330907 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326958 5693 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:32:29.330907 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326961 5693 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:32:29.330907 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326964 5693 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:32:29.330907 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326966 5693 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:32:29.330907 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326970 5693 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:32:29.330907 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326973 5693 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:32:29.330907 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326976 5693 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:32:29.331380 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326978 5693 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:32:29.331380 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326981 5693 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:32:29.331380 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326983 5693 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:32:29.331380 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326985 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:32:29.331380 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326988 5693 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:32:29.331380 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326990 5693 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:32:29.331380 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326993 5693 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:32:29.331380 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326995 5693 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:32:29.331380 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.326998 5693 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:32:29.331380 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327001 5693 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:32:29.331380 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327003 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:32:29.331380 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327005 5693 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:32:29.331380 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327008 5693 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:32:29.331380 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327011 5693 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:32:29.331380 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327013 5693 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:32:29.331380 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327016 5693 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:32:29.331380 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327018 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:32:29.331380 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327022 5693 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:32:29.331380 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327024 5693 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:32:29.331380 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327026 5693 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:32:29.332087 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327029 5693 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:32:29.332087 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327031 5693 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:32:29.332087 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327034 5693 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:32:29.332087 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327036 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:32:29.332087 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327039 5693 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:32:29.332087 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327041 5693 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:32:29.332087 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327044 5693 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:32:29.332087 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327046 5693 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:32:29.332087 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327049 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:32:29.332087 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327051 5693 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:32:29.332087 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327054 5693 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:32:29.332087 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327056 5693 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:32:29.332087 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327058 5693 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:32:29.332087 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327061 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:32:29.332087 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327064 5693 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:32:29.332087 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327066 5693 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:32:29.332087 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327068 5693 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:32:29.332087 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327071 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:32:29.332087 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327074 5693 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:32:29.332087 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327076 5693 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:32:29.332870 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327079 5693 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:32:29.332870 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327082 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:32:29.332870 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327085 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:32:29.332870 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327087 5693 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:32:29.332870 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327090 5693 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:32:29.332870 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327092 5693 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:32:29.332870 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327095 5693 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:32:29.332870 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327097 5693 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:32:29.332870 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327100 5693 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:32:29.332870 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327102 5693 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:32:29.332870 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327105 5693 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:32:29.332870 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327107 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:32:29.332870 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327109 5693 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:32:29.332870 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327112 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:32:29.332870 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327114 5693 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:32:29.332870 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327117 5693 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:32:29.332870 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327119 5693 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:32:29.332870 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327122 5693 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:32:29.332870 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327124 5693 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:32:29.332870 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327127 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:32:29.333811 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327130 5693 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:32:29.333811 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327133 5693 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:32:29.333811 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327135 5693 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:32:29.333811 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327137 5693 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:32:29.333811 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327248 5693 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 12:32:29.333811 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327259 5693 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 12:32:29.333811 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327265 5693 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 12:32:29.333811 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327269 5693 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 12:32:29.333811 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327274 5693 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 12:32:29.333811 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327277 5693 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 12:32:29.333811 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327281 5693 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 12:32:29.333811 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327289 5693 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 12:32:29.333811 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327293 5693 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 12:32:29.333811 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327295 5693 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 12:32:29.333811 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327299 5693 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 12:32:29.333811 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327303 5693 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 12:32:29.333811 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327306 5693 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 12:32:29.333811 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327311 5693 flags.go:64] FLAG: --cgroup-root="" Oct 31 12:32:29.333811 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327314 5693 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 12:32:29.333811 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327317 5693 flags.go:64] FLAG: --client-ca-file="" Oct 31 12:32:29.333811 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327323 5693 flags.go:64] FLAG: --cloud-provider="external" Oct 31 12:32:29.333811 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327326 5693 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 12:32:29.333811 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327330 5693 flags.go:64] FLAG: --cluster-domain="" Oct 31 12:32:29.334814 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327333 5693 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 12:32:29.334814 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327337 5693 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 12:32:29.334814 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327340 5693 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 12:32:29.334814 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327343 5693 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 12:32:29.334814 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327347 5693 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 12:32:29.334814 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327350 5693 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 12:32:29.334814 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327353 5693 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 12:32:29.334814 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327358 5693 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 12:32:29.334814 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327361 5693 flags.go:64] FLAG: --contention-profiling="false" Oct 31 12:32:29.334814 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327363 5693 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 12:32:29.334814 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327367 5693 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 12:32:29.334814 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327370 5693 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 12:32:29.334814 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327373 5693 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 12:32:29.334814 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327377 5693 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 12:32:29.334814 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327380 5693 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 12:32:29.334814 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327383 5693 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 12:32:29.334814 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327386 5693 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 12:32:29.334814 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327389 5693 flags.go:64] FLAG: --enable-server="true" Oct 31 12:32:29.334814 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327392 5693 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 12:32:29.334814 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327395 5693 flags.go:64] FLAG: --event-burst="100" Oct 31 12:32:29.334814 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327398 5693 flags.go:64] FLAG: --event-qps="50" Oct 31 12:32:29.334814 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327401 5693 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 12:32:29.334814 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327404 5693 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 12:32:29.334814 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327407 5693 flags.go:64] FLAG: --eviction-hard="" Oct 31 12:32:29.335958 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327412 5693 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 12:32:29.335958 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327415 5693 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 12:32:29.335958 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327419 5693 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 12:32:29.335958 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327422 5693 flags.go:64] FLAG: --eviction-soft="" Oct 31 12:32:29.335958 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327425 5693 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 12:32:29.335958 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327429 5693 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 12:32:29.335958 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327432 5693 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 12:32:29.335958 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327435 5693 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 12:32:29.335958 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327438 5693 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 12:32:29.335958 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327441 5693 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 12:32:29.335958 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327444 5693 flags.go:64] FLAG: --feature-gates="" Oct 31 12:32:29.335958 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327448 5693 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 12:32:29.335958 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327451 5693 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 12:32:29.335958 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327454 5693 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 12:32:29.335958 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327457 5693 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 12:32:29.335958 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327460 5693 flags.go:64] FLAG: --healthz-port="10248" Oct 31 12:32:29.335958 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327463 5693 flags.go:64] FLAG: --help="false" Oct 31 12:32:29.335958 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327466 5693 flags.go:64] FLAG: --hostname-override="ip-10-0-131-157.us-east-2.compute.internal" Oct 31 12:32:29.335958 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327469 5693 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 12:32:29.335958 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327472 5693 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 12:32:29.335958 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327475 5693 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 12:32:29.335958 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327478 5693 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 12:32:29.335958 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327481 5693 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 12:32:29.336984 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327485 5693 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 12:32:29.336984 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327487 5693 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 12:32:29.336984 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327490 5693 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 12:32:29.336984 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327493 5693 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 12:32:29.336984 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327495 5693 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 12:32:29.336984 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327499 5693 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 12:32:29.336984 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327502 5693 flags.go:64] FLAG: --kube-reserved="" Oct 31 12:32:29.336984 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327504 5693 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 12:32:29.336984 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327507 5693 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 12:32:29.336984 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327510 5693 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 12:32:29.336984 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327513 5693 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 12:32:29.336984 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327517 5693 flags.go:64] FLAG: --lock-file="" Oct 31 12:32:29.336984 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327520 5693 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 12:32:29.336984 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327523 5693 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 12:32:29.336984 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327527 5693 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 12:32:29.336984 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327533 5693 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 12:32:29.336984 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327535 5693 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 12:32:29.336984 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327539 5693 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 12:32:29.336984 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327542 5693 flags.go:64] FLAG: --logging-format="text" Oct 31 12:32:29.336984 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327545 5693 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 12:32:29.336984 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327548 5693 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 12:32:29.336984 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327552 5693 flags.go:64] FLAG: --manifest-url="" Oct 31 12:32:29.336984 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327554 5693 flags.go:64] FLAG: --manifest-url-header="" Oct 31 12:32:29.336984 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327559 5693 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 12:32:29.336984 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327562 5693 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 12:32:29.338097 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327566 5693 flags.go:64] FLAG: --max-pods="110" Oct 31 12:32:29.338097 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327569 5693 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 12:32:29.338097 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327572 5693 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 12:32:29.338097 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327574 5693 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 12:32:29.338097 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327577 5693 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 12:32:29.338097 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327580 5693 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 12:32:29.338097 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327583 5693 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 12:32:29.338097 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327586 5693 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 12:32:29.338097 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327592 5693 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 12:32:29.338097 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327595 5693 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 12:32:29.338097 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327599 5693 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 12:32:29.338097 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327601 5693 flags.go:64] FLAG: --pod-cidr="" Oct 31 12:32:29.338097 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327604 5693 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 12:32:29.338097 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327607 5693 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 12:32:29.338097 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327610 5693 flags.go:64] FLAG: --pods-per-core="0" Oct 31 12:32:29.338097 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327613 5693 flags.go:64] FLAG: --port="10250" Oct 31 12:32:29.338097 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327616 5693 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 12:32:29.338097 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327619 5693 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0bc5251286bf527fa" Oct 31 12:32:29.338097 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327623 5693 flags.go:64] FLAG: --qos-reserved="" Oct 31 12:32:29.338097 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327626 5693 flags.go:64] FLAG: --read-only-port="10255" Oct 31 12:32:29.338097 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327629 5693 flags.go:64] FLAG: --register-node="true" Oct 31 12:32:29.338097 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327633 5693 flags.go:64] FLAG: --register-with-taints="" Oct 31 12:32:29.338097 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327655 5693 flags.go:64] FLAG: --registry-burst="10" Oct 31 12:32:29.338097 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327660 5693 flags.go:64] FLAG: --registry-qps="5" Oct 31 12:32:29.338097 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327664 5693 flags.go:64] FLAG: --reserved-cpus="" Oct 31 12:32:29.338097 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327668 5693 flags.go:64] FLAG: --reserved-memory="" Oct 31 12:32:29.339212 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327673 5693 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 12:32:29.339212 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327676 5693 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 12:32:29.339212 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327679 5693 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 12:32:29.339212 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327682 5693 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 12:32:29.339212 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327684 5693 flags.go:64] FLAG: --runonce="false" Oct 31 12:32:29.339212 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327687 5693 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 12:32:29.339212 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327690 5693 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 12:32:29.339212 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327693 5693 flags.go:64] FLAG: --seccomp-default="false" Oct 31 12:32:29.339212 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327696 5693 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 12:32:29.339212 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327699 5693 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 12:32:29.339212 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327702 5693 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 12:32:29.339212 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327705 5693 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 12:32:29.339212 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327708 5693 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 12:32:29.339212 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327711 5693 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 12:32:29.339212 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327714 5693 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 12:32:29.339212 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327717 5693 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 12:32:29.339212 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327720 5693 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 12:32:29.339212 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327723 5693 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 12:32:29.339212 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327726 5693 flags.go:64] FLAG: --system-cgroups="" Oct 31 12:32:29.339212 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327728 5693 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 12:32:29.339212 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327735 5693 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 12:32:29.339212 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327738 5693 flags.go:64] FLAG: --tls-cert-file="" Oct 31 12:32:29.339212 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327741 5693 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 12:32:29.339212 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327746 5693 flags.go:64] FLAG: --tls-min-version="" Oct 31 12:32:29.339212 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327750 5693 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 12:32:29.340393 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327753 5693 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 12:32:29.340393 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327756 5693 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 12:32:29.340393 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327760 5693 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 12:32:29.340393 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327772 5693 flags.go:64] FLAG: --v="2" Oct 31 12:32:29.340393 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327777 5693 flags.go:64] FLAG: --version="false" Oct 31 12:32:29.340393 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327782 5693 flags.go:64] FLAG: --vmodule="" Oct 31 12:32:29.340393 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327786 5693 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 12:32:29.340393 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.327789 5693 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 12:32:29.340393 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327947 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:32:29.340393 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327951 5693 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:32:29.340393 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327954 5693 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:32:29.340393 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327957 5693 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:32:29.340393 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327960 5693 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:32:29.340393 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327963 5693 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:32:29.340393 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327965 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:32:29.340393 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327968 5693 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:32:29.340393 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327970 5693 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:32:29.340393 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327973 5693 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:32:29.340393 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327975 5693 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:32:29.340393 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327978 5693 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:32:29.340393 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327981 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:32:29.340393 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327983 5693 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:32:29.341340 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327986 5693 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:32:29.341340 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327989 5693 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:32:29.341340 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327991 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:32:29.341340 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327994 5693 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:32:29.341340 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327996 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:32:29.341340 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.327999 5693 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:32:29.341340 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328001 5693 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:32:29.341340 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328004 5693 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:32:29.341340 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328006 5693 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:32:29.341340 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328011 5693 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:32:29.341340 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328014 5693 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:32:29.341340 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328017 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:32:29.341340 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328021 5693 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:32:29.341340 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328023 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:32:29.341340 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328026 5693 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:32:29.341340 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328028 5693 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:32:29.341340 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328031 5693 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:32:29.341340 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328034 5693 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:32:29.341340 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328036 5693 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:32:29.341340 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328039 5693 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:32:29.341972 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328041 5693 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:32:29.341972 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328044 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:32:29.341972 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328046 5693 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:32:29.341972 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328049 5693 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:32:29.341972 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328051 5693 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:32:29.341972 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328054 5693 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:32:29.341972 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328056 5693 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:32:29.341972 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328058 5693 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:32:29.341972 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328061 5693 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:32:29.341972 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328064 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:32:29.341972 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328068 5693 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:32:29.341972 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328071 5693 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:32:29.341972 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328074 5693 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:32:29.341972 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328076 5693 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:32:29.341972 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328078 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:32:29.341972 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328081 5693 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:32:29.341972 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328083 5693 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:32:29.341972 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328086 5693 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:32:29.341972 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328090 5693 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:32:29.342486 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328093 5693 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:32:29.342486 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328096 5693 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:32:29.342486 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328099 5693 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:32:29.342486 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328102 5693 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:32:29.342486 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328104 5693 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:32:29.342486 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328111 5693 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:32:29.342486 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328114 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:32:29.342486 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328116 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:32:29.342486 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328119 5693 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:32:29.342486 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328122 5693 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:32:29.342486 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328124 5693 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:32:29.342486 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328127 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:32:29.342486 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328129 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:32:29.342486 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328132 5693 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:32:29.342486 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328134 5693 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:32:29.342486 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328137 5693 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:32:29.342486 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328140 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:32:29.342486 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328142 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:32:29.342486 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328144 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:32:29.342486 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328147 5693 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:32:29.343105 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328150 5693 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:32:29.343105 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328152 5693 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:32:29.343105 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328154 5693 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:32:29.343105 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328157 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:32:29.343105 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328160 5693 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:32:29.343105 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328162 5693 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:32:29.343105 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328165 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:32:29.343105 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328167 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:32:29.343105 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328170 5693 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:32:29.343105 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328172 5693 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:32:29.343105 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328175 5693 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:32:29.343105 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328177 5693 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:32:29.343105 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328180 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:32:29.343105 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328182 5693 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:32:29.343105 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328187 5693 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:32:29.343105 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328190 5693 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:32:29.343105 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328193 5693 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:32:29.343105 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328197 5693 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:32:29.343105 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328200 5693 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:32:29.343613 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328203 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:32:29.343613 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328205 5693 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:32:29.343613 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328208 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:32:29.343613 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328210 5693 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:32:29.343613 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328213 5693 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:32:29.343613 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328215 5693 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:32:29.343613 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328218 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:32:29.343613 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328221 5693 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:32:29.343613 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328223 5693 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:32:29.343613 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328226 5693 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:32:29.343613 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328228 5693 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:32:29.343613 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328231 5693 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:32:29.343613 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328233 5693 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:32:29.343613 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328235 5693 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:32:29.343613 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.328261 5693 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 12:32:29.343613 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328297 5693 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:32:29.344039 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328302 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:32:29.344039 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328305 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:32:29.344039 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328308 5693 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:32:29.344039 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328311 5693 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:32:29.344039 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328314 5693 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:32:29.344039 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328316 5693 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:32:29.344039 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328319 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:32:29.344039 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328321 5693 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:32:29.344039 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328324 5693 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:32:29.344039 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328326 5693 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:32:29.344039 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328329 5693 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:32:29.344039 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328333 5693 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:32:29.344039 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328335 5693 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:32:29.344039 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328338 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:32:29.344039 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328340 5693 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:32:29.344039 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328344 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:32:29.344039 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328347 5693 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:32:29.344039 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328350 5693 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:32:29.344039 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328353 5693 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:32:29.344039 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328355 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:32:29.344608 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328358 5693 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:32:29.344608 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328360 5693 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:32:29.344608 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328363 5693 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:32:29.344608 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328365 5693 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:32:29.344608 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328368 5693 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:32:29.344608 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328370 5693 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:32:29.344608 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328373 5693 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:32:29.344608 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328375 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:32:29.344608 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328378 5693 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:32:29.344608 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328380 5693 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:32:29.344608 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328383 5693 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:32:29.344608 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328385 5693 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:32:29.344608 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328387 5693 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:32:29.344608 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328390 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:32:29.344608 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328392 5693 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:32:29.344608 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328395 5693 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:32:29.344608 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328397 5693 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:32:29.344608 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328400 5693 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:32:29.344608 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328402 5693 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:32:29.344608 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328405 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:32:29.345243 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328407 5693 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:32:29.345243 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328410 5693 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:32:29.345243 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328412 5693 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:32:29.345243 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328416 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:32:29.345243 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328419 5693 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:32:29.345243 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328422 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:32:29.345243 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328424 5693 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:32:29.345243 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328427 5693 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:32:29.345243 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328431 5693 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:32:29.345243 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328434 5693 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:32:29.345243 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328437 5693 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:32:29.345243 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328439 5693 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:32:29.345243 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328442 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:32:29.345243 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328445 5693 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:32:29.345243 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328447 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:32:29.345243 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328450 5693 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:32:29.345243 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328452 5693 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:32:29.345243 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328454 5693 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:32:29.345243 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328457 5693 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:32:29.345243 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328460 5693 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:32:29.345789 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328462 5693 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:32:29.345789 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328464 5693 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:32:29.345789 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328467 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:32:29.345789 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328469 5693 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:32:29.345789 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328472 5693 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:32:29.345789 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328474 5693 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:32:29.345789 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328477 5693 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:32:29.345789 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328479 5693 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:32:29.345789 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328482 5693 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:32:29.345789 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328485 5693 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:32:29.345789 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328487 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:32:29.345789 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328491 5693 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:32:29.345789 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328494 5693 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:32:29.345789 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328498 5693 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:32:29.345789 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328501 5693 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:32:29.345789 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328505 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:32:29.345789 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328507 5693 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:32:29.345789 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328510 5693 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:32:29.345789 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328512 5693 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:32:29.346236 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328518 5693 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:32:29.346236 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328522 5693 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:32:29.346236 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328524 5693 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:32:29.346236 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328531 5693 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:32:29.346236 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328534 5693 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:32:29.346236 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328537 5693 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:32:29.346236 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328540 5693 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:32:29.346236 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328542 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:32:29.346236 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328545 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:32:29.346236 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328547 5693 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:32:29.346236 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328550 5693 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:32:29.346236 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328552 5693 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:32:29.346236 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328555 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:32:29.346236 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328557 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:32:29.346236 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328559 5693 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:32:29.346236 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328563 5693 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:32:29.346236 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328566 5693 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:32:29.346236 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328568 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:32:29.346236 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328570 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:32:29.346711 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328573 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:32:29.346711 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328575 5693 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:32:29.346711 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328578 5693 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:32:29.346711 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328580 5693 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:32:29.346711 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328583 5693 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:32:29.346711 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328585 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:32:29.346711 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.328588 5693 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:32:29.346711 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.334446 5693 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 12:32:29.346711 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.334467 5693 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 12:32:29.346711 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.334493 5693 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 12:32:29.346711 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.334501 5693 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 12:32:29.346711 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334610 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:32:29.346711 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334620 5693 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:32:29.346711 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334625 5693 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:32:29.346711 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334630 5693 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:32:29.346711 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334634 5693 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:32:29.346711 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334661 5693 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:32:29.346711 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334666 5693 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:32:29.346711 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334671 5693 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:32:29.347161 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334675 5693 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:32:29.347161 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334679 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:32:29.347161 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334683 5693 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:32:29.347161 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334687 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:32:29.347161 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334691 5693 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:32:29.347161 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334695 5693 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:32:29.347161 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334700 5693 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:32:29.347161 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334704 5693 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:32:29.347161 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334709 5693 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:32:29.347161 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334713 5693 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:32:29.347161 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334717 5693 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:32:29.347161 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334721 5693 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:32:29.347161 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334725 5693 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:32:29.347161 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334730 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:32:29.347161 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334734 5693 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:32:29.347161 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334738 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:32:29.347161 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334743 5693 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:32:29.347161 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334748 5693 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:32:29.347161 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334752 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:32:29.347161 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334756 5693 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:32:29.347630 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334761 5693 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:32:29.347630 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334766 5693 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:32:29.347630 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334770 5693 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:32:29.347630 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334774 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:32:29.347630 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334778 5693 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:32:29.347630 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334782 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:32:29.347630 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334787 5693 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:32:29.347630 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334791 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:32:29.347630 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334795 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:32:29.347630 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334800 5693 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:32:29.347630 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334804 5693 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:32:29.347630 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334809 5693 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:32:29.347630 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334813 5693 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:32:29.347630 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334817 5693 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:32:29.347630 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334821 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:32:29.347630 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334827 5693 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:32:29.347630 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334833 5693 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:32:29.347630 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334837 5693 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:32:29.347630 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334843 5693 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:32:29.348243 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334861 5693 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:32:29.348243 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334866 5693 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:32:29.348243 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334870 5693 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:32:29.348243 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334875 5693 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:32:29.348243 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334880 5693 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:32:29.348243 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334884 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:32:29.348243 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334888 5693 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:32:29.348243 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334893 5693 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:32:29.348243 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334897 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:32:29.348243 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334902 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:32:29.348243 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334907 5693 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:32:29.348243 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334912 5693 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:32:29.348243 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334917 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:32:29.348243 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334921 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:32:29.348243 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334925 5693 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:32:29.348243 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334929 5693 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:32:29.348243 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334933 5693 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:32:29.348243 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334937 5693 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:32:29.348243 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334942 5693 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:32:29.348243 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334946 5693 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:32:29.348752 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334950 5693 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:32:29.348752 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334954 5693 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:32:29.348752 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334959 5693 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:32:29.348752 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334963 5693 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:32:29.348752 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334967 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:32:29.348752 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334971 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:32:29.348752 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334976 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:32:29.348752 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334980 5693 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:32:29.348752 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334984 5693 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:32:29.348752 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334989 5693 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:32:29.348752 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334993 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:32:29.348752 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.334997 5693 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:32:29.348752 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335011 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:32:29.348752 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335015 5693 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:32:29.348752 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335020 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:32:29.348752 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335024 5693 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:32:29.348752 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335028 5693 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:32:29.348752 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335032 5693 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:32:29.348752 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335037 5693 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:32:29.348752 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335041 5693 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:32:29.349271 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335045 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:32:29.349271 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335051 5693 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:32:29.349271 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335058 5693 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:32:29.349271 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335063 5693 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:32:29.349271 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335067 5693 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:32:29.349271 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335072 5693 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:32:29.349271 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335076 5693 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:32:29.349271 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335081 5693 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:32:29.349271 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335085 5693 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:32:29.349271 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335089 5693 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:32:29.349271 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335093 5693 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:32:29.349271 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335097 5693 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:32:29.349271 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335102 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:32:29.349271 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335106 5693 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:32:29.349271 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335110 5693 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:32:29.349271 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335114 5693 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:32:29.349271 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335118 5693 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:32:29.349271 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335123 5693 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:32:29.349271 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335127 5693 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:32:29.349755 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.335174 5693 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 12:32:29.349755 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335422 5693 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:32:29.349755 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335434 5693 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:32:29.349755 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335440 5693 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:32:29.349755 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335445 5693 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:32:29.349755 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335449 5693 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:32:29.349755 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335462 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:32:29.349755 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335467 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:32:29.349755 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335471 5693 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:32:29.349755 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335475 5693 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:32:29.349755 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335480 5693 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:32:29.349755 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335484 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:32:29.349755 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335488 5693 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:32:29.349755 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335493 5693 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:32:29.349755 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335497 5693 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:32:29.349755 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335502 5693 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:32:29.350129 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335506 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:32:29.350129 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335510 5693 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:32:29.350129 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335517 5693 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:32:29.350129 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335524 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:32:29.350129 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335529 5693 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:32:29.350129 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335533 5693 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:32:29.350129 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335537 5693 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:32:29.350129 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335542 5693 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:32:29.350129 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335546 5693 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:32:29.350129 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335551 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:32:29.350129 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335555 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:32:29.350129 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335560 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:32:29.350129 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335564 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:32:29.350129 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335568 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:32:29.350129 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335572 5693 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:32:29.350129 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335576 5693 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:32:29.350129 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335580 5693 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:32:29.350129 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335585 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:32:29.350129 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335589 5693 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:32:29.350591 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335594 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:32:29.350591 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335598 5693 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:32:29.350591 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335602 5693 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:32:29.350591 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335607 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:32:29.350591 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335613 5693 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:32:29.350591 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335617 5693 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:32:29.350591 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335622 5693 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:32:29.350591 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335627 5693 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:32:29.350591 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335631 5693 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:32:29.350591 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335653 5693 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:32:29.350591 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335658 5693 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:32:29.350591 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335663 5693 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:32:29.350591 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335668 5693 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:32:29.350591 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335672 5693 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:32:29.350591 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335676 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:32:29.350591 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335681 5693 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:32:29.350591 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335685 5693 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:32:29.350591 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335689 5693 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:32:29.350591 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335693 5693 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:32:29.351360 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335697 5693 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:32:29.351360 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335702 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:32:29.351360 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335706 5693 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:32:29.351360 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335710 5693 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:32:29.351360 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335714 5693 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:32:29.351360 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335719 5693 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:32:29.351360 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335723 5693 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:32:29.351360 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335727 5693 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:32:29.351360 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335731 5693 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:32:29.351360 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335735 5693 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:32:29.351360 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335740 5693 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:32:29.351360 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335745 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:32:29.351360 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335749 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:32:29.351360 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335753 5693 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:32:29.351360 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335757 5693 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:32:29.351360 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335762 5693 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:32:29.351360 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335773 5693 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:32:29.351360 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335778 5693 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:32:29.351360 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335784 5693 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:32:29.351360 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335790 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:32:29.352084 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335795 5693 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:32:29.352084 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335800 5693 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:32:29.352084 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335804 5693 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:32:29.352084 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335808 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:32:29.352084 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335812 5693 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:32:29.352084 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335817 5693 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:32:29.352084 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335821 5693 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:32:29.352084 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335825 5693 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:32:29.352084 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335830 5693 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:32:29.352084 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335834 5693 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:32:29.352084 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335838 5693 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:32:29.352084 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335843 5693 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:32:29.352084 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335847 5693 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:32:29.352084 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335852 5693 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:32:29.352084 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335857 5693 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:32:29.352084 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335861 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:32:29.352084 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335865 5693 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:32:29.352084 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335869 5693 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:32:29.352084 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335873 5693 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:32:29.352536 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335877 5693 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:32:29.352536 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335882 5693 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:32:29.352536 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335886 5693 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:32:29.352536 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335890 5693 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:32:29.352536 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335895 5693 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:32:29.352536 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335899 5693 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:32:29.352536 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335903 5693 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:32:29.352536 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335907 5693 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:32:29.352536 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335911 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:32:29.352536 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335915 5693 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:32:29.352536 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335920 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:32:29.352536 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335924 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:32:29.352536 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335930 5693 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:32:29.352536 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.335934 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:32:29.352536 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.335978 5693 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 12:32:29.352536 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336033 5693 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:32:29.352997 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336040 5693 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:32:29.352997 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336044 5693 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:32:29.352997 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336049 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:32:29.352997 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336053 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:32:29.352997 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336057 5693 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:32:29.352997 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336061 5693 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:32:29.352997 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336065 5693 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:32:29.352997 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336069 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:32:29.352997 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336078 5693 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:32:29.352997 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336083 5693 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:32:29.352997 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336087 5693 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:32:29.352997 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336091 5693 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:32:29.352997 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336096 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:32:29.352997 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336100 5693 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:32:29.352997 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336107 5693 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:32:29.352997 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336112 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:32:29.352997 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336117 5693 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:32:29.352997 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336121 5693 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:32:29.352997 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336126 5693 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:32:29.353437 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336130 5693 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:32:29.353437 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336134 5693 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:32:29.353437 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336138 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:32:29.353437 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336142 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:32:29.353437 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336147 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:32:29.353437 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336151 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:32:29.353437 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336155 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:32:29.353437 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336159 5693 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:32:29.353437 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336163 5693 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:32:29.353437 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336168 5693 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:32:29.353437 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336173 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:32:29.353437 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336177 5693 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:32:29.353437 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336181 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:32:29.353437 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336186 5693 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:32:29.353437 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336190 5693 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:32:29.353437 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336195 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:32:29.353437 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336200 5693 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:32:29.353437 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336204 5693 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:32:29.353437 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336208 5693 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:32:29.353910 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336212 5693 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:32:29.353910 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336216 5693 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:32:29.353910 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336222 5693 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:32:29.353910 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336227 5693 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:32:29.353910 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336231 5693 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:32:29.353910 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336235 5693 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:32:29.353910 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336239 5693 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:32:29.353910 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336243 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:32:29.353910 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336248 5693 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:32:29.353910 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336252 5693 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:32:29.353910 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336257 5693 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:32:29.353910 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336261 5693 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:32:29.353910 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336265 5693 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:32:29.353910 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336270 5693 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:32:29.353910 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336274 5693 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:32:29.353910 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336278 5693 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:32:29.353910 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336282 5693 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:32:29.353910 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336286 5693 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:32:29.353910 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336291 5693 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:32:29.353910 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336295 5693 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:32:29.354367 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336299 5693 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:32:29.354367 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336303 5693 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:32:29.354367 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336309 5693 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:32:29.354367 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336313 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:32:29.354367 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336318 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:32:29.354367 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336322 5693 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:32:29.354367 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336326 5693 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:32:29.354367 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336331 5693 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:32:29.354367 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336336 5693 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:32:29.354367 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336340 5693 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:32:29.354367 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336346 5693 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:32:29.354367 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336352 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:32:29.354367 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336357 5693 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:32:29.354367 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336361 5693 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:32:29.354367 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336367 5693 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:32:29.354367 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336371 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:32:29.354367 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336376 5693 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:32:29.354367 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336380 5693 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:32:29.354367 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336385 5693 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:32:29.354842 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336389 5693 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:32:29.354842 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336394 5693 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:32:29.354842 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336398 5693 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:32:29.354842 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336402 5693 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:32:29.354842 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336406 5693 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:32:29.354842 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336411 5693 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:32:29.354842 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336415 5693 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:32:29.354842 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336421 5693 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:32:29.354842 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336425 5693 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:32:29.354842 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336429 5693 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:32:29.354842 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336433 5693 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:32:29.354842 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336437 5693 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:32:29.354842 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336441 5693 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:32:29.354842 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336445 5693 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:32:29.354842 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336449 5693 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:32:29.354842 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336454 5693 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:32:29.354842 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336458 5693 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:32:29.354842 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336462 5693 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:32:29.354842 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336467 5693 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:32:29.354842 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336471 5693 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:32:29.355347 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336478 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:32:29.355347 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336483 5693 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:32:29.355347 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336487 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:32:29.355347 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336491 5693 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:32:29.355347 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336495 5693 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:32:29.355347 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336499 5693 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:32:29.355347 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336503 5693 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:32:29.355347 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:29.336509 5693 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:32:29.355347 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.336590 5693 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 12:32:29.355347 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.338172 5693 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 12:32:29.355347 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.338250 5693 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 12:32:29.355347 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.338520 5693 server.go:1014] "Starting client certificate rotation" Oct 31 12:32:29.355347 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.338663 5693 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 12:32:29.355347 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.338712 5693 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 12:26:16 +0000 UTC" deadline="2028-05-04 09:49:26.156100836 +0000 UTC" Oct 31 12:32:29.355347 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.338740 5693 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13221h16m56.817363134s" Oct 31 12:32:29.355347 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.339265 5693 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 12:32:29.355748 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.341114 5693 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 12:32:29.355748 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.342478 5693 log.go:25] "Validated CRI v1 runtime API" Oct 31 12:32:29.355748 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.344320 5693 log.go:25] "Validated CRI v1 image API" Oct 31 12:32:29.355748 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.345060 5693 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 12:32:29.355748 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.346976 5693 fs.go:135] Filesystem UUIDs: map[1b5eaec0-cd27-4b2e-a677-03c9a4fb30a2:/dev/nvme0n1p4 266238b3-2d5c-426e-835b-cefe6aae45c6:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 31 12:32:29.355903 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.346992 5693 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a1787850d6b84588d61e49e4b5d640845f0c97caebe94b71026f54cd02be1a4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a1787850d6b84588d61e49e4b5d640845f0c97caebe94b71026f54cd02be1a4/userdata/shm major:0 minor:308 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0b35abf8cdb506dcaacb590ad330fb6cbbaee3964124e6b0ca499b097254feb4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0b35abf8cdb506dcaacb590ad330fb6cbbaee3964124e6b0ca499b097254feb4/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2371284ce749382e13be426c9d83dd544dfdcc771669afc0a19abb73b59a2845/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2371284ce749382e13be426c9d83dd544dfdcc771669afc0a19abb73b59a2845/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2449fb987accd7f8129f3ad9d4ad0d8aeb202aea937d539aca589a09d925778e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2449fb987accd7f8129f3ad9d4ad0d8aeb202aea937d539aca589a09d925778e/userdata/shm major:0 minor:335 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2a2f94ca992e3771f5713cbd3ada4bc4cab041afa0a1eb45530fe3ee42469c15/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2a2f94ca992e3771f5713cbd3ada4bc4cab041afa0a1eb45530fe3ee42469c15/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3504f1ff87a9423da711361f7da6db57c5bd88c69326687759db34467ddcfb1b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3504f1ff87a9423da711361f7da6db57c5bd88c69326687759db34467ddcfb1b/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3b8f659ef7a491710aa5a36a9d4f2c3b2469a212d3bb308827483dda8be8b556/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3b8f659ef7a491710aa5a36a9d4f2c3b2469a212d3bb308827483dda8be8b556/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4242a5b80f9bf08c64de825adaa8d447534331d8362cdedc8dbf2fc82d20f911/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4242a5b80f9bf08c64de825adaa8d447534331d8362cdedc8dbf2fc82d20f911/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4334c12d573521f95b214022be018ae2961732b1ee6e21a1d7c07ef1b08b3dce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4334c12d573521f95b214022be018ae2961732b1ee6e21a1d7c07ef1b08b3dce/userdata/shm major:0 minor:353 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7f9fad3cf08ec8fb0a264377931414fafbf489a06b13f1a62d1867691c252545/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7f9fad3cf08ec8fb0a264377931414fafbf489a06b13f1a62d1867691c252545/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/86e20428c0c192b9f1bdf2e62481619c797e84aeffec2ec17676284c65292fe7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/86e20428c0c192b9f1bdf2e62481619c797e84aeffec2ec17676284c65292fe7/userdata/shm major:0 minor:254 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/98c00b91a1a67a918bfc7bdefc29cf5f033528396d20cfa2be189cc550975729/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/98c00b91a1a67a918bfc7bdefc29cf5f033528396d20cfa2be189cc550975729/userdata/shm major:0 minor:302 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9af65c148112ebb9dd1268dbe8c909387f2ca55b28da2b69649c5428e5be0e76/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9af65c148112ebb9dd1268dbe8c909387f2ca55b28da2b69649c5428e5be0e76/userdata/shm major:0 minor:326 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a32690afe6d52b867aceffb58e48753674527d54678cbb3cdbb2b441b7c99e8c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a32690afe6d52b867aceffb58e48753674527d54678cbb3cdbb2b441b7c99e8c/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a633f11a094758484648b16d5585aebc43099d3f40a08aa8223d97224bf41d0e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a633f11a094758484648b16d5585aebc43099d3f40a08aa8223d97224bf41d0e/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/af8a3287d760c49bab4cd6f000a6ccf86a22d13d3c19f124b2e2654ed0084904/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/af8a3287d760c49bab4cd6f000a6ccf86a22d13d3c19f124b2e2654ed0084904/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b391fcf82f11c31b74103e72e0102b9d81913ae13eb810d71671a5954464aad3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b391fcf82f11c31b74103e72e0102b9d81913ae13eb810d71671a5954464aad3/userdata/shm major:0 minor:328 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cc809f08fd1b1ec8bba0f6df66aee099d71617ec4e73c31c6b501d2db29a95c7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cc809f08fd1b1ec8bba0f6df66aee099d71617ec4e73c31c6b501d2db29a95c7/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d80f43156c9865079efe4848eefacddbdee81a7f18ec7222d77c5723944c269a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d80f43156c9865079efe4848eefacddbdee81a7f18ec7222d77c5723944c269a/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/da22f1b02aab199a2fd5d6f85284e273cd2aed5d3d2946debb29989dc43335d2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/da22f1b02aab199a2fd5d6f85284e273cd2aed5d3d2946debb29989dc43335d2/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e6eb922f4a0cd05a8f419575ee3f34f4ac887c9b83ebc2138465e7b10d647228/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e6eb922f4a0cd05a8f419575ee3f34f4ac887c9b83ebc2138465e7b10d647228/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f763f3a95d574aac7ca59fd0a9b3957bb56261dce7e237013b93f0c6cd662978/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f763f3a95d574aac7ca59fd0a9b3957bb56261dce7e237013b93f0c6cd662978/userdata/shm major:0 minor:284 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/02c6d8f5-97ff-41e0-8e38-71e1b0c87f54/volumes/kubernetes.io~projected/kube-api-access-ph9jm:{mountpoint:/var/lib/kubelet/pods/02c6d8f5-97ff-41e0-8e38-71e1b0c87f54/volumes/kubernetes.io~projected/kube-api-access-ph9jm major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b091d91-47bc-4480-85b2-e23a8139df66/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/0b091d91-47bc-4480-85b2-e23a8139df66/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b091d91-47bc-4480-85b2-e23a8139df66/volumes/kubernetes.io~projected/kube-api-access-4n79b:{mountpoint:/var/lib/kubelet/pods/0b091d91-47bc-4480-85b2-e23a8139df66/volumes/kubernetes.io~projected/kube-api-access-4n79b major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b091d91-47bc-4480-85b2-e23a8139df66/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/0b091d91-47bc-4480-85b2-e23a8139df66/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f343e95-e397-49aa-b023-98337d3be3ff/volumes/kubernetes.io~projected/kube-api-access-kpx9x:{mountpoint:/var/lib/kubelet/pods/0f343e95-e397-49aa-b023-98337d3be3ff/volumes/kubernetes.io~projected/kube-api-access-kpx9x major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f343e95-e397-49aa-b023-98337d3be3ff/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/0f343e95-e397-49aa-b023-98337d3be3ff/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f343e95-e397-49aa-b023-98337d3be3ff/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/0f343e95-e397-49aa-b023-98337d3be3ff/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14dc1e6d-70e1-4ba5-bffb-d9299433756d/volumes/kubernetes.io~projected/kube-api-access-sd2xl:{mountpoint:/var/lib/kubelet/pods/14dc1e6d-70e1-4ba5-bffb-d9299433756d/volumes/kubernetes.io~projected/kube-api-access-sd2xl major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24e17633-30a9-4410-b000-2fbb97202951/volumes/kubernetes.io~projected/kube-api-access-zhppb:{mountpoint:/var/lib/kubelet/pods/24e17633-30a9-4410-b000-2fbb97202951/volumes/kubernetes.io~projected/kube-api-access-zhppb major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25a02fd6-8ab0-4cca-94cd-d4082f874170/volumes/kubernetes.io~projected/kube-api-access-srn9m:{mountpoint:/var/lib/kubelet/pods/25a02fd6-8ab0-4cca-94cd-d4082f874170/volumes/kubernetes.io~projected/kube-api-access-srn9m major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25a02fd6-8ab0-4cca-94cd-d4082f874170/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/25a02fd6-8ab0-4cca-94cd-d4082f874170/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25a02fd6-8ab0-4cca-94cd-d4082f874170/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/25a02fd6-8ab0-4cca-94cd-d4082f874170/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25a02fd6-8ab0-4cca-94cd-d4082f874170/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/25a02fd6-8ab0-4cca-94cd-d4082f874170/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26913ad4-6613-4f3f-9819-9be238088c0a/volumes/kubernetes.io~projected/kube-api-access-779z6:{mountpoint:/var/lib/kubelet/pods/26913ad4-6613-4f3f-9819-9be238088c0a/volumes/kubernetes.io~projected/kube-api-access-779z6 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3015bfb0-80c8-4c01-9053-85e5ff2318fb/volumes/kubernetes.io~projected/kube-api-access-5d92p:{mountpoint:/var/lib/kubelet/pods/3015bfb0-80c8-4c01-9053-85e5ff2318fb/volumes/kubernetes.io~projected/kube-api-access-5d92p major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3015bfb0-80c8-4c01-9053-85e5ff2318fb/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/3015bfb0-80c8-4c01-9053-85e5ff2318fb/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43421968-0e5b-4e27-be79-104aae5f72e0/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/43421968-0e5b-4e27-be79-104aae5f72e0/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45870e34-5e37-443a-826b-81c9c031051c/volumes/kubernetes.io~projected/kube-api-access-kdlgx:{mountpoint:/var/lib/kubelet/pods/45870e34-5e37-443a-826b-81c9c031051c/volumes/kubernetes.io~projected/kube-api-access-kdlgx major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45870e34-5e37-443a-826b-81c9c031051c/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/45870e34-5e37-443a-826b-81c9c031051c/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b6af1c5-6045-4e4c-aa87-2473d19bd000/volumes/kubernetes.io~projected/kube-api-access-rprnd:{mountpoint:/var/lib/kubelet/pods/4b6af1c5-6045-4e4c-aa87-2473d19bd000/volumes/kubernetes.io~projected/kube-api-access-rprnd major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b6af1c5-6045-4e4c-aa87-2473d19bd000/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/4b6af1c5-6045-4e4c-aa87-2473d19bd000/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b6af1c5-6045-4e4c-aa87-2473d19bd000/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/4b6af1c5-6045-4e4c-aa87-2473d19bd000/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d3565a5-d07a-4dc0-abd1-e697573baf22/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/5d3565a5-d07a-4dc0-abd1-e697573baf22/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d3565a5-d07a-4dc0-abd1-e697573baf22/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/5d3565a5-d07a-4dc0-abd1-e697573baf22/volumes/kubernetes.io~empty-dir/tmp major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d3565a5-d07a-4dc0-abd1-e697573baf22/volumes/kubernetes.io~projected/kube-api-access-h5hf6:{mountpoint:/var/lib/kubelet/pods/5d3565a5-d07a-4dc0-abd1-e697573baf22/volumes/kubernetes.io~projected/kube-api-access-h5hf6 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35/volumes/kubernetes.io~projected/kube-api-access-rhnsq:{mountpoint:/var/lib/kubelet/pods/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35/volumes/kubernetes.io~projected/kube-api-access-rhnsq major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c91a35c-7933-4399-b7ff-4c1226d72d20/volumes/kubernetes.io~projected/kube-api-access-ls87f:{mountpoint:/var/lib/kubelet/pods/8c91a35c-7933-4399-b7ff-4c1226d72d20/volumes/kubernetes.io~projected/kube-api-access-ls87f major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c91a35c-7933-4399-b7ff-4c1226d72d20/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/8c91a35c-7933-4399-b7ff-4c1226d72d20/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c91a35c-7933-4399-b7ff-4c1226d72d20/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/8c91a35c-7933-4399-b7ff-4c1226d72d20/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97cf3735-d794-4628-9048-252f949317b9/volumes/kubernetes.io~projected/kube-api-access-2xpnl:{mountpoint:/var/lib/kubelet/pods/97cf3735-d794-4628-9048-252f949317b9/volumes/kubernetes.io~projected/kube-api-access-2xpnl major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97cf3735-d794-4628-9048-252f949317b9/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/97cf3735-d794-4628-9048-252f949317b9/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b4c0e6ed-9c41-4023-9e84-524635b9f1fb/volumes/kubernetes.io~projected/kube-api-access-tvd64:{mountpoint:/var/lib/kubelet/pods/b4c0e6ed-9c41-4023-9e84-524635b9f1fb/volumes/kubernetes.io~projected/kube-api-access-tvd64 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbf2e0b3-d31b-44ca-934c-3710f4262779/volumes/kubernetes.io~projected/kube-api-access-j2z5c:{mountpoint:/var/lib/kubelet/pods/bbf2e0b3-d31b-44ca-934c-3710f4262779/volumes/kubernetes.io~projected/kube-api-access-j2z5c major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbf2e0b3-d31b-44ca-934c-3710f4262779/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/bbf2e0b3-d31b-44ca-934c-3710f4262779/volumes/kubernetes.io~secret/metrics-certs major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd53d085-6474-45d6-bd59-825fea180a79/volumes/kubernetes.io~projected/kube-api-access-zgs94:{mountpoint:/var/lib/kubelet/pods/bd53d085-6474-45d6-bd59-825fea180a79/volumes/kubernetes.io~projected/kube-api-access-zgs94 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddebd541-4d2f-4861-9dd0-0ee66b03aaf2/volumes/kubernetes.io~projected/kube-api-access-m8shn:{mountpoint:/var/lib/kubelet/pods/ddebd541-4d2f-4861-9dd0-0ee66b03aaf2/volumes/kubernetes.io~projected/kube-api-access-m8shn major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddebd541-4d2f-4861-9dd0-0ee66b03aaf2/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/ddebd541-4d2f-4861-9dd0-0ee66b03aaf2/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:248 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddebd541-4d2f-4861-9dd0-0ee66b03aaf2/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/ddebd541-4d2f-4861-9dd0-0ee66b03aaf2/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4a19150-beb0-4289-91ee-b1668968e6ca/volumes/kubernetes.io~projected/kube-api-access-64hd7:{mountpoint:/var/lib/kubelet/pods/f4a19150-beb0-4289-91ee-b1668968e6ca/volumes/kubernetes.io~projected/kube-api-access-64hd7 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4a19150-beb0-4289-91ee-b1668968e6ca/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/f4a19150-beb0-4289-91ee-b1668968e6ca/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/72a31b017885c81ef876c00423111c076a9980976b807279fe6e1e90c2d6ead3/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/afa4598650949caa3b06e4cdebe94e77553187ef683cffaa0cbceb6d8b6927d9/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/2f3b0c7437cdaaca5b8cd1c090841edd290ec6173c488e4983ccd0b4aa1da1d8/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/e917493dba20715e0a2b842842a8ba43e9839df22fb4aa14fe4c3bb5ba9f8260/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/43e41e8a0c3641972fe34d093d915a2f9497364875c308f882bac9442531d324/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/be177d41fa31dee5fb8259922c4693dccc49c429d0b60c1acbca4b5d0d089c5d/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/af8eeb62ffc4f216a4388a25e1ac310e5023541e4fd2f7084b78f6593efba644/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/029af4152b049dad504a84115a4595b062a16d02b6b370a11409c0f1b0ab74cd/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/20239de9c9e25d1a448b38647eecd99442a831d00d321f4c2c787763e18ab7cf/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/5c779435200260fff1d7926c9b6492cf71d37f80e2722824b52b145808347433/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/1b97811d446cc15a34bc2d8588a50bcbc8d921ccbc84c4879be7937c801473a1/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/abb12e6c7a5cf0720a315a1858abbde44b51cf089cb289ad0fdd72ae9522a204/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-148:{mountpoint:/var/lib/containers/storage/overlay/032785c7530e190522a07e85203ad751395c45c11b81ed2b24a32e60186e70d9/merged major:0 minor:148 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/e3a2c16c80245db233c0a9ebefff8532bacb9a3b6146d07cdee6487277e560dc/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/6d14528ea96f59a47ea1592322d7de1e5852c47fe482741068038c842fee7473/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/3f4386f60b983752c4cd3706aa312a50f88c45a8f3b568c074984b79c7674091/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/236e1fe0997d001f2dc26375b8eec855751bfa9bde4b647d54096d8b98542026/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/cfb5f2fd0993d06b0cfd77709f54c9b7ef92cd9876084525fe4929e77b28f6b5/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/dec8fed7e508eb6e7058bae36d9dc707a82c11a41963bd03645c99772c5090ba/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/237b755b4169522e87a1148a451b6516670ebc5866abc59e37c95beefe7e5352/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/7db99ceb6dcab30a47f1bb16aa7d918509ba3253db9c65e1002ec54e402033c1/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/f8078fecac8cdb457dc7808ccb34cdbdbbd452d1d8b66688d0ce52bdccd95f9d/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/d36f1d1717aa4a21a1cd8f144904c7df85bce7fedfd2fb4c1530c7f251014f44/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/11a9e5441239c2eb78db01e2354b822a319f82843187cee850ebfc5e3fd8a787/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/6f4e43d8a03728ef4b2d09689ac17ea94999011cb1349f98c65c6424fda664c7/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/5274ab3287e7877e7052e00e336001abfb1b272fd2ef733fca7d4d6fc5ec4395/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/70f3a459a9a562caf17a24910c6780a57a5c94e012c49e124cc0d48949e1a0d9/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/7c7fd9fe73ac35f31d59fdd828f59bbec822f34a1b360ffb37d6c5fe14495881/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-244:{mountpoint:/var/lib/containers/storage/overlay/6bc26274c7e72330e6e0fcdc56f2b5d57306e7deb0cbc509ebb93334b0df0956/merged major:0 minor:244 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/d4d7984e3bedf5178914ef9c7e479ebd643276b05811a5ba3b807e59619f9585/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/cc51cf9b655159cb0b8767aa0c3a32c5efa87c0413edf3fa603c0b464605ac93/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/668c83c490cce71ef3571132dfb34fde9b94db45c32be720bcb0285f16507327/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/5ffdbdf5263f4b6fc96faa258f1f388bd0c9affc08e473c63e1b17a386cfebcc/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/1276a48dea64fd87ed24b34773cb7adf8209100640a1fe95ca2f96b3cd7ab631/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/df5e2406bf47889d1f47add8ede1911ac7ba3c1410d7da03d41125df09fbfa2c/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/592db74e5c3d5d0e1cddc4dd931ecc9ed7f3b1ef34c5a742d436c3a057065c8b/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/49376b210fd860610ed4fa919db28287de8a837d53f38634356faa7fc2e2d909/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/4cfc5345aca54f2b9e774af2a9c7e1c96bf8aeba83d49ecd682820461dbb3eea/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/92d5d4e28aca70624756156690597059e4ffb9de1d76f13a3957de6014a02ff5/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/a76ebb41d3f969d8a506061046a053560cfb3966f62d39be1c48cef7434d9c66/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/da00deb6b3fda2828752e6a77327cd553b0b6347a382df175684781fb65aca8c/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/4ef72710232ba713cf1a02ca18e88608f00e011fe9c8e3d4083202135d21ef8d/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-331:{mountpoint:/var/lib/containers/storage/overlay/699c3e4c351951a3795332c116b56d8daff05f48881110b66778c0d1588e66fb/merged major:0 minor:331 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/16f70c48604d8694595b8f524fcfe12602bb365aa4e9baf7c89177e9700d5e90/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/31936aee7c21c2ceb4ecc4cdc58dfa7178c1b5181569881fb662269bfe30f331/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/6f70ff7ede73571f92de6fb69a23c844ebd12eae85932aced5eacef506bdbcaa/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/cdf40fffad947349e31955294c99fae33881fe09e5b814d8d99809e801b16836/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-355:{mountpoint:/var/lib/containers/storage/overlay/8384e1d3172ef782278fad147a6db3138d1d87954db8d1bfa6936c4996d389af/merged major:0 minor:355 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/4229304ae39deb0d4872564efca073d62eac9466a3c20b1021de13dcb5f9c42d/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/2e9f48f41f275aa088e100ed51cd7b208db25b8eae22566d8d94e4dbbd6abedd/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-365:{mountpoint:/var/lib/containers/storage/overlay/d543186fd5b532b6f2b848d5b4aacbbd964d62415b79939d9c81ef2e816563d2/merged major:0 minor:365 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/fdcedc97f05813bd3cdfdbd9746d0f3829101dfad4dce2c87a5cda79e69e2b33/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/efc7aeb351d8e9d3a9050e0db41734c55590087786a8b374bdac93f0c6068542/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/f2638d752fea208e458711e7e991365bc85b8645ea5787934ab59140c6b9e04d/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/30c09b35728e0299fbf4ce134898fe6973cb79ad83d0badceb4d0b5ecd345fad/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/3c9cff3f972872995b44e19f50378e65666229c4f0366c77dafdc3856e87e1c7/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/353281fe7c45c8bd4917172d0afe02270e2b2a6128ae9b5670ce172ffbd841c6/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/b8f0fc7389623c1d4c37cc868d69c84745493ae64129d1abf73682d9f0f1d2d1/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/1569ae282dd9bfff88665c936080ae6b13605042fd213ce0d4a494aeaa0eb300/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/884eb43403d7919b42c822c9a25ca26109808fd931ab09039a5c43e567a20550/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/911724f9a55b0f95df584bde5760bfdb73390b3e3eddb6f6da9e446a21e9e575/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 31 12:32:29.356089 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.355038 5693 manager.go:217] Machine: {Timestamp:2026-10-31 12:32:29.354565066 +0000 UTC m=+0.062151352 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100082 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c2c03d63203b0a644ce9fad41a60c SystemUUID:ec2c2c03-d632-03b0-a644-ce9fad41a60c BootID:94656512-3c53-44de-aad9-07c262579e1a Filesystems:[{Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b091d91-47bc-4480-85b2-e23a8139df66/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/8c91a35c-7933-4399-b7ff-4c1226d72d20/volumes/kubernetes.io~projected/kube-api-access-ls87f DeviceMajor:0 DeviceMinor:241 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/25a02fd6-8ab0-4cca-94cd-d4082f874170/volumes/kubernetes.io~projected/kube-api-access-srn9m DeviceMajor:0 DeviceMinor:301 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5d3565a5-d07a-4dc0-abd1-e697573baf22/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/97cf3735-d794-4628-9048-252f949317b9/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0f343e95-e397-49aa-b023-98337d3be3ff/volumes/kubernetes.io~projected/kube-api-access-kpx9x DeviceMajor:0 DeviceMinor:352 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a32690afe6d52b867aceffb58e48753674527d54678cbb3cdbb2b441b7c99e8c/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-355 DeviceMajor:0 DeviceMinor:355 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8c91a35c-7933-4399-b7ff-4c1226d72d20/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:236 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35/volumes/kubernetes.io~projected/kube-api-access-rhnsq DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8c91a35c-7933-4399-b7ff-4c1226d72d20/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:240 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e6eb922f4a0cd05a8f419575ee3f34f4ac887c9b83ebc2138465e7b10d647228/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4b6af1c5-6045-4e4c-aa87-2473d19bd000/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:282 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/97cf3735-d794-4628-9048-252f949317b9/volumes/kubernetes.io~projected/kube-api-access-2xpnl DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a633f11a094758484648b16d5585aebc43099d3f40a08aa8223d97224bf41d0e/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-148 DeviceMajor:0 DeviceMinor:148 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/25a02fd6-8ab0-4cca-94cd-d4082f874170/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:300 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0b35abf8cdb506dcaacb590ad330fb6cbbaee3964124e6b0ca499b097254feb4/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d80f43156c9865079efe4848eefacddbdee81a7f18ec7222d77c5723944c269a/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f4a19150-beb0-4289-91ee-b1668968e6ca/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-244 DeviceMajor:0 DeviceMinor:244 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3015bfb0-80c8-4c01-9053-85e5ff2318fb/volumes/kubernetes.io~projected/kube-api-access-5d92p DeviceMajor:0 DeviceMinor:307 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/02c6d8f5-97ff-41e0-8e38-71e1b0c87f54/volumes/kubernetes.io~projected/kube-api-access-ph9jm DeviceMajor:0 DeviceMinor:330 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/b4c0e6ed-9c41-4023-9e84-524635b9f1fb/volumes/kubernetes.io~projected/kube-api-access-tvd64 DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f4a19150-beb0-4289-91ee-b1668968e6ca/volumes/kubernetes.io~projected/kube-api-access-64hd7 DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ddebd541-4d2f-4861-9dd0-0ee66b03aaf2/volumes/kubernetes.io~projected/kube-api-access-m8shn DeviceMajor:0 DeviceMinor:253 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4b6af1c5-6045-4e4c-aa87-2473d19bd000/volumes/kubernetes.io~projected/kube-api-access-rprnd DeviceMajor:0 DeviceMinor:283 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4334c12d573521f95b214022be018ae2961732b1ee6e21a1d7c07ef1b08b3dce/userdata/shm DeviceMajor:0 DeviceMinor:353 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4b6af1c5-6045-4e4c-aa87-2473d19bd000/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:277 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:325 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cc809f08fd1b1ec8bba0f6df66aee099d71617ec4e73c31c6b501d2db29a95c7/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0b091d91-47bc-4480-85b2-e23a8139df66/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/43421968-0e5b-4e27-be79-104aae5f72e0/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bbf2e0b3-d31b-44ca-934c-3710f4262779/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:320 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/14dc1e6d-70e1-4ba5-bffb-d9299433756d/volumes/kubernetes.io~projected/kube-api-access-sd2xl DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2371284ce749382e13be426c9d83dd544dfdcc771669afc0a19abb73b59a2845/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7f9fad3cf08ec8fb0a264377931414fafbf489a06b13f1a62d1867691c252545/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2a2f94ca992e3771f5713cbd3ada4bc4cab041afa0a1eb45530fe3ee42469c15/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9af65c148112ebb9dd1268dbe8c909387f2ca55b28da2b69649c5428e5be0e76/userdata/shm DeviceMajor:0 DeviceMinor:326 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-331 DeviceMajor:0 DeviceMinor:331 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0f343e95-e397-49aa-b023-98337d3be3ff/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:347 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4242a5b80f9bf08c64de825adaa8d447534331d8362cdedc8dbf2fc82d20f911/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/86e20428c0c192b9f1bdf2e62481619c797e84aeffec2ec17676284c65292fe7/userdata/shm DeviceMajor:0 DeviceMinor:254 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/da22f1b02aab199a2fd5d6f85284e273cd2aed5d3d2946debb29989dc43335d2/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b391fcf82f11c31b74103e72e0102b9d81913ae13eb810d71671a5954464aad3/userdata/shm DeviceMajor:0 DeviceMinor:328 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3504f1ff87a9423da711361f7da6db57c5bd88c69326687759db34467ddcfb1b/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2449fb987accd7f8129f3ad9d4ad0d8aeb202aea937d539aca589a09d925778e/userdata/shm DeviceMajor:0 DeviceMinor:335 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0f343e95-e397-49aa-b023-98337d3be3ff/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:351 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5d3565a5-d07a-4dc0-abd1-e697573baf22/volumes/kubernetes.io~projected/kube-api-access-h5hf6 DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3b8f659ef7a491710aa5a36a9d4f2c3b2469a212d3bb308827483dda8be8b556/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/98c00b91a1a67a918bfc7bdefc29cf5f033528396d20cfa2be189cc550975729/userdata/shm DeviceMajor:0 DeviceMinor:302 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bbf2e0b3-d31b-44ca-934c-3710f4262779/volumes/kubernetes.io~projected/kube-api-access-j2z5c DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3015bfb0-80c8-4c01-9053-85e5ff2318fb/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:306 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/26913ad4-6613-4f3f-9819-9be238088c0a/volumes/kubernetes.io~projected/kube-api-access-779z6 DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0b091d91-47bc-4480-85b2-e23a8139df66/volumes/kubernetes.io~projected/kube-api-access-4n79b DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/24e17633-30a9-4410-b000-2fbb97202951/volumes/kubernetes.io~projected/kube-api-access-zhppb DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/45870e34-5e37-443a-826b-81c9c031051c/volumes/kubernetes.io~projected/kube-api-access-kdlgx DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ddebd541-4d2f-4861-9dd0-0ee66b03aaf2/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:252 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-365 DeviceMajor:0 DeviceMinor:365 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/5d3565a5-d07a-4dc0-abd1-e697573baf22/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bd53d085-6474-45d6-bd59-825fea180a79/volumes/kubernetes.io~projected/kube-api-access-zgs94 DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f763f3a95d574aac7ca59fd0a9b3957bb56261dce7e237013b93f0c6cd662978/userdata/shm DeviceMajor:0 DeviceMinor:284 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/45870e34-5e37-443a-826b-81c9c031051c/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a1787850d6b84588d61e49e4b5d640845f0c97caebe94b71026f54cd02be1a4/userdata/shm DeviceMajor:0 DeviceMinor:308 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/af8a3287d760c49bab4cd6f000a6ccf86a22d13d3c19f124b2e2654ed0084904/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ddebd541-4d2f-4861-9dd0-0ee66b03aaf2/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:248 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/25a02fd6-8ab0-4cca-94cd-d4082f874170/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:299 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/25a02fd6-8ab0-4cca-94cd-d4082f874170/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:295 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0a1787850d6b845 MacAddress:fa:c5:bc:52:21:36 Speed:10000 Mtu:8901} {Name:2449fb987accd7f MacAddress:ca:5d:19:89:d9:48 Speed:10000 Mtu:8901} {Name:4242a5b80f9bf08 MacAddress:ce:12:d6:dd:24:6a Speed:10000 Mtu:8901} {Name:4334c12d573521f MacAddress:02:2b:bf:ab:c1:ad Speed:10000 Mtu:8901} {Name:7f9fad3cf08ec8f MacAddress:ea:4e:77:b9:13:d0 Speed:10000 Mtu:8901} {Name:86e20428c0c192b MacAddress:5a:2f:15:85:2b:63 Speed:10000 Mtu:8901} {Name:98c00b91a1a67a9 MacAddress:6a:b1:54:34:fb:2e Speed:10000 Mtu:8901} {Name:9af65c148112ebb MacAddress:b2:74:fd:68:40:b4 Speed:10000 Mtu:8901} {Name:a633f11a0947584 MacAddress:ee:58:d8:c0:a0:bf Speed:10000 Mtu:8901} {Name:b391fcf82f11c31 MacAddress:ca:76:06:c9:ac:4d Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e0:ac:7d:57 Speed:0 Mtu:9001} {Name:br-int MacAddress:e2:ec:75:98:d6:7c Speed:0 Mtu:8901} {Name:e6eb922f4a0cd05 MacAddress:ea:6e:f3:56:c9:1e Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:e0:ac:7d:57 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:16:c2:a4:6c:d9:65 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:82:4c:9f:e9:93:28 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 12:32:29.356089 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.355428 5693 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 12:32:29.356089 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.355475 5693 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 12:32:29.356089 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.355835 5693 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 12:32:29.356089 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.355859 5693 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-157.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 12:32:29.356089 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.355991 5693 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 12:32:29.356089 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.355999 5693 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 12:32:29.356089 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.356010 5693 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 12:32:29.356089 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.356023 5693 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 12:32:29.356450 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.356117 5693 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 12:32:29.356450 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.356275 5693 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 12:32:29.356450 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.356403 5693 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 12:32:29.356450 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.356442 5693 kubelet.go:493] "Attempting to sync node with API server" Oct 31 12:32:29.356624 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.356461 5693 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 12:32:29.356624 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.356472 5693 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 12:32:29.356624 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.356481 5693 kubelet.go:405] "Adding apiserver pod source" Oct 31 12:32:29.356624 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.356488 5693 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 12:32:29.357507 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.357491 5693 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 12:32:29.357542 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.357509 5693 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 12:32:29.358106 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.358082 5693 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 12:32:29.360509 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.360494 5693 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 12:32:29.360509 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.360508 5693 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 12:32:29.360623 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.360613 5693 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 12:32:29.360844 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.360833 5693 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 12:32:29.360887 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.360863 5693 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 12:32:29.360994 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.360981 5693 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 12:32:29.361026 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.361007 5693 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 12:32:29.361026 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.361020 5693 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 12:32:29.361078 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.361033 5693 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 12:32:29.361078 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.361045 5693 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 12:32:29.361078 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.361058 5693 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 12:32:29.361211 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.361083 5693 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 12:32:29.361211 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.361097 5693 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 12:32:29.361211 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.361112 5693 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 12:32:29.361211 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.361124 5693 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 12:32:29.361211 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.361140 5693 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 12:32:29.361211 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.361157 5693 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 12:32:29.361211 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.361202 5693 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 12:32:29.361399 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.361215 5693 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 12:32:29.364857 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.364843 5693 server.go:1263] "Started kubelet" Oct 31 12:32:29.364983 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.364937 5693 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 12:32:29.365037 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.364943 5693 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 12:32:29.365097 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.365084 5693 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 12:32:29.365617 ip-10-0-131-157 systemd[1]: Started Kubernetes Kubelet. Oct 31 12:32:29.366282 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.366258 5693 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 12:32:29.366394 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.366314 5693 server.go:317] "Adding debug handlers to kubelet server" Oct 31 12:32:29.373237 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.370673 5693 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 12:32:29.373237 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.370701 5693 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 12:32:29.373237 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.370780 5693 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 12:32:29.373237 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.370795 5693 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 12:32:29.373237 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.370760 5693 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 12:26:18 +0000 UTC" deadline="2028-04-06 00:34:57.455641024 +0000 UTC" Oct 31 12:32:29.373237 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.371020 5693 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12540h2m28.084627453s" Oct 31 12:32:29.373237 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.371062 5693 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 12:32:29.373675 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.373523 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f343e95-e397-49aa-b023-98337d3be3ff" volumeName="kubernetes.io/projected/0f343e95-e397-49aa-b023-98337d3be3ff-kube-api-access-kpx9x" seLinuxMountContext="" Oct 31 12:32:29.373675 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.373553 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43421968-0e5b-4e27-be79-104aae5f72e0" volumeName="kubernetes.io/configmap/43421968-0e5b-4e27-be79-104aae5f72e0-konnectivity-ca" seLinuxMountContext="" Oct 31 12:32:29.373675 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.373566 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c91a35c-7933-4399-b7ff-4c1226d72d20" volumeName="kubernetes.io/secret/8c91a35c-7933-4399-b7ff-4c1226d72d20-console-serving-cert" seLinuxMountContext="" Oct 31 12:32:29.373675 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.373575 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd53d085-6474-45d6-bd59-825fea180a79" volumeName="kubernetes.io/projected/bd53d085-6474-45d6-bd59-825fea180a79-kube-api-access-zgs94" seLinuxMountContext="" Oct 31 12:32:29.373675 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.373585 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddebd541-4d2f-4861-9dd0-0ee66b03aaf2" volumeName="kubernetes.io/secret/ddebd541-4d2f-4861-9dd0-0ee66b03aaf2-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 12:32:29.373675 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.373609 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f343e95-e397-49aa-b023-98337d3be3ff" volumeName="kubernetes.io/configmap/0f343e95-e397-49aa-b023-98337d3be3ff-console-config" seLinuxMountContext="" Oct 31 12:32:29.373675 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.373617 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f343e95-e397-49aa-b023-98337d3be3ff" volumeName="kubernetes.io/secret/0f343e95-e397-49aa-b023-98337d3be3ff-console-oauth-config" seLinuxMountContext="" Oct 31 12:32:29.373675 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.373624 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45870e34-5e37-443a-826b-81c9c031051c" volumeName="kubernetes.io/projected/45870e34-5e37-443a-826b-81c9c031051c-kube-api-access-kdlgx" seLinuxMountContext="" Oct 31 12:32:29.373675 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.373633 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b6af1c5-6045-4e4c-aa87-2473d19bd000" volumeName="kubernetes.io/configmap/4b6af1c5-6045-4e4c-aa87-2473d19bd000-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 31 12:32:29.373675 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.373661 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b6af1c5-6045-4e4c-aa87-2473d19bd000" volumeName="kubernetes.io/secret/4b6af1c5-6045-4e4c-aa87-2473d19bd000-node-exporter-tls" seLinuxMountContext="" Oct 31 12:32:29.373675 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.373673 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d3565a5-d07a-4dc0-abd1-e697573baf22" volumeName="kubernetes.io/projected/5d3565a5-d07a-4dc0-abd1-e697573baf22-kube-api-access-h5hf6" seLinuxMountContext="" Oct 31 12:32:29.373675 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.373683 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c75b6d0-3fc9-4e2e-aba2-f464a48cab35" volumeName="kubernetes.io/projected/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-kube-api-access-rhnsq" seLinuxMountContext="" Oct 31 12:32:29.374175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.373690 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f343e95-e397-49aa-b023-98337d3be3ff" volumeName="kubernetes.io/configmap/0f343e95-e397-49aa-b023-98337d3be3ff-service-ca" seLinuxMountContext="" Oct 31 12:32:29.374175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.373698 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b6af1c5-6045-4e4c-aa87-2473d19bd000" volumeName="kubernetes.io/secret/4b6af1c5-6045-4e4c-aa87-2473d19bd000-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 12:32:29.374175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.373707 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4a19150-beb0-4289-91ee-b1668968e6ca" volumeName="kubernetes.io/empty-dir/f4a19150-beb0-4289-91ee-b1668968e6ca-tmp-dir" seLinuxMountContext="" Oct 31 12:32:29.374175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.373714 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b091d91-47bc-4480-85b2-e23a8139df66" volumeName="kubernetes.io/configmap/0b091d91-47bc-4480-85b2-e23a8139df66-ovnkube-script-lib" seLinuxMountContext="" Oct 31 12:32:29.374175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.373722 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24e17633-30a9-4410-b000-2fbb97202951" volumeName="kubernetes.io/projected/24e17633-30a9-4410-b000-2fbb97202951-kube-api-access-zhppb" seLinuxMountContext="" Oct 31 12:32:29.374175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.373731 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b6af1c5-6045-4e4c-aa87-2473d19bd000" volumeName="kubernetes.io/empty-dir/4b6af1c5-6045-4e4c-aa87-2473d19bd000-node-exporter-textfile" seLinuxMountContext="" Oct 31 12:32:29.374175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.373741 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c75b6d0-3fc9-4e2e-aba2-f464a48cab35" volumeName="kubernetes.io/configmap/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 12:32:29.374175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.373752 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c91a35c-7933-4399-b7ff-4c1226d72d20" volumeName="kubernetes.io/configmap/8c91a35c-7933-4399-b7ff-4c1226d72d20-oauth-serving-cert" seLinuxMountContext="" Oct 31 12:32:29.374175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.373764 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c91a35c-7933-4399-b7ff-4c1226d72d20" volumeName="kubernetes.io/configmap/8c91a35c-7933-4399-b7ff-4c1226d72d20-service-ca" seLinuxMountContext="" Oct 31 12:32:29.374175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.373792 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4c0e6ed-9c41-4023-9e84-524635b9f1fb" volumeName="kubernetes.io/projected/b4c0e6ed-9c41-4023-9e84-524635b9f1fb-kube-api-access-tvd64" seLinuxMountContext="" Oct 31 12:32:29.374175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.373805 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddebd541-4d2f-4861-9dd0-0ee66b03aaf2" volumeName="kubernetes.io/projected/ddebd541-4d2f-4861-9dd0-0ee66b03aaf2-kube-api-access-m8shn" seLinuxMountContext="" Oct 31 12:32:29.374175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.373818 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b091d91-47bc-4480-85b2-e23a8139df66" volumeName="kubernetes.io/configmap/0b091d91-47bc-4480-85b2-e23a8139df66-env-overrides" seLinuxMountContext="" Oct 31 12:32:29.374175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.373833 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f343e95-e397-49aa-b023-98337d3be3ff" volumeName="kubernetes.io/secret/0f343e95-e397-49aa-b023-98337d3be3ff-console-serving-cert" seLinuxMountContext="" Oct 31 12:32:29.374175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.373845 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b6af1c5-6045-4e4c-aa87-2473d19bd000" volumeName="kubernetes.io/configmap/4b6af1c5-6045-4e4c-aa87-2473d19bd000-metrics-client-ca" seLinuxMountContext="" Oct 31 12:32:29.374175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.373857 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97cf3735-d794-4628-9048-252f949317b9" volumeName="kubernetes.io/projected/97cf3735-d794-4628-9048-252f949317b9-kube-api-access-2xpnl" seLinuxMountContext="" Oct 31 12:32:29.374175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.373869 5693 factory.go:55] Registering systemd factory Oct 31 12:32:29.374175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.373888 5693 factory.go:223] Registration of the systemd container factory successfully Oct 31 12:32:29.374175 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:32:29.373915 5693 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-157.us-east-2.compute.internal\" not found" Oct 31 12:32:29.374175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.373872 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02c6d8f5-97ff-41e0-8e38-71e1b0c87f54" volumeName="kubernetes.io/projected/02c6d8f5-97ff-41e0-8e38-71e1b0c87f54-kube-api-access-ph9jm" seLinuxMountContext="" Oct 31 12:32:29.374175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.373958 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14dc1e6d-70e1-4ba5-bffb-d9299433756d" volumeName="kubernetes.io/projected/14dc1e6d-70e1-4ba5-bffb-d9299433756d-kube-api-access-sd2xl" seLinuxMountContext="" Oct 31 12:32:29.374175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.373981 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c91a35c-7933-4399-b7ff-4c1226d72d20" volumeName="kubernetes.io/secret/8c91a35c-7933-4399-b7ff-4c1226d72d20-console-oauth-config" seLinuxMountContext="" Oct 31 12:32:29.374175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.373994 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4a19150-beb0-4289-91ee-b1668968e6ca" volumeName="kubernetes.io/projected/f4a19150-beb0-4289-91ee-b1668968e6ca-kube-api-access-64hd7" seLinuxMountContext="" Oct 31 12:32:29.374175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374007 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f343e95-e397-49aa-b023-98337d3be3ff" volumeName="kubernetes.io/configmap/0f343e95-e397-49aa-b023-98337d3be3ff-oauth-serving-cert" seLinuxMountContext="" Oct 31 12:32:29.374175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374019 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25a02fd6-8ab0-4cca-94cd-d4082f874170" volumeName="kubernetes.io/projected/25a02fd6-8ab0-4cca-94cd-d4082f874170-kube-api-access-srn9m" seLinuxMountContext="" Oct 31 12:32:29.374175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374031 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c91a35c-7933-4399-b7ff-4c1226d72d20" volumeName="kubernetes.io/empty-dir/8c91a35c-7933-4399-b7ff-4c1226d72d20-tmp" seLinuxMountContext="" Oct 31 12:32:29.374175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374044 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97cf3735-d794-4628-9048-252f949317b9" volumeName="kubernetes.io/secret/97cf3735-d794-4628-9048-252f949317b9-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 31 12:32:29.374175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374055 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd53d085-6474-45d6-bd59-825fea180a79" volumeName="kubernetes.io/configmap/bd53d085-6474-45d6-bd59-825fea180a79-cni-binary-copy" seLinuxMountContext="" Oct 31 12:32:29.374175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374068 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4a19150-beb0-4289-91ee-b1668968e6ca" volumeName="kubernetes.io/secret/f4a19150-beb0-4289-91ee-b1668968e6ca-metrics-tls" seLinuxMountContext="" Oct 31 12:32:29.374175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374081 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25a02fd6-8ab0-4cca-94cd-d4082f874170" volumeName="kubernetes.io/secret/25a02fd6-8ab0-4cca-94cd-d4082f874170-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 31 12:32:29.374175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374092 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d3565a5-d07a-4dc0-abd1-e697573baf22" volumeName="kubernetes.io/empty-dir/5d3565a5-d07a-4dc0-abd1-e697573baf22-tmp" seLinuxMountContext="" Oct 31 12:32:29.374175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374103 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c75b6d0-3fc9-4e2e-aba2-f464a48cab35" volumeName="kubernetes.io/configmap/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-cni-binary-copy" seLinuxMountContext="" Oct 31 12:32:29.374175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374115 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4a19150-beb0-4289-91ee-b1668968e6ca" volumeName="kubernetes.io/configmap/f4a19150-beb0-4289-91ee-b1668968e6ca-config-volume" seLinuxMountContext="" Oct 31 12:32:29.374175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374127 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b091d91-47bc-4480-85b2-e23a8139df66" volumeName="kubernetes.io/configmap/0b091d91-47bc-4480-85b2-e23a8139df66-ovnkube-config" seLinuxMountContext="" Oct 31 12:32:29.374175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374138 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14dc1e6d-70e1-4ba5-bffb-d9299433756d" volumeName="kubernetes.io/configmap/14dc1e6d-70e1-4ba5-bffb-d9299433756d-serviceca" seLinuxMountContext="" Oct 31 12:32:29.374175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374151 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d3565a5-d07a-4dc0-abd1-e697573baf22" volumeName="kubernetes.io/empty-dir/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-tuned" seLinuxMountContext="" Oct 31 12:32:29.374175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374163 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f343e95-e397-49aa-b023-98337d3be3ff" volumeName="kubernetes.io/configmap/0f343e95-e397-49aa-b023-98337d3be3ff-trusted-ca-bundle" seLinuxMountContext="" Oct 31 12:32:29.374175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374175 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f343e95-e397-49aa-b023-98337d3be3ff" volumeName="kubernetes.io/empty-dir/0f343e95-e397-49aa-b023-98337d3be3ff-tmp" seLinuxMountContext="" Oct 31 12:32:29.374175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374187 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c75b6d0-3fc9-4e2e-aba2-f464a48cab35" volumeName="kubernetes.io/configmap/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 12:32:29.375481 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374198 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b091d91-47bc-4480-85b2-e23a8139df66" volumeName="kubernetes.io/secret/0b091d91-47bc-4480-85b2-e23a8139df66-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 12:32:29.375481 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374210 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45870e34-5e37-443a-826b-81c9c031051c" volumeName="kubernetes.io/secret/45870e34-5e37-443a-826b-81c9c031051c-cert" seLinuxMountContext="" Oct 31 12:32:29.375481 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374221 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbf2e0b3-d31b-44ca-934c-3710f4262779" volumeName="kubernetes.io/projected/bbf2e0b3-d31b-44ca-934c-3710f4262779-kube-api-access-j2z5c" seLinuxMountContext="" Oct 31 12:32:29.375481 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374233 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddebd541-4d2f-4861-9dd0-0ee66b03aaf2" volumeName="kubernetes.io/secret/ddebd541-4d2f-4861-9dd0-0ee66b03aaf2-prometheus-operator-tls" seLinuxMountContext="" Oct 31 12:32:29.375481 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374244 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3015bfb0-80c8-4c01-9053-85e5ff2318fb" volumeName="kubernetes.io/projected/3015bfb0-80c8-4c01-9053-85e5ff2318fb-kube-api-access-5d92p" seLinuxMountContext="" Oct 31 12:32:29.375481 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374258 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3015bfb0-80c8-4c01-9053-85e5ff2318fb" volumeName="kubernetes.io/secret/3015bfb0-80c8-4c01-9053-85e5ff2318fb-monitoring-plugin-cert" seLinuxMountContext="" Oct 31 12:32:29.375481 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374269 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c91a35c-7933-4399-b7ff-4c1226d72d20" volumeName="kubernetes.io/projected/8c91a35c-7933-4399-b7ff-4c1226d72d20-kube-api-access-ls87f" seLinuxMountContext="" Oct 31 12:32:29.375481 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374281 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97cf3735-d794-4628-9048-252f949317b9" volumeName="kubernetes.io/configmap/97cf3735-d794-4628-9048-252f949317b9-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 31 12:32:29.375481 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374291 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97cf3735-d794-4628-9048-252f949317b9" volumeName="kubernetes.io/empty-dir/97cf3735-d794-4628-9048-252f949317b9-data-volume" seLinuxMountContext="" Oct 31 12:32:29.375481 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374311 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbf2e0b3-d31b-44ca-934c-3710f4262779" volumeName="kubernetes.io/secret/bbf2e0b3-d31b-44ca-934c-3710f4262779-metrics-certs" seLinuxMountContext="" Oct 31 12:32:29.375481 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374324 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddebd541-4d2f-4861-9dd0-0ee66b03aaf2" volumeName="kubernetes.io/configmap/ddebd541-4d2f-4861-9dd0-0ee66b03aaf2-metrics-client-ca" seLinuxMountContext="" Oct 31 12:32:29.375481 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374337 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b091d91-47bc-4480-85b2-e23a8139df66" volumeName="kubernetes.io/projected/0b091d91-47bc-4480-85b2-e23a8139df66-kube-api-access-4n79b" seLinuxMountContext="" Oct 31 12:32:29.375481 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374345 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25a02fd6-8ab0-4cca-94cd-d4082f874170" volumeName="kubernetes.io/empty-dir/25a02fd6-8ab0-4cca-94cd-d4082f874170-audit-log" seLinuxMountContext="" Oct 31 12:32:29.375481 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374353 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26913ad4-6613-4f3f-9819-9be238088c0a" volumeName="kubernetes.io/empty-dir/26913ad4-6613-4f3f-9819-9be238088c0a-tmp-dir" seLinuxMountContext="" Oct 31 12:32:29.375481 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374361 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25a02fd6-8ab0-4cca-94cd-d4082f874170" volumeName="kubernetes.io/configmap/25a02fd6-8ab0-4cca-94cd-d4082f874170-metrics-server-audit-profiles" seLinuxMountContext="" Oct 31 12:32:29.375481 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374368 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25a02fd6-8ab0-4cca-94cd-d4082f874170" volumeName="kubernetes.io/secret/25a02fd6-8ab0-4cca-94cd-d4082f874170-secret-metrics-server-tls" seLinuxMountContext="" Oct 31 12:32:29.375481 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374376 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4c0e6ed-9c41-4023-9e84-524635b9f1fb" volumeName="kubernetes.io/configmap/b4c0e6ed-9c41-4023-9e84-524635b9f1fb-iptables-alerter-script" seLinuxMountContext="" Oct 31 12:32:29.375481 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374410 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd53d085-6474-45d6-bd59-825fea180a79" volumeName="kubernetes.io/configmap/bd53d085-6474-45d6-bd59-825fea180a79-multus-daemon-config" seLinuxMountContext="" Oct 31 12:32:29.375481 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374419 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25a02fd6-8ab0-4cca-94cd-d4082f874170" volumeName="kubernetes.io/secret/25a02fd6-8ab0-4cca-94cd-d4082f874170-client-ca-bundle" seLinuxMountContext="" Oct 31 12:32:29.375481 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374428 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43421968-0e5b-4e27-be79-104aae5f72e0" volumeName="kubernetes.io/secret/43421968-0e5b-4e27-be79-104aae5f72e0-agent-certs" seLinuxMountContext="" Oct 31 12:32:29.375481 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374435 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c91a35c-7933-4399-b7ff-4c1226d72d20" volumeName="kubernetes.io/configmap/8c91a35c-7933-4399-b7ff-4c1226d72d20-console-config" seLinuxMountContext="" Oct 31 12:32:29.375481 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374443 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25a02fd6-8ab0-4cca-94cd-d4082f874170" volumeName="kubernetes.io/configmap/25a02fd6-8ab0-4cca-94cd-d4082f874170-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 31 12:32:29.375481 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374452 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26913ad4-6613-4f3f-9819-9be238088c0a" volumeName="kubernetes.io/projected/26913ad4-6613-4f3f-9819-9be238088c0a-kube-api-access-779z6" seLinuxMountContext="" Oct 31 12:32:29.375481 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374461 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b6af1c5-6045-4e4c-aa87-2473d19bd000" volumeName="kubernetes.io/projected/4b6af1c5-6045-4e4c-aa87-2473d19bd000-kube-api-access-rprnd" seLinuxMountContext="" Oct 31 12:32:29.375481 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374469 5693 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a" volumeName="kubernetes.io/secret/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a-original-pull-secret" seLinuxMountContext="" Oct 31 12:32:29.375481 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374476 5693 reconstruct.go:97] "Volume reconstruction finished" Oct 31 12:32:29.375481 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374481 5693 reconciler.go:29] "Reconciler: start to sync state" Oct 31 12:32:29.375481 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374833 5693 factory.go:153] Registering CRI-O factory Oct 31 12:32:29.375481 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374848 5693 factory.go:223] Registration of the crio container factory successfully Oct 31 12:32:29.375481 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374903 5693 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 12:32:29.375481 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374921 5693 factory.go:103] Registering Raw factory Oct 31 12:32:29.375481 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.374936 5693 manager.go:1199] Started watching for new ooms in manager Oct 31 12:32:29.375481 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.375309 5693 manager.go:319] Starting recovery of all containers Oct 31 12:32:29.380812 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.380092 5693 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:32:29.380812 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.380184 5693 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:32:29.384317 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.382480 5693 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 12:32:29.384317 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.383694 5693 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:32:29.398427 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.398383 5693 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 12:32:29.400030 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.400008 5693 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 12:32:29.400030 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.400029 5693 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 12:32:29.400169 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.400069 5693 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 12:32:29.400169 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:32:29.400109 5693 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 12:32:29.403929 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.403904 5693 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:32:29.406505 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.406474 5693 generic.go:356] "Generic (PLEG): container finished" podID="4b6af1c5-6045-4e4c-aa87-2473d19bd000" containerID="98658f29d2c7d3a00b3307cbb1941b286c18083cc24ffdb1212124d9ecd17777" exitCode=0 Oct 31 12:32:29.431348 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.431314 5693 generic.go:356] "Generic (PLEG): container finished" podID="8a68def89a8e2f4b1d88fecf85bfdb5e" containerID="50f6ee2338bf62416e92e9e2bfe9d4c8bdd96dc04ed17c57a22d6fb2fdfad98f" exitCode=0 Oct 31 12:32:29.439250 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.439223 5693 generic.go:356] "Generic (PLEG): container finished" podID="8c75b6d0-3fc9-4e2e-aba2-f464a48cab35" containerID="c3f14f59145ed299d794b96da8d04940f13a270c0db13c7b72220643aafbe741" exitCode=0 Oct 31 12:32:29.439250 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.439244 5693 generic.go:356] "Generic (PLEG): container finished" podID="8c75b6d0-3fc9-4e2e-aba2-f464a48cab35" containerID="e540611cac188a7b5c4e6d95d8f3d5a6a39ca151cb1a75c651ce67cd5473ec33" exitCode=0 Oct 31 12:32:29.439250 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.439253 5693 generic.go:356] "Generic (PLEG): container finished" podID="8c75b6d0-3fc9-4e2e-aba2-f464a48cab35" containerID="da49b8aafa449ef6ed4ba8a2af64cc32422fbec861ce60e20b8feb07b6cb4993" exitCode=0 Oct 31 12:32:29.439250 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.439259 5693 generic.go:356] "Generic (PLEG): container finished" podID="8c75b6d0-3fc9-4e2e-aba2-f464a48cab35" containerID="9aa4f45abe694c91bf2614b64265805783a11ca8d562997fb51744b32f27ae94" exitCode=0 Oct 31 12:32:29.439250 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.439264 5693 generic.go:356] "Generic (PLEG): container finished" podID="8c75b6d0-3fc9-4e2e-aba2-f464a48cab35" containerID="dc9ba54ab45f379e93ca2b9ef1c767af2ee2e6e9b08039345a6fbb992b80ff00" exitCode=0 Oct 31 12:32:29.439250 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.439269 5693 generic.go:356] "Generic (PLEG): container finished" podID="8c75b6d0-3fc9-4e2e-aba2-f464a48cab35" containerID="39b31960d2335ff6536593a02304d9fb9762569b0418f0c120601d3fff86cae1" exitCode=0 Oct 31 12:32:29.469220 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.469196 5693 manager.go:324] Recovery completed Oct 31 12:32:29.488429 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.488361 5693 cpu_manager.go:226] "Starting" policy="none" Oct 31 12:32:29.488429 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.488381 5693 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 12:32:29.488429 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.488398 5693 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 12:32:29.488612 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.488530 5693 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 12:32:29.488612 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.488541 5693 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 12:32:29.488612 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.488560 5693 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 12:32:29.488612 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.488565 5693 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 12:32:29.488612 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.488571 5693 policy_none.go:50] "Start" Oct 31 12:32:29.488612 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.488579 5693 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 12:32:29.488612 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.488587 5693 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 12:32:29.488821 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.488725 5693 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 12:32:29.488821 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.488733 5693 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 12:32:29.488821 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.488739 5693 policy_none.go:44] "Start" Oct 31 12:32:29.495164 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.495130 5693 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 12:32:29.497760 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.497742 5693 manager.go:341] "Starting Device Plugin manager" Oct 31 12:32:29.498281 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:32:29.497769 5693 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 12:32:29.498338 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.498285 5693 server.go:87] "Starting device plugin registration server" Oct 31 12:32:29.498548 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.498538 5693 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 12:32:29.498583 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.498550 5693 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 12:32:29.498696 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.498674 5693 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 12:32:29.498806 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.498775 5693 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 12:32:29.498878 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.498814 5693 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 12:32:29.498878 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.498821 5693 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 12:32:29.498968 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.498890 5693 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T12:32:29.498803634Z","UUID":"dbd0460e-334d-4f33-8141-5b9657f04022","Handler":null,"Name":"","Endpoint":""} Oct 31 12:32:29.500113 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.500094 5693 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 12:32:29.500215 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.500131 5693 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 12:32:29.500273 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:32:29.500241 5693 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 12:32:29.500338 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.500266 5693 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal"] Oct 31 12:32:29.500899 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.500866 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" event={"ID":"8a68def89a8e2f4b1d88fecf85bfdb5e","Type":"ContainerStarted","Data":"d295a68fcf3ba939e56428b90189cdde34a8f545d27bec40113d5649d0189664"} Oct 31 12:32:29.500978 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.500904 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" event={"ID":"8a68def89a8e2f4b1d88fecf85bfdb5e","Type":"ContainerDied","Data":"50f6ee2338bf62416e92e9e2bfe9d4c8bdd96dc04ed17c57a22d6fb2fdfad98f"} Oct 31 12:32:29.500978 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.500917 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" event={"ID":"8a68def89a8e2f4b1d88fecf85bfdb5e","Type":"ContainerStarted","Data":"0b35abf8cdb506dcaacb590ad330fb6cbbaee3964124e6b0ca499b097254feb4"} Oct 31 12:32:29.500978 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.500965 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal" event={"ID":"72de75e4a93b82a0133be1d6ed9e0a51","Type":"ContainerStarted","Data":"eae2685bf8a0bc14141e2122daca3e98f4a2d00dcf5a141c15c4d232cbb482f8"} Oct 31 12:32:29.501191 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.500980 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal" event={"ID":"72de75e4a93b82a0133be1d6ed9e0a51","Type":"ContainerStarted","Data":"cc809f08fd1b1ec8bba0f6df66aee099d71617ec4e73c31c6b501d2db29a95c7"} Oct 31 12:32:29.501241 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.501213 5693 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 12:32:29.501285 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.501255 5693 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 12:32:29.516021 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.516004 5693 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 12:32:29.516107 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:32:29.516065 5693 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 12:32:29.516543 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.516533 5693 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 12:32:29.516579 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:32:29.516554 5693 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 12:32:29.583736 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.583681 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8a68def89a8e2f4b1d88fecf85bfdb5e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal\" (UID: \"8a68def89a8e2f4b1d88fecf85bfdb5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 12:32:29.583736 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.583734 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a68def89a8e2f4b1d88fecf85bfdb5e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal\" (UID: \"8a68def89a8e2f4b1d88fecf85bfdb5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 12:32:29.583736 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.583754 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/72de75e4a93b82a0133be1d6ed9e0a51-config\") pod \"kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal\" (UID: \"72de75e4a93b82a0133be1d6ed9e0a51\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 12:32:29.598788 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.598762 5693 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 12:32:29.600674 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.600629 5693 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 12:32:29.600758 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.600687 5693 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 12:32:29.600758 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.600697 5693 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 12:32:29.600758 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.600737 5693 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-157.us-east-2.compute.internal" Oct 31 12:32:29.613037 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.613017 5693 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-131-157.us-east-2.compute.internal" Oct 31 12:32:29.613087 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.613078 5693 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-157.us-east-2.compute.internal" Oct 31 12:32:29.684854 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.684816 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8a68def89a8e2f4b1d88fecf85bfdb5e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal\" (UID: \"8a68def89a8e2f4b1d88fecf85bfdb5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 12:32:29.684996 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.684873 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a68def89a8e2f4b1d88fecf85bfdb5e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal\" (UID: \"8a68def89a8e2f4b1d88fecf85bfdb5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 12:32:29.684996 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.684890 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8a68def89a8e2f4b1d88fecf85bfdb5e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal\" (UID: \"8a68def89a8e2f4b1d88fecf85bfdb5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 12:32:29.684996 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.684897 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/72de75e4a93b82a0133be1d6ed9e0a51-config\") pod \"kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal\" (UID: \"72de75e4a93b82a0133be1d6ed9e0a51\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 12:32:29.684996 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.684939 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/72de75e4a93b82a0133be1d6ed9e0a51-config\") pod \"kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal\" (UID: \"72de75e4a93b82a0133be1d6ed9e0a51\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 12:32:29.684996 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:29.684968 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a68def89a8e2f4b1d88fecf85bfdb5e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal\" (UID: \"8a68def89a8e2f4b1d88fecf85bfdb5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 12:32:30.357393 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.357356 5693 apiserver.go:52] "Watching apiserver" Oct 31 12:32:30.365935 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.365905 5693 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 12:32:30.366475 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.366452 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-zbjvn","kube-system/global-pull-secret-syncer-wgrhs","openshift-monitoring/metrics-server-847b6d5877-sxkd8","openshift-cluster-node-tuning-operator/tuned-4wqp6","openshift-multus/multus-additional-cni-plugins-j7fd8","openshift-monitoring/prometheus-operator-9cd89b74c-227cc","openshift-network-diagnostics/network-check-target-bgcqc","openshift-ingress-canary/ingress-canary-pwtgm","openshift-multus/multus-dklfq","openshift-image-registry/node-ca-tk5vl","openshift-multus/network-metrics-daemon-8nbvz","openshift-network-operator/iptables-alerter-d7j6l","kube-system/konnectivity-agent-crhzp","kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal","openshift-dns/dns-default-bdwh8","openshift-dns/node-resolver-5gdlb","openshift-console/console-5b4df7846b-2pjqf","openshift-monitoring/node-exporter-gqmhl","openshift-ovn-kubernetes/ovnkube-node-hghw2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal","openshift-insights/insights-runtime-extractor-9t68r","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl","openshift-console/console-57b586f5c7-hbnwz"] Oct 31 12:32:30.372720 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.372694 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 12:32:30.372994 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.372814 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 12:32:30.373152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.373129 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 12:32:30.373441 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.373418 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 12:32:30.373563 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.373541 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7phhl\"" Oct 31 12:32:30.373620 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.373571 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zfd9q\"" Oct 31 12:32:30.373814 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.373798 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 12:32:30.373936 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.373918 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 12:32:30.374277 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.374256 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-w8vnc\"" Oct 31 12:32:30.374385 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.374366 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zzvpb\"" Oct 31 12:32:30.374453 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.374439 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 12:32:30.374502 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.374264 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 12:32:30.374692 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.374674 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 12:32:30.374860 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.374840 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 12:32:30.374954 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.374933 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 12:32:30.375017 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.374990 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 12:32:30.375128 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.375113 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 12:32:30.375177 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.375137 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 12:32:30.375177 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.375163 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 12:32:30.375272 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.375231 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fmbl2\"" Oct 31 12:32:30.375324 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.375274 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 12:32:30.375412 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.375392 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 12:32:30.375536 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.375520 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 12:32:30.375668 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.375653 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 12:32:30.375780 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.375665 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 12:32:30.375947 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.375797 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 12:32:30.375947 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.374949 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bkb2g\"" Oct 31 12:32:30.375947 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.375520 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 12:32:30.375947 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.374754 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 12:32:30.375947 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.375679 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hk7cx\"" Oct 31 12:32:30.375947 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.375801 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-l5d6v\"" Oct 31 12:32:30.376598 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.376555 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 12:32:30.376731 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.376600 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 12:32:30.376731 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.376618 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8c8p7\"" Oct 31 12:32:30.376831 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.376792 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 12:32:30.376898 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.376878 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 12:32:30.376979 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.376911 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-mrn76\"" Oct 31 12:32:30.376979 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.376962 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 12:32:30.377112 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.377042 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 12:32:30.377405 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.377385 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6k457\"" Oct 31 12:32:30.377475 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.377394 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 12:32:30.377519 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.377499 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 12:32:30.377519 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.377514 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 12:32:30.377600 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.377533 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 12:32:30.377600 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.377579 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 12:32:30.377717 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.377616 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 12:32:30.377717 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.377633 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l77g8\"" Oct 31 12:32:30.377717 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.377673 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 12:32:30.377847 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.377755 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nthx6\"" Oct 31 12:32:30.377976 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.377960 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 12:32:30.378025 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.377982 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 12:32:30.378025 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.377988 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zbx6v\"" Oct 31 12:32:30.378292 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.378275 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-52dg7\"" Oct 31 12:32:30.378970 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.378956 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 12:32:30.379003 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.378990 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 12:32:30.385559 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.385541 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 31 12:32:30.385650 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.385567 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 31 12:32:30.385737 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.385725 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 12:32:30.386050 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.386033 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 12:32:30.386097 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.386037 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 12:32:30.388000 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.387969 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 31 12:32:30.388064 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.388003 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-6k99scgtm5t5r\"" Oct 31 12:32:30.388064 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.388018 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 31 12:32:30.388686 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.388670 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-systemd-units\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.388719 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.388695 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-cnibin\") pod \"multus-additional-cni-plugins-j7fd8\" (UID: \"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35\") " pod="openshift-multus/multus-additional-cni-plugins-j7fd8" Oct 31 12:32:30.388747 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.388714 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64hd7\" (UniqueName: \"kubernetes.io/projected/f4a19150-beb0-4289-91ee-b1668968e6ca-kube-api-access-64hd7\") pod \"dns-default-bdwh8\" (UID: \"f4a19150-beb0-4289-91ee-b1668968e6ca\") " pod="openshift-dns/dns-default-bdwh8" Oct 31 12:32:30.388747 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.388740 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgs94\" (UniqueName: \"kubernetes.io/projected/bd53d085-6474-45d6-bd59-825fea180a79-kube-api-access-zgs94\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.388816 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.388783 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/24e17633-30a9-4410-b000-2fbb97202951-etc-selinux\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:32:30.388872 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.388812 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 12:32:30.388872 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.388829 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 31 12:32:30.388872 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.388846 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-slash\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.388872 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.388863 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-ghpjn\"" Oct 31 12:32:30.388872 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.388870 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-node-log\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.389092 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.388872 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-m7qs2\"" Oct 31 12:32:30.389092 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.388921 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 31 12:32:30.389092 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.388939 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-cni-binary-copy\") pod \"multus-additional-cni-plugins-j7fd8\" (UID: \"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35\") " pod="openshift-multus/multus-additional-cni-plugins-j7fd8" Oct 31 12:32:30.389092 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.388923 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 31 12:32:30.389092 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.388964 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf2e0b3-d31b-44ca-934c-3710f4262779-metrics-certs\") pod \"network-metrics-daemon-8nbvz\" (UID: \"bbf2e0b3-d31b-44ca-934c-3710f4262779\") " pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:32:30.389092 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.388972 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 12:32:30.389092 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.389013 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 31 12:32:30.389092 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.389018 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 31 12:32:30.389092 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.389044 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 31 12:32:30.389092 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.389053 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 31 12:32:30.389092 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.389012 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvd64\" (UniqueName: \"kubernetes.io/projected/b4c0e6ed-9c41-4023-9e84-524635b9f1fb-kube-api-access-tvd64\") pod \"iptables-alerter-d7j6l\" (UID: \"b4c0e6ed-9c41-4023-9e84-524635b9f1fb\") " pod="openshift-network-operator/iptables-alerter-d7j6l" Oct 31 12:32:30.389092 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.389072 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 31 12:32:30.389092 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.389016 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 31 12:32:30.389508 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.389094 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-vttbs\"" Oct 31 12:32:30.389508 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.389099 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-5jsz7\"" Oct 31 12:32:30.389508 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.389107 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zhwtd\"" Oct 31 12:32:30.389508 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.389110 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 12:32:30.389508 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.389159 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-system-cni-dir\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.389508 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.389205 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 31 12:32:30.389508 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.389228 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-cni-binary-copy\") pod \"multus-additional-cni-plugins-j7fd8\" (UID: \"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35\") " pod="openshift-multus/multus-additional-cni-plugins-j7fd8" Oct 31 12:32:30.389508 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.389258 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf2e0b3-d31b-44ca-934c-3710f4262779-metrics-certs\") pod \"network-metrics-daemon-8nbvz\" (UID: \"bbf2e0b3-d31b-44ca-934c-3710f4262779\") " pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:32:30.389508 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.389271 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 12:32:30.389508 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.389285 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-sysctl-conf\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:32:30.389508 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.389323 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-tuned\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:32:30.389508 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.389354 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/43421968-0e5b-4e27-be79-104aae5f72e0-konnectivity-ca\") pod \"konnectivity-agent-crhzp\" (UID: \"43421968-0e5b-4e27-be79-104aae5f72e0\") " pod="kube-system/konnectivity-agent-crhzp" Oct 31 12:32:30.389508 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.389386 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/24e17633-30a9-4410-b000-2fbb97202951-sys-fs\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:32:30.389508 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.389420 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-tuned\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:32:30.389508 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.389453 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-cni-netd\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.389508 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.389474 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j7fd8\" (UID: \"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35\") " pod="openshift-multus/multus-additional-cni-plugins-j7fd8" Oct 31 12:32:30.389508 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.389509 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-multus-conf-dir\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.389988 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.389539 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/24e17633-30a9-4410-b000-2fbb97202951-registration-dir\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:32:30.389988 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.389587 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-host-run-multus-certs\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.389988 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.389595 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/43421968-0e5b-4e27-be79-104aae5f72e0-konnectivity-ca\") pod \"konnectivity-agent-crhzp\" (UID: \"43421968-0e5b-4e27-be79-104aae5f72e0\") " pod="kube-system/konnectivity-agent-crhzp" Oct 31 12:32:30.389988 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.389695 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n79b\" (UniqueName: \"kubernetes.io/projected/0b091d91-47bc-4480-85b2-e23a8139df66-kube-api-access-4n79b\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.389988 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.389718 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a-original-pull-secret\") pod \"global-pull-secret-syncer-wgrhs\" (UID: \"ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a\") " pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:32:30.389988 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.389734 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-host-run-netns\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.389988 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.389742 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j7fd8\" (UID: \"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35\") " pod="openshift-multus/multus-additional-cni-plugins-j7fd8" Oct 31 12:32:30.389988 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.389749 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-sys\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:32:30.389988 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.389778 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-run-openvswitch\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.389988 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.389862 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4a19150-beb0-4289-91ee-b1668968e6ca-tmp-dir\") pod \"dns-default-bdwh8\" (UID: \"f4a19150-beb0-4289-91ee-b1668968e6ca\") " pod="openshift-dns/dns-default-bdwh8" Oct 31 12:32:30.389988 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.389907 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4a19150-beb0-4289-91ee-b1668968e6ca-tmp-dir\") pod \"dns-default-bdwh8\" (UID: \"f4a19150-beb0-4289-91ee-b1668968e6ca\") " pod="openshift-dns/dns-default-bdwh8" Oct 31 12:32:30.389988 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.389940 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-host\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:32:30.389988 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.389983 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.390545 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.390023 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b091d91-47bc-4480-85b2-e23a8139df66-env-overrides\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.390545 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.390052 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j7fd8\" (UID: \"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35\") " pod="openshift-multus/multus-additional-cni-plugins-j7fd8" Oct 31 12:32:30.390545 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.390053 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a-original-pull-secret\") pod \"global-pull-secret-syncer-wgrhs\" (UID: \"ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a\") " pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:32:30.390545 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.390149 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhnsq\" (UniqueName: \"kubernetes.io/projected/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-kube-api-access-rhnsq\") pod \"multus-additional-cni-plugins-j7fd8\" (UID: \"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35\") " pod="openshift-multus/multus-additional-cni-plugins-j7fd8" Oct 31 12:32:30.390545 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.390207 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b091d91-47bc-4480-85b2-e23a8139df66-env-overrides\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.390545 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.390223 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-multus-socket-dir-parent\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.390545 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.390240 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-log-socket\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.390545 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.390274 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-cni-bin\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.390545 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.390311 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b091d91-47bc-4480-85b2-e23a8139df66-ovnkube-script-lib\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.390545 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.390340 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14dc1e6d-70e1-4ba5-bffb-d9299433756d-host\") pod \"node-ca-tk5vl\" (UID: \"14dc1e6d-70e1-4ba5-bffb-d9299433756d\") " pod="openshift-image-registry/node-ca-tk5vl" Oct 31 12:32:30.390545 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.390348 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j7fd8\" (UID: \"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35\") " pod="openshift-multus/multus-additional-cni-plugins-j7fd8" Oct 31 12:32:30.390545 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.390381 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-lib-modules\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:32:30.390545 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.390405 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-var-lib-kubelet\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:32:30.390545 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.390438 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-run-netns\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.390545 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.390468 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b4c0e6ed-9c41-4023-9e84-524635b9f1fb-iptables-alerter-script\") pod \"iptables-alerter-d7j6l\" (UID: \"b4c0e6ed-9c41-4023-9e84-524635b9f1fb\") " pod="openshift-network-operator/iptables-alerter-d7j6l" Oct 31 12:32:30.390545 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.390504 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/14dc1e6d-70e1-4ba5-bffb-d9299433756d-serviceca\") pod \"node-ca-tk5vl\" (UID: \"14dc1e6d-70e1-4ba5-bffb-d9299433756d\") " pod="openshift-image-registry/node-ca-tk5vl" Oct 31 12:32:30.390545 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.390532 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b091d91-47bc-4480-85b2-e23a8139df66-ovnkube-script-lib\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.390545 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.390537 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a-dbus\") pod \"global-pull-secret-syncer-wgrhs\" (UID: \"ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a\") " pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:32:30.391099 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.390611 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdlgx\" (UniqueName: \"kubernetes.io/projected/45870e34-5e37-443a-826b-81c9c031051c-kube-api-access-kdlgx\") pod \"ingress-canary-pwtgm\" (UID: \"45870e34-5e37-443a-826b-81c9c031051c\") " pod="openshift-ingress-canary/ingress-canary-pwtgm" Oct 31 12:32:30.391099 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.390633 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bd53d085-6474-45d6-bd59-825fea180a79-multus-daemon-config\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.391099 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.390681 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-sysconfig\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:32:30.391099 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.390714 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b4c0e6ed-9c41-4023-9e84-524635b9f1fb-iptables-alerter-script\") pod \"iptables-alerter-d7j6l\" (UID: \"b4c0e6ed-9c41-4023-9e84-524635b9f1fb\") " pod="openshift-network-operator/iptables-alerter-d7j6l" Oct 31 12:32:30.391099 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.390744 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/14dc1e6d-70e1-4ba5-bffb-d9299433756d-serviceca\") pod \"node-ca-tk5vl\" (UID: \"14dc1e6d-70e1-4ba5-bffb-d9299433756d\") " pod="openshift-image-registry/node-ca-tk5vl" Oct 31 12:32:30.391099 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.390756 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/24e17633-30a9-4410-b000-2fbb97202951-socket-dir\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:32:30.391099 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.390783 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j7fd8\" (UID: \"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35\") " pod="openshift-multus/multus-additional-cni-plugins-j7fd8" Oct 31 12:32:30.391099 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.390803 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26913ad4-6613-4f3f-9819-9be238088c0a-tmp-dir\") pod \"node-resolver-5gdlb\" (UID: \"26913ad4-6613-4f3f-9819-9be238088c0a\") " pod="openshift-dns/node-resolver-5gdlb" Oct 31 12:32:30.391099 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.390821 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-etc-kubernetes\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.391099 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.390823 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bd53d085-6474-45d6-bd59-825fea180a79-multus-daemon-config\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.391099 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.390842 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-sysctl-d\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:32:30.391099 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.390865 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-run\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:32:30.391099 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.390923 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26913ad4-6613-4f3f-9819-9be238088c0a-tmp-dir\") pod \"node-resolver-5gdlb\" (UID: \"26913ad4-6613-4f3f-9819-9be238088c0a\") " pod="openshift-dns/node-resolver-5gdlb" Oct 31 12:32:30.391099 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.390924 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-var-lib-openvswitch\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.391099 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.390961 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-system-cni-dir\") pod \"multus-additional-cni-plugins-j7fd8\" (UID: \"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35\") " pod="openshift-multus/multus-additional-cni-plugins-j7fd8" Oct 31 12:32:30.391099 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.390984 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f4a19150-beb0-4289-91ee-b1668968e6ca-config-volume\") pod \"dns-default-bdwh8\" (UID: \"f4a19150-beb0-4289-91ee-b1668968e6ca\") " pod="openshift-dns/dns-default-bdwh8" Oct 31 12:32:30.391099 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.391034 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-multus-cni-dir\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.391099 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.391094 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-host-run-k8s-cni-cncf-io\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.391556 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.391135 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-host-var-lib-cni-multus\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.391556 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.391169 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-host-var-lib-kubelet\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.391556 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.391187 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f4a19150-beb0-4289-91ee-b1668968e6ca-config-volume\") pod \"dns-default-bdwh8\" (UID: \"f4a19150-beb0-4289-91ee-b1668968e6ca\") " pod="openshift-dns/dns-default-bdwh8" Oct 31 12:32:30.391556 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.391213 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph9jm\" (UniqueName: \"kubernetes.io/projected/02c6d8f5-97ff-41e0-8e38-71e1b0c87f54-kube-api-access-ph9jm\") pod \"network-check-target-bgcqc\" (UID: \"02c6d8f5-97ff-41e0-8e38-71e1b0c87f54\") " pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:32:30.391556 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.391355 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-cnibin\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.391556 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.391372 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-host-var-lib-cni-bin\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.391556 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.391429 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-systemd\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:32:30.391556 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.391451 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/43421968-0e5b-4e27-be79-104aae5f72e0-agent-certs\") pod \"konnectivity-agent-crhzp\" (UID: \"43421968-0e5b-4e27-be79-104aae5f72e0\") " pod="kube-system/konnectivity-agent-crhzp" Oct 31 12:32:30.391556 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.391481 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-run-systemd\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.391556 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.391496 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-etc-openvswitch\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.391556 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.391515 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b091d91-47bc-4480-85b2-e23a8139df66-ovnkube-config\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.392008 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.391564 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b4c0e6ed-9c41-4023-9e84-524635b9f1fb-host-slash\") pod \"iptables-alerter-d7j6l\" (UID: \"b4c0e6ed-9c41-4023-9e84-524635b9f1fb\") " pod="openshift-network-operator/iptables-alerter-d7j6l" Oct 31 12:32:30.392008 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.391584 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd53d085-6474-45d6-bd59-825fea180a79-cni-binary-copy\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.392008 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.391600 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-modprobe-d\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:32:30.392008 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.391618 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5hf6\" (UniqueName: \"kubernetes.io/projected/5d3565a5-d07a-4dc0-abd1-e697573baf22-kube-api-access-h5hf6\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:32:30.392008 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.391659 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/24e17633-30a9-4410-b000-2fbb97202951-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:32:30.392008 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.391675 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/24e17633-30a9-4410-b000-2fbb97202951-device-dir\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:32:30.392008 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.391691 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-run-ovn-kubernetes\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.392008 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.391710 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b091d91-47bc-4480-85b2-e23a8139df66-ovnkube-config\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.392008 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.391716 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b091d91-47bc-4480-85b2-e23a8139df66-ovn-node-metrics-cert\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.392008 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.391781 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2z5c\" (UniqueName: \"kubernetes.io/projected/bbf2e0b3-d31b-44ca-934c-3710f4262779-kube-api-access-j2z5c\") pod \"network-metrics-daemon-8nbvz\" (UID: \"bbf2e0b3-d31b-44ca-934c-3710f4262779\") " pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:32:30.392008 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.391797 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd53d085-6474-45d6-bd59-825fea180a79-cni-binary-copy\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.392008 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.391833 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/43421968-0e5b-4e27-be79-104aae5f72e0-agent-certs\") pod \"konnectivity-agent-crhzp\" (UID: \"43421968-0e5b-4e27-be79-104aae5f72e0\") " pod="kube-system/konnectivity-agent-crhzp" Oct 31 12:32:30.392008 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.391847 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/26913ad4-6613-4f3f-9819-9be238088c0a-hosts-file\") pod \"node-resolver-5gdlb\" (UID: \"26913ad4-6613-4f3f-9819-9be238088c0a\") " pod="openshift-dns/node-resolver-5gdlb" Oct 31 12:32:30.392008 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.391906 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-779z6\" (UniqueName: \"kubernetes.io/projected/26913ad4-6613-4f3f-9819-9be238088c0a-kube-api-access-779z6\") pod \"node-resolver-5gdlb\" (UID: \"26913ad4-6613-4f3f-9819-9be238088c0a\") " pod="openshift-dns/node-resolver-5gdlb" Oct 31 12:32:30.392008 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.391943 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd2xl\" (UniqueName: \"kubernetes.io/projected/14dc1e6d-70e1-4ba5-bffb-d9299433756d-kube-api-access-sd2xl\") pod \"node-ca-tk5vl\" (UID: \"14dc1e6d-70e1-4ba5-bffb-d9299433756d\") " pod="openshift-image-registry/node-ca-tk5vl" Oct 31 12:32:30.392008 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.391962 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b091d91-47bc-4480-85b2-e23a8139df66-ovn-node-metrics-cert\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.392823 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.392040 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a-kubelet-config\") pod \"global-pull-secret-syncer-wgrhs\" (UID: \"ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a\") " pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:32:30.392823 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.392125 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/45870e34-5e37-443a-826b-81c9c031051c-cert\") pod \"ingress-canary-pwtgm\" (UID: \"45870e34-5e37-443a-826b-81c9c031051c\") " pod="openshift-ingress-canary/ingress-canary-pwtgm" Oct 31 12:32:30.392823 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.392144 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4a19150-beb0-4289-91ee-b1668968e6ca-metrics-tls\") pod \"dns-default-bdwh8\" (UID: \"f4a19150-beb0-4289-91ee-b1668968e6ca\") " pod="openshift-dns/dns-default-bdwh8" Oct 31 12:32:30.392823 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.392171 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-kubernetes\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:32:30.392823 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.392213 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-run-ovn\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.392823 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.392228 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-hostroot\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.392823 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.392265 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d3565a5-d07a-4dc0-abd1-e697573baf22-tmp\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:32:30.392823 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.392322 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhppb\" (UniqueName: \"kubernetes.io/projected/24e17633-30a9-4410-b000-2fbb97202951-kube-api-access-zhppb\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:32:30.392823 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.392369 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d3565a5-d07a-4dc0-abd1-e697573baf22-tmp\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:32:30.392823 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.392384 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4a19150-beb0-4289-91ee-b1668968e6ca-metrics-tls\") pod \"dns-default-bdwh8\" (UID: \"f4a19150-beb0-4289-91ee-b1668968e6ca\") " pod="openshift-dns/dns-default-bdwh8" Oct 31 12:32:30.392823 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.392402 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/45870e34-5e37-443a-826b-81c9c031051c-cert\") pod \"ingress-canary-pwtgm\" (UID: \"45870e34-5e37-443a-826b-81c9c031051c\") " pod="openshift-ingress-canary/ingress-canary-pwtgm" Oct 31 12:32:30.392823 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.392399 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-kubelet\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.399581 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.399558 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64hd7\" (UniqueName: \"kubernetes.io/projected/f4a19150-beb0-4289-91ee-b1668968e6ca-kube-api-access-64hd7\") pod \"dns-default-bdwh8\" (UID: \"f4a19150-beb0-4289-91ee-b1668968e6ca\") " pod="openshift-dns/dns-default-bdwh8" Oct 31 12:32:30.399670 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.399633 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvd64\" (UniqueName: \"kubernetes.io/projected/b4c0e6ed-9c41-4023-9e84-524635b9f1fb-kube-api-access-tvd64\") pod \"iptables-alerter-d7j6l\" (UID: \"b4c0e6ed-9c41-4023-9e84-524635b9f1fb\") " pod="openshift-network-operator/iptables-alerter-d7j6l" Oct 31 12:32:30.403974 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.403957 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgs94\" (UniqueName: \"kubernetes.io/projected/bd53d085-6474-45d6-bd59-825fea180a79-kube-api-access-zgs94\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.404988 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.404973 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n79b\" (UniqueName: \"kubernetes.io/projected/0b091d91-47bc-4480-85b2-e23a8139df66-kube-api-access-4n79b\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.407150 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.407116 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhnsq\" (UniqueName: \"kubernetes.io/projected/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-kube-api-access-rhnsq\") pod \"multus-additional-cni-plugins-j7fd8\" (UID: \"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35\") " pod="openshift-multus/multus-additional-cni-plugins-j7fd8" Oct 31 12:32:30.407478 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.407460 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdlgx\" (UniqueName: \"kubernetes.io/projected/45870e34-5e37-443a-826b-81c9c031051c-kube-api-access-kdlgx\") pod \"ingress-canary-pwtgm\" (UID: \"45870e34-5e37-443a-826b-81c9c031051c\") " pod="openshift-ingress-canary/ingress-canary-pwtgm" Oct 31 12:32:30.407508 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.407460 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2z5c\" (UniqueName: \"kubernetes.io/projected/bbf2e0b3-d31b-44ca-934c-3710f4262779-kube-api-access-j2z5c\") pod \"network-metrics-daemon-8nbvz\" (UID: \"bbf2e0b3-d31b-44ca-934c-3710f4262779\") " pod="openshift-multus/network-metrics-daemon-8nbvz" Oct 31 12:32:30.407606 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.407589 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhppb\" (UniqueName: \"kubernetes.io/projected/24e17633-30a9-4410-b000-2fbb97202951-kube-api-access-zhppb\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:32:30.409980 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.409965 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph9jm\" (UniqueName: \"kubernetes.io/projected/02c6d8f5-97ff-41e0-8e38-71e1b0c87f54-kube-api-access-ph9jm\") pod \"network-check-target-bgcqc\" (UID: \"02c6d8f5-97ff-41e0-8e38-71e1b0c87f54\") " pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:32:30.410433 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.410419 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-779z6\" (UniqueName: \"kubernetes.io/projected/26913ad4-6613-4f3f-9819-9be238088c0a-kube-api-access-779z6\") pod \"node-resolver-5gdlb\" (UID: \"26913ad4-6613-4f3f-9819-9be238088c0a\") " pod="openshift-dns/node-resolver-5gdlb" Oct 31 12:32:30.411023 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.411008 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5hf6\" (UniqueName: \"kubernetes.io/projected/5d3565a5-d07a-4dc0-abd1-e697573baf22-kube-api-access-h5hf6\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:32:30.412329 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.412313 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd2xl\" (UniqueName: \"kubernetes.io/projected/14dc1e6d-70e1-4ba5-bffb-d9299433756d-kube-api-access-sd2xl\") pod \"node-ca-tk5vl\" (UID: \"14dc1e6d-70e1-4ba5-bffb-d9299433756d\") " pod="openshift-image-registry/node-ca-tk5vl" Oct 31 12:32:30.471903 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.471875 5693 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 12:32:30.492965 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.492932 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-host-run-netns\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.492965 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.492964 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-sys\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:32:30.493169 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.492982 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-run-openvswitch\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.493169 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.492998 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4b6af1c5-6045-4e4c-aa87-2473d19bd000-root\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:30.493169 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493057 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-host\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:32:30.493169 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493093 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ddebd541-4d2f-4861-9dd0-0ee66b03aaf2-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-227cc\" (UID: \"ddebd541-4d2f-4861-9dd0-0ee66b03aaf2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-227cc" Oct 31 12:32:30.493169 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493094 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-run-openvswitch\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.493169 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493066 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-sys\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:32:30.493169 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493145 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-host\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:32:30.493169 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493120 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-host-run-netns\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.493169 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493120 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-multus-socket-dir-parent\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.493444 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493179 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-multus-socket-dir-parent\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.493444 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493215 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/97cf3735-d794-4628-9048-252f949317b9-crio-socket\") pod \"insights-runtime-extractor-9t68r\" (UID: \"97cf3735-d794-4628-9048-252f949317b9\") " pod="openshift-insights/insights-runtime-extractor-9t68r" Oct 31 12:32:30.493444 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493249 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-var-lib-kubelet\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:32:30.493444 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493287 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-run-netns\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.493444 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493318 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-run-netns\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.493444 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493325 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b6af1c5-6045-4e4c-aa87-2473d19bd000-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:30.493444 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493348 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ddebd541-4d2f-4861-9dd0-0ee66b03aaf2-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-227cc\" (UID: \"ddebd541-4d2f-4861-9dd0-0ee66b03aaf2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-227cc" Oct 31 12:32:30.493444 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493355 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-var-lib-kubelet\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:32:30.493444 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493417 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a-dbus\") pod \"global-pull-secret-syncer-wgrhs\" (UID: \"ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a\") " pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:32:30.493718 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493448 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/24e17633-30a9-4410-b000-2fbb97202951-socket-dir\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:32:30.493718 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493470 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a-dbus\") pod \"global-pull-secret-syncer-wgrhs\" (UID: \"ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a\") " pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:32:30.493718 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493474 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/97cf3735-d794-4628-9048-252f949317b9-data-volume\") pod \"insights-runtime-extractor-9t68r\" (UID: \"97cf3735-d794-4628-9048-252f949317b9\") " pod="openshift-insights/insights-runtime-extractor-9t68r" Oct 31 12:32:30.493718 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493534 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/24e17633-30a9-4410-b000-2fbb97202951-socket-dir\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:32:30.493718 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493543 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/97cf3735-d794-4628-9048-252f949317b9-data-volume\") pod \"insights-runtime-extractor-9t68r\" (UID: \"97cf3735-d794-4628-9048-252f949317b9\") " pod="openshift-insights/insights-runtime-extractor-9t68r" Oct 31 12:32:30.493718 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493555 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/25a02fd6-8ab0-4cca-94cd-d4082f874170-metrics-server-audit-profiles\") pod \"metrics-server-847b6d5877-sxkd8\" (UID: \"25a02fd6-8ab0-4cca-94cd-d4082f874170\") " pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:30.493718 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493546 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b6af1c5-6045-4e4c-aa87-2473d19bd000-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:30.493718 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493575 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-var-lib-openvswitch\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.493718 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493590 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-system-cni-dir\") pod \"multus-additional-cni-plugins-j7fd8\" (UID: \"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35\") " pod="openshift-multus/multus-additional-cni-plugins-j7fd8" Oct 31 12:32:30.493718 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493608 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpx9x\" (UniqueName: \"kubernetes.io/projected/0f343e95-e397-49aa-b023-98337d3be3ff-kube-api-access-kpx9x\") pod \"console-5b4df7846b-2pjqf\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:30.493718 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493660 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-multus-cni-dir\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.493718 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493683 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-host-var-lib-cni-multus\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.493718 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493686 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-system-cni-dir\") pod \"multus-additional-cni-plugins-j7fd8\" (UID: \"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35\") " pod="openshift-multus/multus-additional-cni-plugins-j7fd8" Oct 31 12:32:30.493718 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493717 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-host-var-lib-kubelet\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.494314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493740 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/25a02fd6-8ab0-4cca-94cd-d4082f874170-client-ca-bundle\") pod \"metrics-server-847b6d5877-sxkd8\" (UID: \"25a02fd6-8ab0-4cca-94cd-d4082f874170\") " pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:30.494314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493761 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0f343e95-e397-49aa-b023-98337d3be3ff-oauth-serving-cert\") pod \"console-5b4df7846b-2pjqf\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:30.494314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493786 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xpnl\" (UniqueName: \"kubernetes.io/projected/97cf3735-d794-4628-9048-252f949317b9-kube-api-access-2xpnl\") pod \"insights-runtime-extractor-9t68r\" (UID: \"97cf3735-d794-4628-9048-252f949317b9\") " pod="openshift-insights/insights-runtime-extractor-9t68r" Oct 31 12:32:30.494314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493803 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-host-var-lib-cni-multus\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.494314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493817 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0f343e95-e397-49aa-b023-98337d3be3ff-console-config\") pod \"console-5b4df7846b-2pjqf\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:30.494314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493840 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/24e17633-30a9-4410-b000-2fbb97202951-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:32:30.494314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493865 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/24e17633-30a9-4410-b000-2fbb97202951-device-dir\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:32:30.494314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493865 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-multus-cni-dir\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.494314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493662 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-var-lib-openvswitch\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.494314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493908 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/24e17633-30a9-4410-b000-2fbb97202951-device-dir\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:32:30.494314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493898 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/25a02fd6-8ab0-4cca-94cd-d4082f874170-metrics-server-audit-profiles\") pod \"metrics-server-847b6d5877-sxkd8\" (UID: \"25a02fd6-8ab0-4cca-94cd-d4082f874170\") " pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:30.494314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493900 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f343e95-e397-49aa-b023-98337d3be3ff-console-serving-cert\") pod \"console-5b4df7846b-2pjqf\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:30.494314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493951 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-host-var-lib-kubelet\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.494314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493962 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a-kubelet-config\") pod \"global-pull-secret-syncer-wgrhs\" (UID: \"ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a\") " pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:32:30.494314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493979 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/24e17633-30a9-4410-b000-2fbb97202951-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:32:30.494314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.493988 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0f343e95-e397-49aa-b023-98337d3be3ff-oauth-serving-cert\") pod \"console-5b4df7846b-2pjqf\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:30.494314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494000 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-kubernetes\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:32:30.494314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494029 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ddebd541-4d2f-4861-9dd0-0ee66b03aaf2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-227cc\" (UID: \"ddebd541-4d2f-4861-9dd0-0ee66b03aaf2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-227cc" Oct 31 12:32:30.494314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494039 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a-kubelet-config\") pod \"global-pull-secret-syncer-wgrhs\" (UID: \"ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a\") " pod="kube-system/global-pull-secret-syncer-wgrhs" Oct 31 12:32:30.494314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494056 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-systemd-units\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.494314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494088 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-kubernetes\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:32:30.494314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494095 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f343e95-e397-49aa-b023-98337d3be3ff-console-serving-cert\") pod \"console-5b4df7846b-2pjqf\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:30.494314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494104 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/25a02fd6-8ab0-4cca-94cd-d4082f874170-client-ca-bundle\") pod \"metrics-server-847b6d5877-sxkd8\" (UID: \"25a02fd6-8ab0-4cca-94cd-d4082f874170\") " pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:30.494314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494097 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-systemd-units\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.494314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494179 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25a02fd6-8ab0-4cca-94cd-d4082f874170-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-847b6d5877-sxkd8\" (UID: \"25a02fd6-8ab0-4cca-94cd-d4082f874170\") " pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:30.494314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494209 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4b6af1c5-6045-4e4c-aa87-2473d19bd000-node-exporter-wtmp\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:30.494314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494234 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f343e95-e397-49aa-b023-98337d3be3ff-tmp\") pod \"console-5b4df7846b-2pjqf\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:30.494314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494258 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/24e17633-30a9-4410-b000-2fbb97202951-etc-selinux\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:32:30.494314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494281 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-node-log\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.494314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494293 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ddebd541-4d2f-4861-9dd0-0ee66b03aaf2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-227cc\" (UID: \"ddebd541-4d2f-4861-9dd0-0ee66b03aaf2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-227cc" Oct 31 12:32:30.494314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494306 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c91a35c-7933-4399-b7ff-4c1226d72d20-service-ca\") pod \"console-57b586f5c7-hbnwz\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:30.494314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494297 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f343e95-e397-49aa-b023-98337d3be3ff-tmp\") pod \"console-5b4df7846b-2pjqf\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:30.494314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494317 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0f343e95-e397-49aa-b023-98337d3be3ff-console-config\") pod \"console-5b4df7846b-2pjqf\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:30.494314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494329 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c91a35c-7933-4399-b7ff-4c1226d72d20-oauth-serving-cert\") pod \"console-57b586f5c7-hbnwz\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:30.494314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494336 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/24e17633-30a9-4410-b000-2fbb97202951-etc-selinux\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494347 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-node-log\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494354 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f343e95-e397-49aa-b023-98337d3be3ff-trusted-ca-bundle\") pod \"console-5b4df7846b-2pjqf\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494358 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25a02fd6-8ab0-4cca-94cd-d4082f874170-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-847b6d5877-sxkd8\" (UID: \"25a02fd6-8ab0-4cca-94cd-d4082f874170\") " pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494377 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-sysctl-conf\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494399 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/24e17633-30a9-4410-b000-2fbb97202951-sys-fs\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494430 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-cni-netd\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494452 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/24e17633-30a9-4410-b000-2fbb97202951-registration-dir\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494471 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c91a35c-7933-4399-b7ff-4c1226d72d20-oauth-serving-cert\") pod \"console-57b586f5c7-hbnwz\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494476 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/25a02fd6-8ab0-4cca-94cd-d4082f874170-audit-log\") pod \"metrics-server-847b6d5877-sxkd8\" (UID: \"25a02fd6-8ab0-4cca-94cd-d4082f874170\") " pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494498 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c91a35c-7933-4399-b7ff-4c1226d72d20-service-ca\") pod \"console-57b586f5c7-hbnwz\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494506 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/24e17633-30a9-4410-b000-2fbb97202951-sys-fs\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494538 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-sysctl-conf\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494557 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-cni-netd\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494589 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/25a02fd6-8ab0-4cca-94cd-d4082f874170-audit-log\") pod \"metrics-server-847b6d5877-sxkd8\" (UID: \"25a02fd6-8ab0-4cca-94cd-d4082f874170\") " pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494599 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/24e17633-30a9-4410-b000-2fbb97202951-registration-dir\") pod \"aws-ebs-csi-driver-node-zg9zl\" (UID: \"24e17633-30a9-4410-b000-2fbb97202951\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg9zl" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494628 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f343e95-e397-49aa-b023-98337d3be3ff-trusted-ca-bundle\") pod \"console-5b4df7846b-2pjqf\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494606 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4b6af1c5-6045-4e4c-aa87-2473d19bd000-node-exporter-accelerators-collector-config\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494695 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c91a35c-7933-4399-b7ff-4c1226d72d20-console-oauth-config\") pod \"console-57b586f5c7-hbnwz\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494710 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0f343e95-e397-49aa-b023-98337d3be3ff-service-ca\") pod \"console-5b4df7846b-2pjqf\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494729 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494753 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-log-socket\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494776 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-cni-bin\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494800 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d92p\" (UniqueName: \"kubernetes.io/projected/3015bfb0-80c8-4c01-9053-85e5ff2318fb-kube-api-access-5d92p\") pod \"monitoring-plugin-77f4f76594-zbjvn\" (UID: \"3015bfb0-80c8-4c01-9053-85e5ff2318fb\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zbjvn" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494828 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c91a35c-7933-4399-b7ff-4c1226d72d20-console-serving-cert\") pod \"console-57b586f5c7-hbnwz\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494852 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14dc1e6d-70e1-4ba5-bffb-d9299433756d-host\") pod \"node-ca-tk5vl\" (UID: \"14dc1e6d-70e1-4ba5-bffb-d9299433756d\") " pod="openshift-image-registry/node-ca-tk5vl" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494876 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-lib-modules\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494895 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0f343e95-e397-49aa-b023-98337d3be3ff-service-ca\") pod \"console-5b4df7846b-2pjqf\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494902 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8shn\" (UniqueName: \"kubernetes.io/projected/ddebd541-4d2f-4861-9dd0-0ee66b03aaf2-kube-api-access-m8shn\") pod \"prometheus-operator-9cd89b74c-227cc\" (UID: \"ddebd541-4d2f-4861-9dd0-0ee66b03aaf2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-227cc" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494937 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srn9m\" (UniqueName: \"kubernetes.io/projected/25a02fd6-8ab0-4cca-94cd-d4082f874170-kube-api-access-srn9m\") pod \"metrics-server-847b6d5877-sxkd8\" (UID: \"25a02fd6-8ab0-4cca-94cd-d4082f874170\") " pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494951 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494964 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-sysconfig\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494963 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-cni-bin\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.494989 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j7fd8\" (UID: \"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35\") " pod="openshift-multus/multus-additional-cni-plugins-j7fd8" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.495015 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rprnd\" (UniqueName: \"kubernetes.io/projected/4b6af1c5-6045-4e4c-aa87-2473d19bd000-kube-api-access-rprnd\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.495037 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3015bfb0-80c8-4c01-9053-85e5ff2318fb-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-zbjvn\" (UID: \"3015bfb0-80c8-4c01-9053-85e5ff2318fb\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zbjvn" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.495179 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c91a35c-7933-4399-b7ff-4c1226d72d20-console-oauth-config\") pod \"console-57b586f5c7-hbnwz\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.495195 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4b6af1c5-6045-4e4c-aa87-2473d19bd000-node-exporter-accelerators-collector-config\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.495238 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14dc1e6d-70e1-4ba5-bffb-d9299433756d-host\") pod \"node-ca-tk5vl\" (UID: \"14dc1e6d-70e1-4ba5-bffb-d9299433756d\") " pod="openshift-image-registry/node-ca-tk5vl" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.495608 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-etc-kubernetes\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.495353 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-sysconfig\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.495385 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-log-socket\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.495453 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c91a35c-7933-4399-b7ff-4c1226d72d20-console-serving-cert\") pod \"console-57b586f5c7-hbnwz\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.495462 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-lib-modules\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.495673 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3015bfb0-80c8-4c01-9053-85e5ff2318fb-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-zbjvn\" (UID: \"3015bfb0-80c8-4c01-9053-85e5ff2318fb\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zbjvn" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.495431 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j7fd8\" (UID: \"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35\") " pod="openshift-multus/multus-additional-cni-plugins-j7fd8" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.495566 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-etc-kubernetes\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.495738 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-sysctl-d\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.495757 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-run\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.495785 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97cf3735-d794-4628-9048-252f949317b9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9t68r\" (UID: \"97cf3735-d794-4628-9048-252f949317b9\") " pod="openshift-insights/insights-runtime-extractor-9t68r" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.495807 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-host-run-k8s-cni-cncf-io\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.495826 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/25a02fd6-8ab0-4cca-94cd-d4082f874170-secret-metrics-server-tls\") pod \"metrics-server-847b6d5877-sxkd8\" (UID: \"25a02fd6-8ab0-4cca-94cd-d4082f874170\") " pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.495845 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b6af1c5-6045-4e4c-aa87-2473d19bd000-metrics-client-ca\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.495866 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-cnibin\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.495874 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-sysctl-d\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.495885 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-host-var-lib-cni-bin\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.495906 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-systemd\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.495915 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-host-run-k8s-cni-cncf-io\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.495925 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-run-systemd\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.495948 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-etc-openvswitch\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.495957 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-run\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.495969 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b4c0e6ed-9c41-4023-9e84-524635b9f1fb-host-slash\") pod \"iptables-alerter-d7j6l\" (UID: \"b4c0e6ed-9c41-4023-9e84-524635b9f1fb\") " pod="openshift-network-operator/iptables-alerter-d7j6l" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.495990 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-modprobe-d\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.496011 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-run-ovn-kubernetes\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.496034 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4b6af1c5-6045-4e4c-aa87-2473d19bd000-node-exporter-textfile\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.496055 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0f343e95-e397-49aa-b023-98337d3be3ff-console-oauth-config\") pod \"console-5b4df7846b-2pjqf\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.496074 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/26913ad4-6613-4f3f-9819-9be238088c0a-hosts-file\") pod \"node-resolver-5gdlb\" (UID: \"26913ad4-6613-4f3f-9819-9be238088c0a\") " pod="openshift-dns/node-resolver-5gdlb" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.496091 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-run-ovn\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.496105 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-cnibin\") pod \"multus-additional-cni-plugins-j7fd8\" (UID: \"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35\") " pod="openshift-multus/multus-additional-cni-plugins-j7fd8" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.496121 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b6af1c5-6045-4e4c-aa87-2473d19bd000-sys\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.496136 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4b6af1c5-6045-4e4c-aa87-2473d19bd000-node-exporter-tls\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.496153 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls87f\" (UniqueName: \"kubernetes.io/projected/8c91a35c-7933-4399-b7ff-4c1226d72d20-kube-api-access-ls87f\") pod \"console-57b586f5c7-hbnwz\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:30.496152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.496161 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97cf3735-d794-4628-9048-252f949317b9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9t68r\" (UID: \"97cf3735-d794-4628-9048-252f949317b9\") " pod="openshift-insights/insights-runtime-extractor-9t68r" Oct 31 12:32:30.498526 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.496209 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-hostroot\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.498526 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.496168 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-hostroot\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.498526 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.496251 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-run-ovn-kubernetes\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.498526 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.496273 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-kubelet\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.498526 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.496297 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-cnibin\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.498526 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.496306 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8c91a35c-7933-4399-b7ff-4c1226d72d20-console-config\") pod \"console-57b586f5c7-hbnwz\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:30.498526 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.496317 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4b6af1c5-6045-4e4c-aa87-2473d19bd000-node-exporter-textfile\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:30.498526 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.496329 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8c75b6d0-3fc9-4e2e-aba2-f464a48cab35-cnibin\") pod \"multus-additional-cni-plugins-j7fd8\" (UID: \"8c75b6d0-3fc9-4e2e-aba2-f464a48cab35\") " pod="openshift-multus/multus-additional-cni-plugins-j7fd8" Oct 31 12:32:30.498526 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.496335 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b6af1c5-6045-4e4c-aa87-2473d19bd000-metrics-client-ca\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:30.498526 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.496358 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-kubelet\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.498526 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.496410 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/25a02fd6-8ab0-4cca-94cd-d4082f874170-secret-metrics-server-tls\") pod \"metrics-server-847b6d5877-sxkd8\" (UID: \"25a02fd6-8ab0-4cca-94cd-d4082f874170\") " pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:30.498526 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.496415 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-run-systemd\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.498526 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.496474 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-modprobe-d\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:32:30.498526 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.496584 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0f343e95-e397-49aa-b023-98337d3be3ff-console-oauth-config\") pod \"console-5b4df7846b-2pjqf\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:30.498526 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.496595 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-etc-openvswitch\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.498526 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.496622 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-host-var-lib-cni-bin\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.498526 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.496754 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-run-ovn\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.498526 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.496803 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-slash\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.498526 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.496832 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c91a35c-7933-4399-b7ff-4c1226d72d20-tmp\") pod \"console-57b586f5c7-hbnwz\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:30.498526 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.496857 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-system-cni-dir\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.498526 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.496884 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddebd541-4d2f-4861-9dd0-0ee66b03aaf2-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-227cc\" (UID: \"ddebd541-4d2f-4861-9dd0-0ee66b03aaf2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-227cc" Oct 31 12:32:30.498526 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.496909 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-multus-conf-dir\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.498526 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.496914 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b091d91-47bc-4480-85b2-e23a8139df66-host-slash\") pod \"ovnkube-node-hghw2\" (UID: \"0b091d91-47bc-4480-85b2-e23a8139df66\") " pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:30.498526 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.496937 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/97cf3735-d794-4628-9048-252f949317b9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9t68r\" (UID: \"97cf3735-d794-4628-9048-252f949317b9\") " pod="openshift-insights/insights-runtime-extractor-9t68r" Oct 31 12:32:30.498526 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.496964 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-host-run-multus-certs\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.498526 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.496990 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/25a02fd6-8ab0-4cca-94cd-d4082f874170-secret-metrics-server-client-certs\") pod \"metrics-server-847b6d5877-sxkd8\" (UID: \"25a02fd6-8ab0-4cca-94cd-d4082f874170\") " pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:30.498526 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.496993 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-system-cni-dir\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.498526 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.496959 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c91a35c-7933-4399-b7ff-4c1226d72d20-tmp\") pod \"console-57b586f5c7-hbnwz\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:30.498526 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.496994 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4b6af1c5-6045-4e4c-aa87-2473d19bd000-node-exporter-tls\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:30.498526 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.497032 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b4c0e6ed-9c41-4023-9e84-524635b9f1fb-host-slash\") pod \"iptables-alerter-d7j6l\" (UID: \"b4c0e6ed-9c41-4023-9e84-524635b9f1fb\") " pod="openshift-network-operator/iptables-alerter-d7j6l" Oct 31 12:32:30.498526 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.497039 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/26913ad4-6613-4f3f-9819-9be238088c0a-hosts-file\") pod \"node-resolver-5gdlb\" (UID: \"26913ad4-6613-4f3f-9819-9be238088c0a\") " pod="openshift-dns/node-resolver-5gdlb" Oct 31 12:32:30.498526 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.497087 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-multus-conf-dir\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.498526 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.497089 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5d3565a5-d07a-4dc0-abd1-e697573baf22-etc-systemd\") pod \"tuned-4wqp6\" (UID: \"5d3565a5-d07a-4dc0-abd1-e697573baf22\") " pod="openshift-cluster-node-tuning-operator/tuned-4wqp6" Oct 31 12:32:30.498526 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.497122 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bd53d085-6474-45d6-bd59-825fea180a79-host-run-multus-certs\") pod \"multus-dklfq\" (UID: \"bd53d085-6474-45d6-bd59-825fea180a79\") " pod="openshift-multus/multus-dklfq" Oct 31 12:32:30.498526 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.497211 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/97cf3735-d794-4628-9048-252f949317b9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9t68r\" (UID: \"97cf3735-d794-4628-9048-252f949317b9\") " pod="openshift-insights/insights-runtime-extractor-9t68r" Oct 31 12:32:30.498526 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.497223 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8c91a35c-7933-4399-b7ff-4c1226d72d20-console-config\") pod \"console-57b586f5c7-hbnwz\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:30.498526 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.497227 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddebd541-4d2f-4861-9dd0-0ee66b03aaf2-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-227cc\" (UID: \"ddebd541-4d2f-4861-9dd0-0ee66b03aaf2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-227cc" Oct 31 12:32:30.498526 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.497300 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/25a02fd6-8ab0-4cca-94cd-d4082f874170-secret-metrics-server-client-certs\") pod \"metrics-server-847b6d5877-sxkd8\" (UID: \"25a02fd6-8ab0-4cca-94cd-d4082f874170\") " pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:30.504738 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.504718 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpx9x\" (UniqueName: \"kubernetes.io/projected/0f343e95-e397-49aa-b023-98337d3be3ff-kube-api-access-kpx9x\") pod \"console-5b4df7846b-2pjqf\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:30.508992 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.508955 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rprnd\" (UniqueName: \"kubernetes.io/projected/4b6af1c5-6045-4e4c-aa87-2473d19bd000-kube-api-access-rprnd\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:30.509080 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.509021 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xpnl\" (UniqueName: \"kubernetes.io/projected/97cf3735-d794-4628-9048-252f949317b9-kube-api-access-2xpnl\") pod \"insights-runtime-extractor-9t68r\" (UID: \"97cf3735-d794-4628-9048-252f949317b9\") " pod="openshift-insights/insights-runtime-extractor-9t68r" Oct 31 12:32:30.509166 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.509148 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srn9m\" (UniqueName: \"kubernetes.io/projected/25a02fd6-8ab0-4cca-94cd-d4082f874170-kube-api-access-srn9m\") pod \"metrics-server-847b6d5877-sxkd8\" (UID: \"25a02fd6-8ab0-4cca-94cd-d4082f874170\") " pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:30.509387 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.509368 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8shn\" (UniqueName: \"kubernetes.io/projected/ddebd541-4d2f-4861-9dd0-0ee66b03aaf2-kube-api-access-m8shn\") pod \"prometheus-operator-9cd89b74c-227cc\" (UID: \"ddebd541-4d2f-4861-9dd0-0ee66b03aaf2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-227cc" Oct 31 12:32:30.509764 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.509749 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d92p\" (UniqueName: \"kubernetes.io/projected/3015bfb0-80c8-4c01-9053-85e5ff2318fb-kube-api-access-5d92p\") pod \"monitoring-plugin-77f4f76594-zbjvn\" (UID: \"3015bfb0-80c8-4c01-9053-85e5ff2318fb\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zbjvn" Oct 31 12:32:30.509799 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.509788 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls87f\" (UniqueName: \"kubernetes.io/projected/8c91a35c-7933-4399-b7ff-4c1226d72d20-kube-api-access-ls87f\") pod \"console-57b586f5c7-hbnwz\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:30.563479 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.563449 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57b586f5c7-hbnwz"] Oct 31 12:32:30.598171 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.598136 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4b6af1c5-6045-4e4c-aa87-2473d19bd000-node-exporter-wtmp\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:30.598319 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.598216 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b6af1c5-6045-4e4c-aa87-2473d19bd000-sys\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:30.598319 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.598254 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4b6af1c5-6045-4e4c-aa87-2473d19bd000-root\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:30.598319 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.598257 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4b6af1c5-6045-4e4c-aa87-2473d19bd000-node-exporter-wtmp\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:30.598319 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.598280 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/97cf3735-d794-4628-9048-252f949317b9-crio-socket\") pod \"insights-runtime-extractor-9t68r\" (UID: \"97cf3735-d794-4628-9048-252f949317b9\") " pod="openshift-insights/insights-runtime-extractor-9t68r" Oct 31 12:32:30.598319 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.598311 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b6af1c5-6045-4e4c-aa87-2473d19bd000-sys\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:30.598514 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.598337 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/97cf3735-d794-4628-9048-252f949317b9-crio-socket\") pod \"insights-runtime-extractor-9t68r\" (UID: \"97cf3735-d794-4628-9048-252f949317b9\") " pod="openshift-insights/insights-runtime-extractor-9t68r" Oct 31 12:32:30.598514 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:30.598349 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4b6af1c5-6045-4e4c-aa87-2473d19bd000-root\") pod \"node-exporter-gqmhl\" (UID: \"4b6af1c5-6045-4e4c-aa87-2473d19bd000\") " pod="openshift-monitoring/node-exporter-gqmhl" Oct 31 12:32:31.015166 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:31.015129 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:31.258114 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:31.258079 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:33.504520 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:33.504491 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bdwh8" Oct 31 12:32:34.618299 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:34.618262 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:35.609262 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:35.609231 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:36.739650 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:36.739606 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:37.488131 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:37.488101 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:37.488316 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:37.488303 5693 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 12:32:38.764007 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:38.763936 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hghw2" Oct 31 12:32:39.309020 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:39.308984 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-zbjvn" Oct 31 12:32:39.313480 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:39.313457 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-zbjvn" Oct 31 12:32:39.941467 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:39.941437 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:32:41.263323 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:41.263290 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:41.267063 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:41.267044 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:32:41.359148 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:41.359109 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:48.685080 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:32:48.685042 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:32:48.882181 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:48.882147 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6f7b7cfbff-7br98"] Oct 31 12:32:48.887320 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:48.887292 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6f7b7cfbff-7br98" Oct 31 12:32:48.899804 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:48.899775 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6f7b7cfbff-7br98"] Oct 31 12:32:48.913760 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:48.913728 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 31 12:32:48.916605 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:48.916583 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 31 12:32:48.976981 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:48.976901 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 31 12:32:48.976981 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:48.976901 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 31 12:32:48.976981 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:48.976906 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 31 12:32:48.976981 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:48.976975 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 31 12:32:48.977200 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:48.977014 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-h5t2j\"" Oct 31 12:32:49.021450 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:49.021418 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c49bcfc5-1bc7-46fc-85ca-5be1fa784116-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f7b7cfbff-7br98\" (UID: \"c49bcfc5-1bc7-46fc-85ca-5be1fa784116\") " pod="openshift-monitoring/telemeter-client-6f7b7cfbff-7br98" Oct 31 12:32:49.021613 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:49.021461 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c49bcfc5-1bc7-46fc-85ca-5be1fa784116-federate-client-tls\") pod \"telemeter-client-6f7b7cfbff-7br98\" (UID: \"c49bcfc5-1bc7-46fc-85ca-5be1fa784116\") " pod="openshift-monitoring/telemeter-client-6f7b7cfbff-7br98" Oct 31 12:32:49.021613 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:49.021491 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c49bcfc5-1bc7-46fc-85ca-5be1fa784116-metrics-client-ca\") pod \"telemeter-client-6f7b7cfbff-7br98\" (UID: \"c49bcfc5-1bc7-46fc-85ca-5be1fa784116\") " pod="openshift-monitoring/telemeter-client-6f7b7cfbff-7br98" Oct 31 12:32:49.021613 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:49.021527 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c49bcfc5-1bc7-46fc-85ca-5be1fa784116-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f7b7cfbff-7br98\" (UID: \"c49bcfc5-1bc7-46fc-85ca-5be1fa784116\") " pod="openshift-monitoring/telemeter-client-6f7b7cfbff-7br98" Oct 31 12:32:49.021738 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:49.021609 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c49bcfc5-1bc7-46fc-85ca-5be1fa784116-telemeter-client-tls\") pod \"telemeter-client-6f7b7cfbff-7br98\" (UID: \"c49bcfc5-1bc7-46fc-85ca-5be1fa784116\") " pod="openshift-monitoring/telemeter-client-6f7b7cfbff-7br98" Oct 31 12:32:49.021738 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:49.021680 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6f6z\" (UniqueName: \"kubernetes.io/projected/c49bcfc5-1bc7-46fc-85ca-5be1fa784116-kube-api-access-n6f6z\") pod \"telemeter-client-6f7b7cfbff-7br98\" (UID: \"c49bcfc5-1bc7-46fc-85ca-5be1fa784116\") " pod="openshift-monitoring/telemeter-client-6f7b7cfbff-7br98" Oct 31 12:32:49.021738 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:49.021710 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c49bcfc5-1bc7-46fc-85ca-5be1fa784116-serving-certs-ca-bundle\") pod \"telemeter-client-6f7b7cfbff-7br98\" (UID: \"c49bcfc5-1bc7-46fc-85ca-5be1fa784116\") " pod="openshift-monitoring/telemeter-client-6f7b7cfbff-7br98" Oct 31 12:32:49.021738 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:49.021730 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c49bcfc5-1bc7-46fc-85ca-5be1fa784116-secret-telemeter-client\") pod \"telemeter-client-6f7b7cfbff-7br98\" (UID: \"c49bcfc5-1bc7-46fc-85ca-5be1fa784116\") " pod="openshift-monitoring/telemeter-client-6f7b7cfbff-7br98" Oct 31 12:32:49.122534 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:49.122501 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c49bcfc5-1bc7-46fc-85ca-5be1fa784116-secret-telemeter-client\") pod \"telemeter-client-6f7b7cfbff-7br98\" (UID: \"c49bcfc5-1bc7-46fc-85ca-5be1fa784116\") " pod="openshift-monitoring/telemeter-client-6f7b7cfbff-7br98" Oct 31 12:32:49.122534 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:49.122540 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c49bcfc5-1bc7-46fc-85ca-5be1fa784116-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f7b7cfbff-7br98\" (UID: \"c49bcfc5-1bc7-46fc-85ca-5be1fa784116\") " pod="openshift-monitoring/telemeter-client-6f7b7cfbff-7br98" Oct 31 12:32:49.122810 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:49.122566 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c49bcfc5-1bc7-46fc-85ca-5be1fa784116-federate-client-tls\") pod \"telemeter-client-6f7b7cfbff-7br98\" (UID: \"c49bcfc5-1bc7-46fc-85ca-5be1fa784116\") " pod="openshift-monitoring/telemeter-client-6f7b7cfbff-7br98" Oct 31 12:32:49.122810 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:49.122597 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c49bcfc5-1bc7-46fc-85ca-5be1fa784116-metrics-client-ca\") pod \"telemeter-client-6f7b7cfbff-7br98\" (UID: \"c49bcfc5-1bc7-46fc-85ca-5be1fa784116\") " pod="openshift-monitoring/telemeter-client-6f7b7cfbff-7br98" Oct 31 12:32:49.122810 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:49.122626 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c49bcfc5-1bc7-46fc-85ca-5be1fa784116-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f7b7cfbff-7br98\" (UID: \"c49bcfc5-1bc7-46fc-85ca-5be1fa784116\") " pod="openshift-monitoring/telemeter-client-6f7b7cfbff-7br98" Oct 31 12:32:49.122810 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:49.122720 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c49bcfc5-1bc7-46fc-85ca-5be1fa784116-telemeter-client-tls\") pod \"telemeter-client-6f7b7cfbff-7br98\" (UID: \"c49bcfc5-1bc7-46fc-85ca-5be1fa784116\") " pod="openshift-monitoring/telemeter-client-6f7b7cfbff-7br98" Oct 31 12:32:49.122810 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:49.122779 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6f6z\" (UniqueName: \"kubernetes.io/projected/c49bcfc5-1bc7-46fc-85ca-5be1fa784116-kube-api-access-n6f6z\") pod \"telemeter-client-6f7b7cfbff-7br98\" (UID: \"c49bcfc5-1bc7-46fc-85ca-5be1fa784116\") " pod="openshift-monitoring/telemeter-client-6f7b7cfbff-7br98" Oct 31 12:32:49.123062 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:49.122812 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c49bcfc5-1bc7-46fc-85ca-5be1fa784116-serving-certs-ca-bundle\") pod \"telemeter-client-6f7b7cfbff-7br98\" (UID: \"c49bcfc5-1bc7-46fc-85ca-5be1fa784116\") " pod="openshift-monitoring/telemeter-client-6f7b7cfbff-7br98" Oct 31 12:32:49.123062 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:49.122988 5693 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 12:32:49.123540 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:49.123516 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c49bcfc5-1bc7-46fc-85ca-5be1fa784116-metrics-client-ca\") pod \"telemeter-client-6f7b7cfbff-7br98\" (UID: \"c49bcfc5-1bc7-46fc-85ca-5be1fa784116\") " pod="openshift-monitoring/telemeter-client-6f7b7cfbff-7br98" Oct 31 12:32:49.123892 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:49.123867 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c49bcfc5-1bc7-46fc-85ca-5be1fa784116-serving-certs-ca-bundle\") pod \"telemeter-client-6f7b7cfbff-7br98\" (UID: \"c49bcfc5-1bc7-46fc-85ca-5be1fa784116\") " pod="openshift-monitoring/telemeter-client-6f7b7cfbff-7br98" Oct 31 12:32:49.123892 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:49.123882 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c49bcfc5-1bc7-46fc-85ca-5be1fa784116-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f7b7cfbff-7br98\" (UID: \"c49bcfc5-1bc7-46fc-85ca-5be1fa784116\") " pod="openshift-monitoring/telemeter-client-6f7b7cfbff-7br98" Oct 31 12:32:49.126147 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:49.126100 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c49bcfc5-1bc7-46fc-85ca-5be1fa784116-secret-telemeter-client\") pod \"telemeter-client-6f7b7cfbff-7br98\" (UID: \"c49bcfc5-1bc7-46fc-85ca-5be1fa784116\") " pod="openshift-monitoring/telemeter-client-6f7b7cfbff-7br98" Oct 31 12:32:49.126714 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:49.126696 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c49bcfc5-1bc7-46fc-85ca-5be1fa784116-telemeter-client-tls\") pod \"telemeter-client-6f7b7cfbff-7br98\" (UID: \"c49bcfc5-1bc7-46fc-85ca-5be1fa784116\") " pod="openshift-monitoring/telemeter-client-6f7b7cfbff-7br98" Oct 31 12:32:49.127747 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:49.127727 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c49bcfc5-1bc7-46fc-85ca-5be1fa784116-federate-client-tls\") pod \"telemeter-client-6f7b7cfbff-7br98\" (UID: \"c49bcfc5-1bc7-46fc-85ca-5be1fa784116\") " pod="openshift-monitoring/telemeter-client-6f7b7cfbff-7br98" Oct 31 12:32:49.127928 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:49.127907 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c49bcfc5-1bc7-46fc-85ca-5be1fa784116-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f7b7cfbff-7br98\" (UID: \"c49bcfc5-1bc7-46fc-85ca-5be1fa784116\") " pod="openshift-monitoring/telemeter-client-6f7b7cfbff-7br98" Oct 31 12:32:49.137088 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:49.137064 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6f6z\" (UniqueName: \"kubernetes.io/projected/c49bcfc5-1bc7-46fc-85ca-5be1fa784116-kube-api-access-n6f6z\") pod \"telemeter-client-6f7b7cfbff-7br98\" (UID: \"c49bcfc5-1bc7-46fc-85ca-5be1fa784116\") " pod="openshift-monitoring/telemeter-client-6f7b7cfbff-7br98" Oct 31 12:32:49.203876 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:49.203838 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6f7b7cfbff-7br98" Oct 31 12:32:49.487297 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:49.487266 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6f7b7cfbff-7br98"] Oct 31 12:32:49.493334 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:49.493317 5693 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 12:32:49.522565 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:49.522493 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f7b7cfbff-7br98" event={"ID":"c49bcfc5-1bc7-46fc-85ca-5be1fa784116","Type":"ContainerStarted","Data":"42aec7ec6d58330bb97c0a330f2ec83b01c06bb81222accee10c0e87cc334709"} Oct 31 12:32:51.530985 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:51.530951 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f7b7cfbff-7br98" event={"ID":"c49bcfc5-1bc7-46fc-85ca-5be1fa784116","Type":"ContainerStarted","Data":"38f3e632144861a4a8be362480f24b5a500da6777489c2bb18fcee64e90ea276"} Oct 31 12:32:52.536391 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:52.536350 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f7b7cfbff-7br98" event={"ID":"c49bcfc5-1bc7-46fc-85ca-5be1fa784116","Type":"ContainerStarted","Data":"3fefb60cf3c3cfd74dc2a83d37a02d7d3ac76b4abfda83269f090e8187918a1c"} Oct 31 12:32:52.536817 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:52.536437 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f7b7cfbff-7br98" event={"ID":"c49bcfc5-1bc7-46fc-85ca-5be1fa784116","Type":"ContainerStarted","Data":"2a6a8e04f26b89c39cb0d8f88c3191c97a8430ce692b22a69796d4312091c241"} Oct 31 12:32:52.563580 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:52.563506 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6f7b7cfbff-7br98" podStartSLOduration=2.031699148 podStartE2EDuration="4.563463798s" totalImagesPullingTime="2.53176465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:32:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:52.561939001 +0000 UTC m=+23.269525296" watchObservedRunningTime="2026-10-31 12:32:52.563463798 +0000 UTC m=+23.271050093" Oct 31 12:32:52.659537 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:52.659508 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-776f6d6cb8-qxwhb"] Oct 31 12:32:52.662553 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:52.662537 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-776f6d6cb8-qxwhb" Oct 31 12:32:52.674982 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:52.674953 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-776f6d6cb8-qxwhb"] Oct 31 12:32:52.755882 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:52.755842 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-trusted-ca-bundle\") pod \"console-776f6d6cb8-qxwhb\" (UID: \"17d0bf06-5cb9-4a79-9ff8-817678cf57b3\") " pod="openshift-console/console-776f6d6cb8-qxwhb" Oct 31 12:32:52.756058 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:52.755895 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-oauth-serving-cert\") pod \"console-776f6d6cb8-qxwhb\" (UID: \"17d0bf06-5cb9-4a79-9ff8-817678cf57b3\") " pod="openshift-console/console-776f6d6cb8-qxwhb" Oct 31 12:32:52.756058 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:52.755988 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-tmp\") pod \"console-776f6d6cb8-qxwhb\" (UID: \"17d0bf06-5cb9-4a79-9ff8-817678cf57b3\") " pod="openshift-console/console-776f6d6cb8-qxwhb" Oct 31 12:32:52.756058 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:52.756045 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-console-oauth-config\") pod \"console-776f6d6cb8-qxwhb\" (UID: \"17d0bf06-5cb9-4a79-9ff8-817678cf57b3\") " pod="openshift-console/console-776f6d6cb8-qxwhb" Oct 31 12:32:52.756254 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:52.756110 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-console-config\") pod \"console-776f6d6cb8-qxwhb\" (UID: \"17d0bf06-5cb9-4a79-9ff8-817678cf57b3\") " pod="openshift-console/console-776f6d6cb8-qxwhb" Oct 31 12:32:52.756254 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:52.756153 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-service-ca\") pod \"console-776f6d6cb8-qxwhb\" (UID: \"17d0bf06-5cb9-4a79-9ff8-817678cf57b3\") " pod="openshift-console/console-776f6d6cb8-qxwhb" Oct 31 12:32:52.756254 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:52.756177 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqwjr\" (UniqueName: \"kubernetes.io/projected/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-kube-api-access-nqwjr\") pod \"console-776f6d6cb8-qxwhb\" (UID: \"17d0bf06-5cb9-4a79-9ff8-817678cf57b3\") " pod="openshift-console/console-776f6d6cb8-qxwhb" Oct 31 12:32:52.756254 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:52.756205 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-console-serving-cert\") pod \"console-776f6d6cb8-qxwhb\" (UID: \"17d0bf06-5cb9-4a79-9ff8-817678cf57b3\") " pod="openshift-console/console-776f6d6cb8-qxwhb" Oct 31 12:32:52.857378 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:52.857275 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-trusted-ca-bundle\") pod \"console-776f6d6cb8-qxwhb\" (UID: \"17d0bf06-5cb9-4a79-9ff8-817678cf57b3\") " pod="openshift-console/console-776f6d6cb8-qxwhb" Oct 31 12:32:52.857378 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:52.857353 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-oauth-serving-cert\") pod \"console-776f6d6cb8-qxwhb\" (UID: \"17d0bf06-5cb9-4a79-9ff8-817678cf57b3\") " pod="openshift-console/console-776f6d6cb8-qxwhb" Oct 31 12:32:52.857589 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:52.857384 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-tmp\") pod \"console-776f6d6cb8-qxwhb\" (UID: \"17d0bf06-5cb9-4a79-9ff8-817678cf57b3\") " pod="openshift-console/console-776f6d6cb8-qxwhb" Oct 31 12:32:52.857589 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:52.857438 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-console-oauth-config\") pod \"console-776f6d6cb8-qxwhb\" (UID: \"17d0bf06-5cb9-4a79-9ff8-817678cf57b3\") " pod="openshift-console/console-776f6d6cb8-qxwhb" Oct 31 12:32:52.857589 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:52.857465 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-console-config\") pod \"console-776f6d6cb8-qxwhb\" (UID: \"17d0bf06-5cb9-4a79-9ff8-817678cf57b3\") " pod="openshift-console/console-776f6d6cb8-qxwhb" Oct 31 12:32:52.857589 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:52.857496 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-service-ca\") pod \"console-776f6d6cb8-qxwhb\" (UID: \"17d0bf06-5cb9-4a79-9ff8-817678cf57b3\") " pod="openshift-console/console-776f6d6cb8-qxwhb" Oct 31 12:32:52.857589 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:52.857518 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqwjr\" (UniqueName: \"kubernetes.io/projected/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-kube-api-access-nqwjr\") pod \"console-776f6d6cb8-qxwhb\" (UID: \"17d0bf06-5cb9-4a79-9ff8-817678cf57b3\") " pod="openshift-console/console-776f6d6cb8-qxwhb" Oct 31 12:32:52.857589 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:52.857544 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-console-serving-cert\") pod \"console-776f6d6cb8-qxwhb\" (UID: \"17d0bf06-5cb9-4a79-9ff8-817678cf57b3\") " pod="openshift-console/console-776f6d6cb8-qxwhb" Oct 31 12:32:52.858022 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:52.857972 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-tmp\") pod \"console-776f6d6cb8-qxwhb\" (UID: \"17d0bf06-5cb9-4a79-9ff8-817678cf57b3\") " pod="openshift-console/console-776f6d6cb8-qxwhb" Oct 31 12:32:52.858291 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:52.858263 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-oauth-serving-cert\") pod \"console-776f6d6cb8-qxwhb\" (UID: \"17d0bf06-5cb9-4a79-9ff8-817678cf57b3\") " pod="openshift-console/console-776f6d6cb8-qxwhb" Oct 31 12:32:52.858382 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:52.858289 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-trusted-ca-bundle\") pod \"console-776f6d6cb8-qxwhb\" (UID: \"17d0bf06-5cb9-4a79-9ff8-817678cf57b3\") " pod="openshift-console/console-776f6d6cb8-qxwhb" Oct 31 12:32:52.858382 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:52.858301 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-console-config\") pod \"console-776f6d6cb8-qxwhb\" (UID: \"17d0bf06-5cb9-4a79-9ff8-817678cf57b3\") " pod="openshift-console/console-776f6d6cb8-qxwhb" Oct 31 12:32:52.858382 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:52.858329 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-service-ca\") pod \"console-776f6d6cb8-qxwhb\" (UID: \"17d0bf06-5cb9-4a79-9ff8-817678cf57b3\") " pod="openshift-console/console-776f6d6cb8-qxwhb" Oct 31 12:32:52.860212 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:52.860196 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-console-serving-cert\") pod \"console-776f6d6cb8-qxwhb\" (UID: \"17d0bf06-5cb9-4a79-9ff8-817678cf57b3\") " pod="openshift-console/console-776f6d6cb8-qxwhb" Oct 31 12:32:52.860268 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:52.860225 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-console-oauth-config\") pod \"console-776f6d6cb8-qxwhb\" (UID: \"17d0bf06-5cb9-4a79-9ff8-817678cf57b3\") " pod="openshift-console/console-776f6d6cb8-qxwhb" Oct 31 12:32:52.869945 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:52.869924 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqwjr\" (UniqueName: \"kubernetes.io/projected/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-kube-api-access-nqwjr\") pod \"console-776f6d6cb8-qxwhb\" (UID: \"17d0bf06-5cb9-4a79-9ff8-817678cf57b3\") " pod="openshift-console/console-776f6d6cb8-qxwhb" Oct 31 12:32:52.973760 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:52.973722 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-776f6d6cb8-qxwhb" Oct 31 12:32:53.256758 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:53.256714 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-776f6d6cb8-qxwhb"] Oct 31 12:32:53.262551 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:32:53.262496 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17d0bf06_5cb9_4a79_9ff8_817678cf57b3.slice/crio-62892c691a15fcf4d20ccfa7271988ab40ee3f873acb3e7eccccd4cb83b1808e WatchSource:0}: Error finding container 62892c691a15fcf4d20ccfa7271988ab40ee3f873acb3e7eccccd4cb83b1808e: Status 404 returned error can't find the container with id 62892c691a15fcf4d20ccfa7271988ab40ee3f873acb3e7eccccd4cb83b1808e Oct 31 12:32:53.543796 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:53.543713 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-776f6d6cb8-qxwhb" event={"ID":"17d0bf06-5cb9-4a79-9ff8-817678cf57b3","Type":"ContainerStarted","Data":"cce8ec057b5eeaf6969cf8c12b2114e579b772c8f1ef4bd92979b01b6984db8a"} Oct 31 12:32:53.544163 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:53.543798 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-776f6d6cb8-qxwhb" event={"ID":"17d0bf06-5cb9-4a79-9ff8-817678cf57b3","Type":"ContainerStarted","Data":"62892c691a15fcf4d20ccfa7271988ab40ee3f873acb3e7eccccd4cb83b1808e"} Oct 31 12:32:53.565116 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:53.565068 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-776f6d6cb8-qxwhb" podStartSLOduration=1.565055746 podStartE2EDuration="1.565055746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:32:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:53.564159189 +0000 UTC m=+24.271745485" watchObservedRunningTime="2026-10-31 12:32:53.565055746 +0000 UTC m=+24.272642040" Oct 31 12:32:54.623118 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:54.623090 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:54.626977 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:54.626954 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-847b6d5877-sxkd8" Oct 31 12:32:55.391825 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:55.391788 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-crhzp" Oct 31 12:32:56.484973 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:56.484934 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-57b586f5c7-hbnwz" podUID="8c91a35c-7933-4399-b7ff-4c1226d72d20" containerName="console" containerID="cri-o://51509f6a0062fbd3559a7d18bd83266e30fc7b243607755ee7b545ba47b32ff9" gracePeriod=15 Oct 31 12:32:56.732628 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:56.732605 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:56.894246 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:56.894212 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c91a35c-7933-4399-b7ff-4c1226d72d20-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c91a35c-7933-4399-b7ff-4c1226d72d20-oauth-serving-cert\") pod \"8c91a35c-7933-4399-b7ff-4c1226d72d20\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " Oct 31 12:32:56.894408 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:56.894281 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c91a35c-7933-4399-b7ff-4c1226d72d20-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c91a35c-7933-4399-b7ff-4c1226d72d20-tmp\") pod \"8c91a35c-7933-4399-b7ff-4c1226d72d20\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " Oct 31 12:32:56.894408 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:56.894318 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c91a35c-7933-4399-b7ff-4c1226d72d20-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c91a35c-7933-4399-b7ff-4c1226d72d20-console-serving-cert\") pod \"8c91a35c-7933-4399-b7ff-4c1226d72d20\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " Oct 31 12:32:56.894408 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:56.894386 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c91a35c-7933-4399-b7ff-4c1226d72d20-kube-api-access-ls87f\" (UniqueName: \"kubernetes.io/projected/8c91a35c-7933-4399-b7ff-4c1226d72d20-kube-api-access-ls87f\") pod \"8c91a35c-7933-4399-b7ff-4c1226d72d20\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " Oct 31 12:32:56.894558 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:56.894424 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c91a35c-7933-4399-b7ff-4c1226d72d20-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c91a35c-7933-4399-b7ff-4c1226d72d20-service-ca\") pod \"8c91a35c-7933-4399-b7ff-4c1226d72d20\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " Oct 31 12:32:56.894558 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:56.894462 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c91a35c-7933-4399-b7ff-4c1226d72d20-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c91a35c-7933-4399-b7ff-4c1226d72d20-console-oauth-config\") pod \"8c91a35c-7933-4399-b7ff-4c1226d72d20\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " Oct 31 12:32:56.894558 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:56.894487 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c91a35c-7933-4399-b7ff-4c1226d72d20-console-config\" (UniqueName: \"kubernetes.io/configmap/8c91a35c-7933-4399-b7ff-4c1226d72d20-console-config\") pod \"8c91a35c-7933-4399-b7ff-4c1226d72d20\" (UID: \"8c91a35c-7933-4399-b7ff-4c1226d72d20\") " Oct 31 12:32:56.894773 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:56.894748 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c91a35c-7933-4399-b7ff-4c1226d72d20-oauth-serving-cert" pod "8c91a35c-7933-4399-b7ff-4c1226d72d20" (UID: "8c91a35c-7933-4399-b7ff-4c1226d72d20"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:32:56.894922 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:56.894895 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c91a35c-7933-4399-b7ff-4c1226d72d20-tmp" pod "8c91a35c-7933-4399-b7ff-4c1226d72d20" (UID: "8c91a35c-7933-4399-b7ff-4c1226d72d20"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:32:56.894996 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:56.894832 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c91a35c-7933-4399-b7ff-4c1226d72d20-service-ca" pod "8c91a35c-7933-4399-b7ff-4c1226d72d20" (UID: "8c91a35c-7933-4399-b7ff-4c1226d72d20"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:32:56.894996 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:56.894982 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c91a35c-7933-4399-b7ff-4c1226d72d20-console-config" pod "8c91a35c-7933-4399-b7ff-4c1226d72d20" (UID: "8c91a35c-7933-4399-b7ff-4c1226d72d20"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:32:56.896915 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:56.896892 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c91a35c-7933-4399-b7ff-4c1226d72d20-kube-api-access-ls87f" pod "8c91a35c-7933-4399-b7ff-4c1226d72d20" (UID: "8c91a35c-7933-4399-b7ff-4c1226d72d20"). InnerVolumeSpecName "kube-api-access-ls87f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:32:56.897037 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:56.896908 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c91a35c-7933-4399-b7ff-4c1226d72d20-console-serving-cert" pod "8c91a35c-7933-4399-b7ff-4c1226d72d20" (UID: "8c91a35c-7933-4399-b7ff-4c1226d72d20"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:32:56.897037 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:56.896963 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c91a35c-7933-4399-b7ff-4c1226d72d20-console-oauth-config" pod "8c91a35c-7933-4399-b7ff-4c1226d72d20" (UID: "8c91a35c-7933-4399-b7ff-4c1226d72d20"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:32:56.996098 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:56.996058 5693 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c91a35c-7933-4399-b7ff-4c1226d72d20-tmp\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:56.996098 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:56.996105 5693 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c91a35c-7933-4399-b7ff-4c1226d72d20-console-serving-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:56.996294 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:56.996116 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls87f\" (UniqueName: \"kubernetes.io/projected/8c91a35c-7933-4399-b7ff-4c1226d72d20-kube-api-access-ls87f\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:56.996294 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:56.996127 5693 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c91a35c-7933-4399-b7ff-4c1226d72d20-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:56.996294 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:56.996135 5693 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c91a35c-7933-4399-b7ff-4c1226d72d20-console-oauth-config\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:56.996294 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:56.996144 5693 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8c91a35c-7933-4399-b7ff-4c1226d72d20-console-config\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:56.996294 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:56.996152 5693 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c91a35c-7933-4399-b7ff-4c1226d72d20-oauth-serving-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:57.560281 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:57.560247 5693 generic.go:356] "Generic (PLEG): container finished" podID="8c91a35c-7933-4399-b7ff-4c1226d72d20" containerID="51509f6a0062fbd3559a7d18bd83266e30fc7b243607755ee7b545ba47b32ff9" exitCode=0 Oct 31 12:32:57.560710 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:57.560303 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57b586f5c7-hbnwz" Oct 31 12:32:57.560710 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:57.560329 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57b586f5c7-hbnwz" event={"ID":"8c91a35c-7933-4399-b7ff-4c1226d72d20","Type":"ContainerDied","Data":"51509f6a0062fbd3559a7d18bd83266e30fc7b243607755ee7b545ba47b32ff9"} Oct 31 12:32:57.560710 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:57.560405 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57b586f5c7-hbnwz" event={"ID":"8c91a35c-7933-4399-b7ff-4c1226d72d20","Type":"ContainerDied","Data":"a633f11a094758484648b16d5585aebc43099d3f40a08aa8223d97224bf41d0e"} Oct 31 12:32:57.560710 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:57.560427 5693 scope.go:122] "RemoveContainer" containerID="51509f6a0062fbd3559a7d18bd83266e30fc7b243607755ee7b545ba47b32ff9" Oct 31 12:32:57.568014 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:57.567999 5693 scope.go:122] "RemoveContainer" containerID="51509f6a0062fbd3559a7d18bd83266e30fc7b243607755ee7b545ba47b32ff9" Oct 31 12:32:57.568316 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:32:57.568293 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51509f6a0062fbd3559a7d18bd83266e30fc7b243607755ee7b545ba47b32ff9\": container with ID starting with 51509f6a0062fbd3559a7d18bd83266e30fc7b243607755ee7b545ba47b32ff9 not found: ID does not exist" containerID="51509f6a0062fbd3559a7d18bd83266e30fc7b243607755ee7b545ba47b32ff9" Oct 31 12:32:57.568413 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:57.568322 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51509f6a0062fbd3559a7d18bd83266e30fc7b243607755ee7b545ba47b32ff9"} err="failed to get container status \"51509f6a0062fbd3559a7d18bd83266e30fc7b243607755ee7b545ba47b32ff9\": rpc error: code = NotFound desc = could not find container \"51509f6a0062fbd3559a7d18bd83266e30fc7b243607755ee7b545ba47b32ff9\": container with ID starting with 51509f6a0062fbd3559a7d18bd83266e30fc7b243607755ee7b545ba47b32ff9 not found: ID does not exist" Oct 31 12:32:57.577632 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:57.577610 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57b586f5c7-hbnwz"] Oct 31 12:32:57.581626 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:57.581608 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57b586f5c7-hbnwz"] Oct 31 12:32:59.404907 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:59.404877 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c91a35c-7933-4399-b7ff-4c1226d72d20" path="/var/lib/kubelet/pods/8c91a35c-7933-4399-b7ff-4c1226d72d20/volumes" Oct 31 12:32:59.944607 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:32:59.944575 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bgcqc" Oct 31 12:33:02.974712 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:02.974677 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-776f6d6cb8-qxwhb" Oct 31 12:33:02.975212 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:02.974833 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-776f6d6cb8-qxwhb" Oct 31 12:33:02.979381 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:02.979362 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-776f6d6cb8-qxwhb" Oct 31 12:33:03.584687 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:03.584660 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-776f6d6cb8-qxwhb" Oct 31 12:33:03.639570 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:03.639537 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b4df7846b-2pjqf"] Oct 31 12:33:18.685152 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:33:18.685115 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:33:28.659432 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:28.659364 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5b4df7846b-2pjqf" podUID="0f343e95-e397-49aa-b023-98337d3be3ff" containerName="console" containerID="cri-o://7f97a460b9fdafec97b3b99dc1dc181965efb1f6ea45c7ca086f42e3aa0bf8c0" gracePeriod=15 Oct 31 12:33:28.896065 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:28.896040 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:33:28.933797 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:28.933719 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f343e95-e397-49aa-b023-98337d3be3ff-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0f343e95-e397-49aa-b023-98337d3be3ff-console-oauth-config\") pod \"0f343e95-e397-49aa-b023-98337d3be3ff\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " Oct 31 12:33:28.933797 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:28.933768 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f343e95-e397-49aa-b023-98337d3be3ff-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f343e95-e397-49aa-b023-98337d3be3ff-service-ca\") pod \"0f343e95-e397-49aa-b023-98337d3be3ff\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " Oct 31 12:33:28.934021 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:28.933801 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f343e95-e397-49aa-b023-98337d3be3ff-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f343e95-e397-49aa-b023-98337d3be3ff-trusted-ca-bundle\") pod \"0f343e95-e397-49aa-b023-98337d3be3ff\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " Oct 31 12:33:28.934021 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:28.933825 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f343e95-e397-49aa-b023-98337d3be3ff-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f343e95-e397-49aa-b023-98337d3be3ff-tmp\") pod \"0f343e95-e397-49aa-b023-98337d3be3ff\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " Oct 31 12:33:28.934021 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:28.933849 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f343e95-e397-49aa-b023-98337d3be3ff-kube-api-access-kpx9x\" (UniqueName: \"kubernetes.io/projected/0f343e95-e397-49aa-b023-98337d3be3ff-kube-api-access-kpx9x\") pod \"0f343e95-e397-49aa-b023-98337d3be3ff\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " Oct 31 12:33:28.934021 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:28.933875 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f343e95-e397-49aa-b023-98337d3be3ff-console-config\" (UniqueName: \"kubernetes.io/configmap/0f343e95-e397-49aa-b023-98337d3be3ff-console-config\") pod \"0f343e95-e397-49aa-b023-98337d3be3ff\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " Oct 31 12:33:28.934021 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:28.933901 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f343e95-e397-49aa-b023-98337d3be3ff-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f343e95-e397-49aa-b023-98337d3be3ff-console-serving-cert\") pod \"0f343e95-e397-49aa-b023-98337d3be3ff\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " Oct 31 12:33:28.934021 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:28.933940 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f343e95-e397-49aa-b023-98337d3be3ff-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0f343e95-e397-49aa-b023-98337d3be3ff-oauth-serving-cert\") pod \"0f343e95-e397-49aa-b023-98337d3be3ff\" (UID: \"0f343e95-e397-49aa-b023-98337d3be3ff\") " Oct 31 12:33:28.934311 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:28.934150 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f343e95-e397-49aa-b023-98337d3be3ff-tmp" pod "0f343e95-e397-49aa-b023-98337d3be3ff" (UID: "0f343e95-e397-49aa-b023-98337d3be3ff"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:33:28.934359 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:28.934323 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f343e95-e397-49aa-b023-98337d3be3ff-trusted-ca-bundle" pod "0f343e95-e397-49aa-b023-98337d3be3ff" (UID: "0f343e95-e397-49aa-b023-98337d3be3ff"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:33:28.934359 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:28.934239 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f343e95-e397-49aa-b023-98337d3be3ff-service-ca" pod "0f343e95-e397-49aa-b023-98337d3be3ff" (UID: "0f343e95-e397-49aa-b023-98337d3be3ff"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:33:28.934460 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:28.934441 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f343e95-e397-49aa-b023-98337d3be3ff-oauth-serving-cert" pod "0f343e95-e397-49aa-b023-98337d3be3ff" (UID: "0f343e95-e397-49aa-b023-98337d3be3ff"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:33:28.934564 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:28.934453 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f343e95-e397-49aa-b023-98337d3be3ff-console-config" pod "0f343e95-e397-49aa-b023-98337d3be3ff" (UID: "0f343e95-e397-49aa-b023-98337d3be3ff"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:33:28.936285 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:28.936259 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f343e95-e397-49aa-b023-98337d3be3ff-console-oauth-config" pod "0f343e95-e397-49aa-b023-98337d3be3ff" (UID: "0f343e95-e397-49aa-b023-98337d3be3ff"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:33:28.936415 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:28.936338 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f343e95-e397-49aa-b023-98337d3be3ff-console-serving-cert" pod "0f343e95-e397-49aa-b023-98337d3be3ff" (UID: "0f343e95-e397-49aa-b023-98337d3be3ff"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:33:28.936622 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:28.936604 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f343e95-e397-49aa-b023-98337d3be3ff-kube-api-access-kpx9x" pod "0f343e95-e397-49aa-b023-98337d3be3ff" (UID: "0f343e95-e397-49aa-b023-98337d3be3ff"). InnerVolumeSpecName "kube-api-access-kpx9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:33:29.034664 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:29.034592 5693 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0f343e95-e397-49aa-b023-98337d3be3ff-console-config\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:33:29.034664 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:29.034635 5693 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f343e95-e397-49aa-b023-98337d3be3ff-console-serving-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:33:29.034664 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:29.034666 5693 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0f343e95-e397-49aa-b023-98337d3be3ff-oauth-serving-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:33:29.034664 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:29.034674 5693 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0f343e95-e397-49aa-b023-98337d3be3ff-console-oauth-config\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:33:29.034664 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:29.034684 5693 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0f343e95-e397-49aa-b023-98337d3be3ff-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:33:29.034957 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:29.034693 5693 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f343e95-e397-49aa-b023-98337d3be3ff-trusted-ca-bundle\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:33:29.034957 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:29.034701 5693 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f343e95-e397-49aa-b023-98337d3be3ff-tmp\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:33:29.034957 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:29.034709 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpx9x\" (UniqueName: \"kubernetes.io/projected/0f343e95-e397-49aa-b023-98337d3be3ff-kube-api-access-kpx9x\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:33:29.670622 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:29.670588 5693 generic.go:356] "Generic (PLEG): container finished" podID="0f343e95-e397-49aa-b023-98337d3be3ff" containerID="7f97a460b9fdafec97b3b99dc1dc181965efb1f6ea45c7ca086f42e3aa0bf8c0" exitCode=0 Oct 31 12:33:29.671005 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:29.670672 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b4df7846b-2pjqf" Oct 31 12:33:29.671005 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:29.670681 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b4df7846b-2pjqf" event={"ID":"0f343e95-e397-49aa-b023-98337d3be3ff","Type":"ContainerDied","Data":"7f97a460b9fdafec97b3b99dc1dc181965efb1f6ea45c7ca086f42e3aa0bf8c0"} Oct 31 12:33:29.671005 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:29.670734 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b4df7846b-2pjqf" event={"ID":"0f343e95-e397-49aa-b023-98337d3be3ff","Type":"ContainerDied","Data":"4334c12d573521f95b214022be018ae2961732b1ee6e21a1d7c07ef1b08b3dce"} Oct 31 12:33:29.671005 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:29.670755 5693 scope.go:122] "RemoveContainer" containerID="7f97a460b9fdafec97b3b99dc1dc181965efb1f6ea45c7ca086f42e3aa0bf8c0" Oct 31 12:33:29.678173 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:29.678157 5693 scope.go:122] "RemoveContainer" containerID="7f97a460b9fdafec97b3b99dc1dc181965efb1f6ea45c7ca086f42e3aa0bf8c0" Oct 31 12:33:29.678441 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:33:29.678420 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f97a460b9fdafec97b3b99dc1dc181965efb1f6ea45c7ca086f42e3aa0bf8c0\": container with ID starting with 7f97a460b9fdafec97b3b99dc1dc181965efb1f6ea45c7ca086f42e3aa0bf8c0 not found: ID does not exist" containerID="7f97a460b9fdafec97b3b99dc1dc181965efb1f6ea45c7ca086f42e3aa0bf8c0" Oct 31 12:33:29.678493 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:29.678453 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f97a460b9fdafec97b3b99dc1dc181965efb1f6ea45c7ca086f42e3aa0bf8c0"} err="failed to get container status \"7f97a460b9fdafec97b3b99dc1dc181965efb1f6ea45c7ca086f42e3aa0bf8c0\": rpc error: code = NotFound desc = could not find container \"7f97a460b9fdafec97b3b99dc1dc181965efb1f6ea45c7ca086f42e3aa0bf8c0\": container with ID starting with 7f97a460b9fdafec97b3b99dc1dc181965efb1f6ea45c7ca086f42e3aa0bf8c0 not found: ID does not exist" Oct 31 12:33:29.687046 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:29.687025 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b4df7846b-2pjqf"] Oct 31 12:33:29.691208 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:29.691188 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5b4df7846b-2pjqf"] Oct 31 12:33:31.404903 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:31.404869 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f343e95-e397-49aa-b023-98337d3be3ff" path="/var/lib/kubelet/pods/0f343e95-e397-49aa-b023-98337d3be3ff/volumes" Oct 31 12:33:48.686102 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:33:48.686066 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:33:48.791599 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:48.791563 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-84bd68cdfb-b2vg6"] Oct 31 12:33:48.791836 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:48.791820 5693 cpu_manager.go:414] "containerMap: removing container" podUID="0f343e95-e397-49aa-b023-98337d3be3ff" containerName="console" Oct 31 12:33:48.791900 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:48.791841 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f343e95-e397-49aa-b023-98337d3be3ff" containerName="console" Oct 31 12:33:48.791900 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:48.791855 5693 cpu_manager.go:414] "containerMap: removing container" podUID="8c91a35c-7933-4399-b7ff-4c1226d72d20" containerName="console" Oct 31 12:33:48.791900 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:48.791882 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c91a35c-7933-4399-b7ff-4c1226d72d20" containerName="console" Oct 31 12:33:48.791994 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:48.791951 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f343e95-e397-49aa-b023-98337d3be3ff" containerName="console" Oct 31 12:33:48.791994 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:48.791959 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c91a35c-7933-4399-b7ff-4c1226d72d20" containerName="console" Oct 31 12:33:48.794609 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:48.794593 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84bd68cdfb-b2vg6" Oct 31 12:33:48.811368 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:48.811343 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84bd68cdfb-b2vg6"] Oct 31 12:33:48.888931 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:48.888899 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/84a1aeec-6740-42f9-b7da-83148de0f73a-console-oauth-config\") pod \"console-84bd68cdfb-b2vg6\" (UID: \"84a1aeec-6740-42f9-b7da-83148de0f73a\") " pod="openshift-console/console-84bd68cdfb-b2vg6" Oct 31 12:33:48.889121 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:48.888940 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/84a1aeec-6740-42f9-b7da-83148de0f73a-oauth-serving-cert\") pod \"console-84bd68cdfb-b2vg6\" (UID: \"84a1aeec-6740-42f9-b7da-83148de0f73a\") " pod="openshift-console/console-84bd68cdfb-b2vg6" Oct 31 12:33:48.889121 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:48.888963 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/84a1aeec-6740-42f9-b7da-83148de0f73a-console-serving-cert\") pod \"console-84bd68cdfb-b2vg6\" (UID: \"84a1aeec-6740-42f9-b7da-83148de0f73a\") " pod="openshift-console/console-84bd68cdfb-b2vg6" Oct 31 12:33:48.889121 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:48.888979 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrqsx\" (UniqueName: \"kubernetes.io/projected/84a1aeec-6740-42f9-b7da-83148de0f73a-kube-api-access-mrqsx\") pod \"console-84bd68cdfb-b2vg6\" (UID: \"84a1aeec-6740-42f9-b7da-83148de0f73a\") " pod="openshift-console/console-84bd68cdfb-b2vg6" Oct 31 12:33:48.889121 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:48.889094 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/84a1aeec-6740-42f9-b7da-83148de0f73a-service-ca\") pod \"console-84bd68cdfb-b2vg6\" (UID: \"84a1aeec-6740-42f9-b7da-83148de0f73a\") " pod="openshift-console/console-84bd68cdfb-b2vg6" Oct 31 12:33:48.889121 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:48.889117 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84a1aeec-6740-42f9-b7da-83148de0f73a-tmp\") pod \"console-84bd68cdfb-b2vg6\" (UID: \"84a1aeec-6740-42f9-b7da-83148de0f73a\") " pod="openshift-console/console-84bd68cdfb-b2vg6" Oct 31 12:33:48.889336 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:48.889143 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/84a1aeec-6740-42f9-b7da-83148de0f73a-console-config\") pod \"console-84bd68cdfb-b2vg6\" (UID: \"84a1aeec-6740-42f9-b7da-83148de0f73a\") " pod="openshift-console/console-84bd68cdfb-b2vg6" Oct 31 12:33:48.889336 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:48.889159 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84a1aeec-6740-42f9-b7da-83148de0f73a-trusted-ca-bundle\") pod \"console-84bd68cdfb-b2vg6\" (UID: \"84a1aeec-6740-42f9-b7da-83148de0f73a\") " pod="openshift-console/console-84bd68cdfb-b2vg6" Oct 31 12:33:48.989796 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:48.989726 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84a1aeec-6740-42f9-b7da-83148de0f73a-tmp\") pod \"console-84bd68cdfb-b2vg6\" (UID: \"84a1aeec-6740-42f9-b7da-83148de0f73a\") " pod="openshift-console/console-84bd68cdfb-b2vg6" Oct 31 12:33:48.989796 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:48.989773 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/84a1aeec-6740-42f9-b7da-83148de0f73a-console-config\") pod \"console-84bd68cdfb-b2vg6\" (UID: \"84a1aeec-6740-42f9-b7da-83148de0f73a\") " pod="openshift-console/console-84bd68cdfb-b2vg6" Oct 31 12:33:48.989796 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:48.989791 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84a1aeec-6740-42f9-b7da-83148de0f73a-trusted-ca-bundle\") pod \"console-84bd68cdfb-b2vg6\" (UID: \"84a1aeec-6740-42f9-b7da-83148de0f73a\") " pod="openshift-console/console-84bd68cdfb-b2vg6" Oct 31 12:33:48.989986 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:48.989822 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/84a1aeec-6740-42f9-b7da-83148de0f73a-console-oauth-config\") pod \"console-84bd68cdfb-b2vg6\" (UID: \"84a1aeec-6740-42f9-b7da-83148de0f73a\") " pod="openshift-console/console-84bd68cdfb-b2vg6" Oct 31 12:33:48.989986 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:48.989855 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/84a1aeec-6740-42f9-b7da-83148de0f73a-oauth-serving-cert\") pod \"console-84bd68cdfb-b2vg6\" (UID: \"84a1aeec-6740-42f9-b7da-83148de0f73a\") " pod="openshift-console/console-84bd68cdfb-b2vg6" Oct 31 12:33:48.989986 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:48.989883 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/84a1aeec-6740-42f9-b7da-83148de0f73a-console-serving-cert\") pod \"console-84bd68cdfb-b2vg6\" (UID: \"84a1aeec-6740-42f9-b7da-83148de0f73a\") " pod="openshift-console/console-84bd68cdfb-b2vg6" Oct 31 12:33:48.989986 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:48.989907 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrqsx\" (UniqueName: \"kubernetes.io/projected/84a1aeec-6740-42f9-b7da-83148de0f73a-kube-api-access-mrqsx\") pod \"console-84bd68cdfb-b2vg6\" (UID: \"84a1aeec-6740-42f9-b7da-83148de0f73a\") " pod="openshift-console/console-84bd68cdfb-b2vg6" Oct 31 12:33:48.989986 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:48.989943 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/84a1aeec-6740-42f9-b7da-83148de0f73a-service-ca\") pod \"console-84bd68cdfb-b2vg6\" (UID: \"84a1aeec-6740-42f9-b7da-83148de0f73a\") " pod="openshift-console/console-84bd68cdfb-b2vg6" Oct 31 12:33:48.990270 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:48.990246 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84a1aeec-6740-42f9-b7da-83148de0f73a-tmp\") pod \"console-84bd68cdfb-b2vg6\" (UID: \"84a1aeec-6740-42f9-b7da-83148de0f73a\") " pod="openshift-console/console-84bd68cdfb-b2vg6" Oct 31 12:33:48.990690 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:48.990612 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/84a1aeec-6740-42f9-b7da-83148de0f73a-console-config\") pod \"console-84bd68cdfb-b2vg6\" (UID: \"84a1aeec-6740-42f9-b7da-83148de0f73a\") " pod="openshift-console/console-84bd68cdfb-b2vg6" Oct 31 12:33:48.990800 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:48.990699 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/84a1aeec-6740-42f9-b7da-83148de0f73a-service-ca\") pod \"console-84bd68cdfb-b2vg6\" (UID: \"84a1aeec-6740-42f9-b7da-83148de0f73a\") " pod="openshift-console/console-84bd68cdfb-b2vg6" Oct 31 12:33:48.990886 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:48.990863 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/84a1aeec-6740-42f9-b7da-83148de0f73a-oauth-serving-cert\") pod \"console-84bd68cdfb-b2vg6\" (UID: \"84a1aeec-6740-42f9-b7da-83148de0f73a\") " pod="openshift-console/console-84bd68cdfb-b2vg6" Oct 31 12:33:48.990972 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:48.990950 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84a1aeec-6740-42f9-b7da-83148de0f73a-trusted-ca-bundle\") pod \"console-84bd68cdfb-b2vg6\" (UID: \"84a1aeec-6740-42f9-b7da-83148de0f73a\") " pod="openshift-console/console-84bd68cdfb-b2vg6" Oct 31 12:33:48.992518 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:48.992497 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/84a1aeec-6740-42f9-b7da-83148de0f73a-console-serving-cert\") pod \"console-84bd68cdfb-b2vg6\" (UID: \"84a1aeec-6740-42f9-b7da-83148de0f73a\") " pod="openshift-console/console-84bd68cdfb-b2vg6" Oct 31 12:33:48.992620 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:48.992524 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/84a1aeec-6740-42f9-b7da-83148de0f73a-console-oauth-config\") pod \"console-84bd68cdfb-b2vg6\" (UID: \"84a1aeec-6740-42f9-b7da-83148de0f73a\") " pod="openshift-console/console-84bd68cdfb-b2vg6" Oct 31 12:33:49.003379 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:49.003352 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrqsx\" (UniqueName: \"kubernetes.io/projected/84a1aeec-6740-42f9-b7da-83148de0f73a-kube-api-access-mrqsx\") pod \"console-84bd68cdfb-b2vg6\" (UID: \"84a1aeec-6740-42f9-b7da-83148de0f73a\") " pod="openshift-console/console-84bd68cdfb-b2vg6" Oct 31 12:33:49.106110 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:49.106066 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84bd68cdfb-b2vg6" Oct 31 12:33:49.379328 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:49.379250 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84bd68cdfb-b2vg6"] Oct 31 12:33:49.383126 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:33:49.383099 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84a1aeec_6740_42f9_b7da_83148de0f73a.slice/crio-032b28d145c3377dd7cc2a830305dd024af54590af85caeba86a1b6d5ebc7387 WatchSource:0}: Error finding container 032b28d145c3377dd7cc2a830305dd024af54590af85caeba86a1b6d5ebc7387: Status 404 returned error can't find the container with id 032b28d145c3377dd7cc2a830305dd024af54590af85caeba86a1b6d5ebc7387 Oct 31 12:33:49.737262 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:49.737225 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84bd68cdfb-b2vg6" event={"ID":"84a1aeec-6740-42f9-b7da-83148de0f73a","Type":"ContainerStarted","Data":"c1460b449e480fa521fd2fab0728ffac979f6b627ff467bfb6b98aa125e30f9b"} Oct 31 12:33:49.737671 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:49.737275 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84bd68cdfb-b2vg6" event={"ID":"84a1aeec-6740-42f9-b7da-83148de0f73a","Type":"ContainerStarted","Data":"032b28d145c3377dd7cc2a830305dd024af54590af85caeba86a1b6d5ebc7387"} Oct 31 12:33:49.758855 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:49.758810 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-84bd68cdfb-b2vg6" podStartSLOduration=1.758795946 podStartE2EDuration="1.758795946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:33:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:33:49.757941813 +0000 UTC m=+80.465528108" watchObservedRunningTime="2026-10-31 12:33:49.758795946 +0000 UTC m=+80.466382241" Oct 31 12:33:59.107184 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:59.107122 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84bd68cdfb-b2vg6" Oct 31 12:33:59.107598 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:59.107213 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84bd68cdfb-b2vg6" Oct 31 12:33:59.112016 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:59.111994 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84bd68cdfb-b2vg6" Oct 31 12:33:59.773561 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:59.773534 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84bd68cdfb-b2vg6" Oct 31 12:33:59.825091 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:33:59.825058 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-776f6d6cb8-qxwhb"] Oct 31 12:34:18.685678 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:34:18.685587 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:34:24.844844 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:34:24.844796 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-776f6d6cb8-qxwhb" podUID="17d0bf06-5cb9-4a79-9ff8-817678cf57b3" containerName="console" containerID="cri-o://cce8ec057b5eeaf6969cf8c12b2114e579b772c8f1ef4bd92979b01b6984db8a" gracePeriod=15 Oct 31 12:34:25.082400 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:34:25.082375 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-776f6d6cb8-qxwhb" Oct 31 12:34:25.163164 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:34:25.163134 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-tmp\" (UniqueName: \"kubernetes.io/empty-dir/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-tmp\") pod \"17d0bf06-5cb9-4a79-9ff8-817678cf57b3\" (UID: \"17d0bf06-5cb9-4a79-9ff8-817678cf57b3\") " Oct 31 12:34:25.163164 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:34:25.163169 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-console-config\" (UniqueName: \"kubernetes.io/configmap/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-console-config\") pod \"17d0bf06-5cb9-4a79-9ff8-817678cf57b3\" (UID: \"17d0bf06-5cb9-4a79-9ff8-817678cf57b3\") " Oct 31 12:34:25.163392 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:34:25.163197 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-console-oauth-config\") pod \"17d0bf06-5cb9-4a79-9ff8-817678cf57b3\" (UID: \"17d0bf06-5cb9-4a79-9ff8-817678cf57b3\") " Oct 31 12:34:25.163392 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:34:25.163217 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-console-serving-cert\") pod \"17d0bf06-5cb9-4a79-9ff8-817678cf57b3\" (UID: \"17d0bf06-5cb9-4a79-9ff8-817678cf57b3\") " Oct 31 12:34:25.163392 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:34:25.163240 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-oauth-serving-cert\") pod \"17d0bf06-5cb9-4a79-9ff8-817678cf57b3\" (UID: \"17d0bf06-5cb9-4a79-9ff8-817678cf57b3\") " Oct 31 12:34:25.163392 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:34:25.163262 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-trusted-ca-bundle\") pod \"17d0bf06-5cb9-4a79-9ff8-817678cf57b3\" (UID: \"17d0bf06-5cb9-4a79-9ff8-817678cf57b3\") " Oct 31 12:34:25.163392 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:34:25.163282 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-service-ca\" (UniqueName: \"kubernetes.io/configmap/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-service-ca\") pod \"17d0bf06-5cb9-4a79-9ff8-817678cf57b3\" (UID: \"17d0bf06-5cb9-4a79-9ff8-817678cf57b3\") " Oct 31 12:34:25.163392 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:34:25.163306 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-kube-api-access-nqwjr\" (UniqueName: \"kubernetes.io/projected/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-kube-api-access-nqwjr\") pod \"17d0bf06-5cb9-4a79-9ff8-817678cf57b3\" (UID: \"17d0bf06-5cb9-4a79-9ff8-817678cf57b3\") " Oct 31 12:34:25.163713 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:34:25.163452 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-tmp" pod "17d0bf06-5cb9-4a79-9ff8-817678cf57b3" (UID: "17d0bf06-5cb9-4a79-9ff8-817678cf57b3"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:34:25.163713 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:34:25.163532 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-console-config" pod "17d0bf06-5cb9-4a79-9ff8-817678cf57b3" (UID: "17d0bf06-5cb9-4a79-9ff8-817678cf57b3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:34:25.163802 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:34:25.163783 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-oauth-serving-cert" pod "17d0bf06-5cb9-4a79-9ff8-817678cf57b3" (UID: "17d0bf06-5cb9-4a79-9ff8-817678cf57b3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:34:25.163866 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:34:25.163807 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-trusted-ca-bundle" pod "17d0bf06-5cb9-4a79-9ff8-817678cf57b3" (UID: "17d0bf06-5cb9-4a79-9ff8-817678cf57b3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:34:25.163866 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:34:25.163820 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-service-ca" pod "17d0bf06-5cb9-4a79-9ff8-817678cf57b3" (UID: "17d0bf06-5cb9-4a79-9ff8-817678cf57b3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:34:25.165583 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:34:25.165556 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-console-oauth-config" pod "17d0bf06-5cb9-4a79-9ff8-817678cf57b3" (UID: "17d0bf06-5cb9-4a79-9ff8-817678cf57b3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:34:25.165719 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:34:25.165604 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-kube-api-access-nqwjr" pod "17d0bf06-5cb9-4a79-9ff8-817678cf57b3" (UID: "17d0bf06-5cb9-4a79-9ff8-817678cf57b3"). InnerVolumeSpecName "kube-api-access-nqwjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:34:25.165781 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:34:25.165711 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-console-serving-cert" pod "17d0bf06-5cb9-4a79-9ff8-817678cf57b3" (UID: "17d0bf06-5cb9-4a79-9ff8-817678cf57b3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:34:25.263766 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:34:25.263725 5693 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-oauth-serving-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:34:25.263766 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:34:25.263759 5693 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-trusted-ca-bundle\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:34:25.263766 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:34:25.263771 5693 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:34:25.264009 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:34:25.263784 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqwjr\" (UniqueName: \"kubernetes.io/projected/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-kube-api-access-nqwjr\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:34:25.264009 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:34:25.263797 5693 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-tmp\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:34:25.264009 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:34:25.263808 5693 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-console-config\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:34:25.264009 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:34:25.263818 5693 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-console-oauth-config\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:34:25.264009 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:34:25.263829 5693 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17d0bf06-5cb9-4a79-9ff8-817678cf57b3-console-serving-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:34:25.852410 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:34:25.852372 5693 generic.go:356] "Generic (PLEG): container finished" podID="17d0bf06-5cb9-4a79-9ff8-817678cf57b3" containerID="cce8ec057b5eeaf6969cf8c12b2114e579b772c8f1ef4bd92979b01b6984db8a" exitCode=0 Oct 31 12:34:25.852939 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:34:25.852424 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-776f6d6cb8-qxwhb" event={"ID":"17d0bf06-5cb9-4a79-9ff8-817678cf57b3","Type":"ContainerDied","Data":"cce8ec057b5eeaf6969cf8c12b2114e579b772c8f1ef4bd92979b01b6984db8a"} Oct 31 12:34:25.852939 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:34:25.852464 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-776f6d6cb8-qxwhb" event={"ID":"17d0bf06-5cb9-4a79-9ff8-817678cf57b3","Type":"ContainerDied","Data":"62892c691a15fcf4d20ccfa7271988ab40ee3f873acb3e7eccccd4cb83b1808e"} Oct 31 12:34:25.852939 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:34:25.852484 5693 scope.go:122] "RemoveContainer" containerID="cce8ec057b5eeaf6969cf8c12b2114e579b772c8f1ef4bd92979b01b6984db8a" Oct 31 12:34:25.852939 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:34:25.852425 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-776f6d6cb8-qxwhb" Oct 31 12:34:25.860171 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:34:25.860154 5693 scope.go:122] "RemoveContainer" containerID="cce8ec057b5eeaf6969cf8c12b2114e579b772c8f1ef4bd92979b01b6984db8a" Oct 31 12:34:25.860420 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:34:25.860402 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cce8ec057b5eeaf6969cf8c12b2114e579b772c8f1ef4bd92979b01b6984db8a\": container with ID starting with cce8ec057b5eeaf6969cf8c12b2114e579b772c8f1ef4bd92979b01b6984db8a not found: ID does not exist" containerID="cce8ec057b5eeaf6969cf8c12b2114e579b772c8f1ef4bd92979b01b6984db8a" Oct 31 12:34:25.860476 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:34:25.860429 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce8ec057b5eeaf6969cf8c12b2114e579b772c8f1ef4bd92979b01b6984db8a"} err="failed to get container status \"cce8ec057b5eeaf6969cf8c12b2114e579b772c8f1ef4bd92979b01b6984db8a\": rpc error: code = NotFound desc = could not find container \"cce8ec057b5eeaf6969cf8c12b2114e579b772c8f1ef4bd92979b01b6984db8a\": container with ID starting with cce8ec057b5eeaf6969cf8c12b2114e579b772c8f1ef4bd92979b01b6984db8a not found: ID does not exist" Oct 31 12:34:25.870394 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:34:25.870369 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-776f6d6cb8-qxwhb"] Oct 31 12:34:25.874797 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:34:25.874776 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-776f6d6cb8-qxwhb"] Oct 31 12:34:27.405316 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:34:27.405282 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17d0bf06-5cb9-4a79-9ff8-817678cf57b3" path="/var/lib/kubelet/pods/17d0bf06-5cb9-4a79-9ff8-817678cf57b3/volumes" Oct 31 12:34:48.685746 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:34:48.685703 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:35:18.684779 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:35:18.684737 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:35:48.685529 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:35:48.685445 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:36:08.631428 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:08.631388 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxt6r"] Oct 31 12:36:08.631946 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:08.631757 5693 cpu_manager.go:414] "containerMap: removing container" podUID="17d0bf06-5cb9-4a79-9ff8-817678cf57b3" containerName="console" Oct 31 12:36:08.631946 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:08.631773 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17d0bf06-5cb9-4a79-9ff8-817678cf57b3" containerName="console" Oct 31 12:36:08.631946 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:08.631850 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="17d0bf06-5cb9-4a79-9ff8-817678cf57b3" containerName="console" Oct 31 12:36:08.634844 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:08.634823 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxt6r" Oct 31 12:36:08.640137 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:08.640107 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 31 12:36:08.640137 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:08.640113 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-brjnq\"" Oct 31 12:36:08.659813 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:08.659791 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 31 12:36:08.675793 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:08.675767 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxt6r"] Oct 31 12:36:08.698343 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:08.698307 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98f41137-2279-4b64-a81f-58adacbc6c0e-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxt6r\" (UID: \"98f41137-2279-4b64-a81f-58adacbc6c0e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxt6r" Oct 31 12:36:08.698489 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:08.698366 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98f41137-2279-4b64-a81f-58adacbc6c0e-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxt6r\" (UID: \"98f41137-2279-4b64-a81f-58adacbc6c0e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxt6r" Oct 31 12:36:08.698489 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:08.698458 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcwlf\" (UniqueName: \"kubernetes.io/projected/98f41137-2279-4b64-a81f-58adacbc6c0e-kube-api-access-jcwlf\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxt6r\" (UID: \"98f41137-2279-4b64-a81f-58adacbc6c0e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxt6r" Oct 31 12:36:08.798968 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:08.798931 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98f41137-2279-4b64-a81f-58adacbc6c0e-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxt6r\" (UID: \"98f41137-2279-4b64-a81f-58adacbc6c0e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxt6r" Oct 31 12:36:08.799134 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:08.798979 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98f41137-2279-4b64-a81f-58adacbc6c0e-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxt6r\" (UID: \"98f41137-2279-4b64-a81f-58adacbc6c0e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxt6r" Oct 31 12:36:08.799134 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:08.799106 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcwlf\" (UniqueName: \"kubernetes.io/projected/98f41137-2279-4b64-a81f-58adacbc6c0e-kube-api-access-jcwlf\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxt6r\" (UID: \"98f41137-2279-4b64-a81f-58adacbc6c0e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxt6r" Oct 31 12:36:08.799362 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:08.799345 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98f41137-2279-4b64-a81f-58adacbc6c0e-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxt6r\" (UID: \"98f41137-2279-4b64-a81f-58adacbc6c0e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxt6r" Oct 31 12:36:08.799399 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:08.799376 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98f41137-2279-4b64-a81f-58adacbc6c0e-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxt6r\" (UID: \"98f41137-2279-4b64-a81f-58adacbc6c0e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxt6r" Oct 31 12:36:08.809342 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:08.809311 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716b4pm"] Oct 31 12:36:08.812747 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:08.812729 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716b4pm" Oct 31 12:36:08.814596 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:08.814573 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcwlf\" (UniqueName: \"kubernetes.io/projected/98f41137-2279-4b64-a81f-58adacbc6c0e-kube-api-access-jcwlf\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxt6r\" (UID: \"98f41137-2279-4b64-a81f-58adacbc6c0e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxt6r" Oct 31 12:36:08.819885 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:08.819867 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-nt2s4\"" Oct 31 12:36:08.829216 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:08.829195 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716b4pm"] Oct 31 12:36:08.900429 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:08.900400 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4f781b2-7a08-4e5b-b844-40f001caf7da-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716b4pm\" (UID: \"b4f781b2-7a08-4e5b-b844-40f001caf7da\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716b4pm" Oct 31 12:36:08.900612 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:08.900439 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4f781b2-7a08-4e5b-b844-40f001caf7da-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716b4pm\" (UID: \"b4f781b2-7a08-4e5b-b844-40f001caf7da\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716b4pm" Oct 31 12:36:08.900612 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:08.900464 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdxx4\" (UniqueName: \"kubernetes.io/projected/b4f781b2-7a08-4e5b-b844-40f001caf7da-kube-api-access-tdxx4\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716b4pm\" (UID: \"b4f781b2-7a08-4e5b-b844-40f001caf7da\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716b4pm" Oct 31 12:36:08.916907 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:08.916875 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-ckmml"] Oct 31 12:36:08.919902 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:08.919886 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ckmml" Oct 31 12:36:08.932280 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:08.932260 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 12:36:08.932419 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:08.932400 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 12:36:08.932494 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:08.932467 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 12:36:08.932494 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:08.932475 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 12:36:08.932580 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:08.932568 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 12:36:08.932798 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:08.932785 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xtsw2\"" Oct 31 12:36:08.948348 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:08.948320 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-ckmml"] Oct 31 12:36:08.950294 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:08.950274 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxt6r" Oct 31 12:36:08.982256 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:08.982226 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 12:36:09.001592 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:09.001051 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcvpc\" (UniqueName: \"kubernetes.io/projected/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-kube-api-access-zcvpc\") pod \"istiod-openshift-gateway-7876bfcb74-ckmml\" (UID: \"a8ca5a81-e73c-48e0-b4d6-bc6469003a4d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ckmml" Oct 31 12:36:09.001592 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:09.001099 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-ckmml\" (UID: \"a8ca5a81-e73c-48e0-b4d6-bc6469003a4d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ckmml" Oct 31 12:36:09.001592 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:09.001198 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-ckmml\" (UID: \"a8ca5a81-e73c-48e0-b4d6-bc6469003a4d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ckmml" Oct 31 12:36:09.001592 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:09.001239 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4f781b2-7a08-4e5b-b844-40f001caf7da-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716b4pm\" (UID: \"b4f781b2-7a08-4e5b-b844-40f001caf7da\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716b4pm" Oct 31 12:36:09.001592 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:09.001281 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4f781b2-7a08-4e5b-b844-40f001caf7da-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716b4pm\" (UID: \"b4f781b2-7a08-4e5b-b844-40f001caf7da\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716b4pm" Oct 31 12:36:09.002334 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:09.001833 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4f781b2-7a08-4e5b-b844-40f001caf7da-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716b4pm\" (UID: \"b4f781b2-7a08-4e5b-b844-40f001caf7da\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716b4pm" Oct 31 12:36:09.002334 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:09.002101 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4f781b2-7a08-4e5b-b844-40f001caf7da-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716b4pm\" (UID: \"b4f781b2-7a08-4e5b-b844-40f001caf7da\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716b4pm" Oct 31 12:36:09.002334 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:09.002144 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-ckmml\" (UID: \"a8ca5a81-e73c-48e0-b4d6-bc6469003a4d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ckmml" Oct 31 12:36:09.002334 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:09.002180 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-ckmml\" (UID: \"a8ca5a81-e73c-48e0-b4d6-bc6469003a4d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ckmml" Oct 31 12:36:09.002334 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:09.002215 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdxx4\" (UniqueName: \"kubernetes.io/projected/b4f781b2-7a08-4e5b-b844-40f001caf7da-kube-api-access-tdxx4\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716b4pm\" (UID: \"b4f781b2-7a08-4e5b-b844-40f001caf7da\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716b4pm" Oct 31 12:36:09.002334 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:09.002253 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-ckmml\" (UID: \"a8ca5a81-e73c-48e0-b4d6-bc6469003a4d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ckmml" Oct 31 12:36:09.002334 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:09.002311 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-ckmml\" (UID: \"a8ca5a81-e73c-48e0-b4d6-bc6469003a4d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ckmml" Oct 31 12:36:09.029988 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:09.029960 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdxx4\" (UniqueName: \"kubernetes.io/projected/b4f781b2-7a08-4e5b-b844-40f001caf7da-kube-api-access-tdxx4\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716b4pm\" (UID: \"b4f781b2-7a08-4e5b-b844-40f001caf7da\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716b4pm" Oct 31 12:36:09.103135 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:09.103089 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-ckmml\" (UID: \"a8ca5a81-e73c-48e0-b4d6-bc6469003a4d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ckmml" Oct 31 12:36:09.103135 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:09.103139 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-ckmml\" (UID: \"a8ca5a81-e73c-48e0-b4d6-bc6469003a4d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ckmml" Oct 31 12:36:09.103365 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:09.103175 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-ckmml\" (UID: \"a8ca5a81-e73c-48e0-b4d6-bc6469003a4d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ckmml" Oct 31 12:36:09.103365 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:09.103219 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-ckmml\" (UID: \"a8ca5a81-e73c-48e0-b4d6-bc6469003a4d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ckmml" Oct 31 12:36:09.103365 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:09.103270 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcvpc\" (UniqueName: \"kubernetes.io/projected/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-kube-api-access-zcvpc\") pod \"istiod-openshift-gateway-7876bfcb74-ckmml\" (UID: \"a8ca5a81-e73c-48e0-b4d6-bc6469003a4d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ckmml" Oct 31 12:36:09.103365 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:09.103298 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-ckmml\" (UID: \"a8ca5a81-e73c-48e0-b4d6-bc6469003a4d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ckmml" Oct 31 12:36:09.103365 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:09.103322 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-ckmml\" (UID: \"a8ca5a81-e73c-48e0-b4d6-bc6469003a4d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ckmml" Oct 31 12:36:09.103831 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:09.103789 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-ckmml\" (UID: \"a8ca5a81-e73c-48e0-b4d6-bc6469003a4d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ckmml" Oct 31 12:36:09.105904 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:09.105875 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-ckmml\" (UID: \"a8ca5a81-e73c-48e0-b4d6-bc6469003a4d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ckmml" Oct 31 12:36:09.105904 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:09.105893 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-ckmml\" (UID: \"a8ca5a81-e73c-48e0-b4d6-bc6469003a4d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ckmml" Oct 31 12:36:09.106110 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:09.106091 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-ckmml\" (UID: \"a8ca5a81-e73c-48e0-b4d6-bc6469003a4d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ckmml" Oct 31 12:36:09.106180 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:09.106154 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-ckmml\" (UID: \"a8ca5a81-e73c-48e0-b4d6-bc6469003a4d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ckmml" Oct 31 12:36:09.115834 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:09.115810 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcvpc\" (UniqueName: \"kubernetes.io/projected/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-kube-api-access-zcvpc\") pod \"istiod-openshift-gateway-7876bfcb74-ckmml\" (UID: \"a8ca5a81-e73c-48e0-b4d6-bc6469003a4d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ckmml" Oct 31 12:36:09.115834 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:09.115829 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-ckmml\" (UID: \"a8ca5a81-e73c-48e0-b4d6-bc6469003a4d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ckmml" Oct 31 12:36:09.133772 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:09.133746 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716b4pm" Oct 31 12:36:09.224912 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:09.224881 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxt6r"] Oct 31 12:36:09.228429 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:36:09.228403 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f41137_2279_4b64_a81f_58adacbc6c0e.slice/crio-5ca1b63c41bb3985a5b1eb018d6c0b09258bc080c463f089589bbc5ed1d4c3e8 WatchSource:0}: Error finding container 5ca1b63c41bb3985a5b1eb018d6c0b09258bc080c463f089589bbc5ed1d4c3e8: Status 404 returned error can't find the container with id 5ca1b63c41bb3985a5b1eb018d6c0b09258bc080c463f089589bbc5ed1d4c3e8 Oct 31 12:36:09.230187 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:09.230170 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ckmml" Oct 31 12:36:09.406374 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:09.406346 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716b4pm"] Oct 31 12:36:09.409569 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:36:09.409545 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4f781b2_7a08_4e5b_b844_40f001caf7da.slice/crio-e33b9dba7407606aa0ce9e5f2b022562badf8f74fe8487ce58f8ffdce31ef9c9 WatchSource:0}: Error finding container e33b9dba7407606aa0ce9e5f2b022562badf8f74fe8487ce58f8ffdce31ef9c9: Status 404 returned error can't find the container with id e33b9dba7407606aa0ce9e5f2b022562badf8f74fe8487ce58f8ffdce31ef9c9 Oct 31 12:36:09.506113 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:09.505960 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-ckmml"] Oct 31 12:36:09.508403 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:36:09.508382 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8ca5a81_e73c_48e0_b4d6_bc6469003a4d.slice/crio-6e7643cb30a0d9bcad833248a1d73509ef4bb0fef21072d8a62eda328a2fd724 WatchSource:0}: Error finding container 6e7643cb30a0d9bcad833248a1d73509ef4bb0fef21072d8a62eda328a2fd724: Status 404 returned error can't find the container with id 6e7643cb30a0d9bcad833248a1d73509ef4bb0fef21072d8a62eda328a2fd724 Oct 31 12:36:10.181124 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:10.181086 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ckmml" event={"ID":"a8ca5a81-e73c-48e0-b4d6-bc6469003a4d","Type":"ContainerStarted","Data":"6e7643cb30a0d9bcad833248a1d73509ef4bb0fef21072d8a62eda328a2fd724"} Oct 31 12:36:10.182465 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:10.182430 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716b4pm" event={"ID":"b4f781b2-7a08-4e5b-b844-40f001caf7da","Type":"ContainerStarted","Data":"e33b9dba7407606aa0ce9e5f2b022562badf8f74fe8487ce58f8ffdce31ef9c9"} Oct 31 12:36:10.183771 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:10.183738 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxt6r" event={"ID":"98f41137-2279-4b64-a81f-58adacbc6c0e","Type":"ContainerStarted","Data":"5ca1b63c41bb3985a5b1eb018d6c0b09258bc080c463f089589bbc5ed1d4c3e8"} Oct 31 12:36:13.949476 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:13.949443 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:36:13.949776 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:13.949510 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:36:14.201195 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:14.201096 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ckmml" event={"ID":"a8ca5a81-e73c-48e0-b4d6-bc6469003a4d","Type":"ContainerStarted","Data":"560ad6c17f696d545828691dc26220219f943eeb05f5116cc63be9da58e921e0"} Oct 31 12:36:14.201341 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:14.201216 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ckmml" Oct 31 12:36:14.227445 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:14.227379 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ckmml" podStartSLOduration=1.78897032 podStartE2EDuration="6.22736182s" totalImagesPullingTime="4.4383915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:36:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:36:14.226702575 +0000 UTC m=+224.934288873" watchObservedRunningTime="2026-10-31 12:36:14.22736182 +0000 UTC m=+224.934948117" Oct 31 12:36:15.206428 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:15.206394 5693 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-ckmml container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 12:36:15.206907 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:15.206452 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ckmml" podUID="a8ca5a81-e73c-48e0-b4d6-bc6469003a4d" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 12:36:18.206143 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:18.206115 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ckmml" Oct 31 12:36:18.695530 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:36:18.695488 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:36:20.225042 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:20.225003 5693 generic.go:356] "Generic (PLEG): container finished" podID="98f41137-2279-4b64-a81f-58adacbc6c0e" containerID="723399d86ffcd1ffa8c8027bf85887750bb4b08449766cf1f6c640ed6d1f7c7a" exitCode=0 Oct 31 12:36:20.225685 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:20.225096 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxt6r" event={"ID":"98f41137-2279-4b64-a81f-58adacbc6c0e","Type":"ContainerDied","Data":"723399d86ffcd1ffa8c8027bf85887750bb4b08449766cf1f6c640ed6d1f7c7a"} Oct 31 12:36:21.230088 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:21.230050 5693 generic.go:356] "Generic (PLEG): container finished" podID="b4f781b2-7a08-4e5b-b844-40f001caf7da" containerID="249e399e995b6f9b7cf4c077c4a732e3d4d98f1756170e761aa0c1dfdcd83fc8" exitCode=0 Oct 31 12:36:21.230471 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:21.230138 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716b4pm" event={"ID":"b4f781b2-7a08-4e5b-b844-40f001caf7da","Type":"ContainerDied","Data":"249e399e995b6f9b7cf4c077c4a732e3d4d98f1756170e761aa0c1dfdcd83fc8"} Oct 31 12:36:22.236382 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:22.236344 5693 generic.go:356] "Generic (PLEG): container finished" podID="b4f781b2-7a08-4e5b-b844-40f001caf7da" containerID="9bd3eba972f1395f9441f719650b34e7d941c5189dbf3d937ed4b40a9f173d63" exitCode=0 Oct 31 12:36:22.236884 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:22.236412 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716b4pm" event={"ID":"b4f781b2-7a08-4e5b-b844-40f001caf7da","Type":"ContainerDied","Data":"9bd3eba972f1395f9441f719650b34e7d941c5189dbf3d937ed4b40a9f173d63"} Oct 31 12:36:23.241950 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:23.241914 5693 generic.go:356] "Generic (PLEG): container finished" podID="98f41137-2279-4b64-a81f-58adacbc6c0e" containerID="1e36ac614eedde025195f984fef1c9f213e26b080e9986e17e04adc0b34b537e" exitCode=0 Oct 31 12:36:23.242390 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:23.241975 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxt6r" event={"ID":"98f41137-2279-4b64-a81f-58adacbc6c0e","Type":"ContainerDied","Data":"1e36ac614eedde025195f984fef1c9f213e26b080e9986e17e04adc0b34b537e"} Oct 31 12:36:33.284018 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:33.283973 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxt6r" event={"ID":"98f41137-2279-4b64-a81f-58adacbc6c0e","Type":"ContainerStarted","Data":"afc7772c8bcfc63bdf2cc5e1a428cc265e90ecd27dc5009125e02218a8f3456e"} Oct 31 12:36:33.305165 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:33.305076 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxt6r" podStartSLOduration=2.705709762 podStartE2EDuration="25.305057856s" totalImagesPullingTime="22.599348094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:36:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:36:33.304360456 +0000 UTC m=+244.011946752" watchObservedRunningTime="2026-10-31 12:36:33.305057856 +0000 UTC m=+244.012644151" Oct 31 12:36:34.289503 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:34.289459 5693 generic.go:356] "Generic (PLEG): container finished" podID="98f41137-2279-4b64-a81f-58adacbc6c0e" containerID="afc7772c8bcfc63bdf2cc5e1a428cc265e90ecd27dc5009125e02218a8f3456e" exitCode=0 Oct 31 12:36:34.289937 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:34.289545 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxt6r" event={"ID":"98f41137-2279-4b64-a81f-58adacbc6c0e","Type":"ContainerDied","Data":"afc7772c8bcfc63bdf2cc5e1a428cc265e90ecd27dc5009125e02218a8f3456e"} Oct 31 12:36:34.291549 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:34.291531 5693 generic.go:356] "Generic (PLEG): container finished" podID="b4f781b2-7a08-4e5b-b844-40f001caf7da" containerID="b7f26e4b796349f56829201d6e9a05fe563913cd326306ecddf71ee82eba4f23" exitCode=0 Oct 31 12:36:34.291650 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:34.291568 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716b4pm" event={"ID":"b4f781b2-7a08-4e5b-b844-40f001caf7da","Type":"ContainerDied","Data":"b7f26e4b796349f56829201d6e9a05fe563913cd326306ecddf71ee82eba4f23"} Oct 31 12:36:35.463906 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:35.463876 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716b4pm" Oct 31 12:36:35.484850 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:35.484813 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxt6r" Oct 31 12:36:35.528585 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:35.528557 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4f781b2-7a08-4e5b-b844-40f001caf7da-kube-api-access-tdxx4\" (UniqueName: \"kubernetes.io/projected/b4f781b2-7a08-4e5b-b844-40f001caf7da-kube-api-access-tdxx4\") pod \"b4f781b2-7a08-4e5b-b844-40f001caf7da\" (UID: \"b4f781b2-7a08-4e5b-b844-40f001caf7da\") " Oct 31 12:36:35.528768 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:35.528607 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4f781b2-7a08-4e5b-b844-40f001caf7da-util\" (UniqueName: \"kubernetes.io/empty-dir/b4f781b2-7a08-4e5b-b844-40f001caf7da-util\") pod \"b4f781b2-7a08-4e5b-b844-40f001caf7da\" (UID: \"b4f781b2-7a08-4e5b-b844-40f001caf7da\") " Oct 31 12:36:35.528768 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:35.528711 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4f781b2-7a08-4e5b-b844-40f001caf7da-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4f781b2-7a08-4e5b-b844-40f001caf7da-bundle\") pod \"b4f781b2-7a08-4e5b-b844-40f001caf7da\" (UID: \"b4f781b2-7a08-4e5b-b844-40f001caf7da\") " Oct 31 12:36:35.529288 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:35.529262 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f781b2-7a08-4e5b-b844-40f001caf7da-bundle" pod "b4f781b2-7a08-4e5b-b844-40f001caf7da" (UID: "b4f781b2-7a08-4e5b-b844-40f001caf7da"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:36:35.531078 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:35.531055 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4f781b2-7a08-4e5b-b844-40f001caf7da-kube-api-access-tdxx4" pod "b4f781b2-7a08-4e5b-b844-40f001caf7da" (UID: "b4f781b2-7a08-4e5b-b844-40f001caf7da"). InnerVolumeSpecName "kube-api-access-tdxx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:36:35.533294 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:35.533273 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f781b2-7a08-4e5b-b844-40f001caf7da-util" pod "b4f781b2-7a08-4e5b-b844-40f001caf7da" (UID: "b4f781b2-7a08-4e5b-b844-40f001caf7da"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:36:35.629183 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:35.629107 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98f41137-2279-4b64-a81f-58adacbc6c0e-kube-api-access-jcwlf\" (UniqueName: \"kubernetes.io/projected/98f41137-2279-4b64-a81f-58adacbc6c0e-kube-api-access-jcwlf\") pod \"98f41137-2279-4b64-a81f-58adacbc6c0e\" (UID: \"98f41137-2279-4b64-a81f-58adacbc6c0e\") " Oct 31 12:36:35.629183 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:35.629151 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98f41137-2279-4b64-a81f-58adacbc6c0e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/98f41137-2279-4b64-a81f-58adacbc6c0e-bundle\") pod \"98f41137-2279-4b64-a81f-58adacbc6c0e\" (UID: \"98f41137-2279-4b64-a81f-58adacbc6c0e\") " Oct 31 12:36:35.629364 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:35.629208 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98f41137-2279-4b64-a81f-58adacbc6c0e-util\" (UniqueName: \"kubernetes.io/empty-dir/98f41137-2279-4b64-a81f-58adacbc6c0e-util\") pod \"98f41137-2279-4b64-a81f-58adacbc6c0e\" (UID: \"98f41137-2279-4b64-a81f-58adacbc6c0e\") " Oct 31 12:36:35.629410 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:35.629392 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdxx4\" (UniqueName: \"kubernetes.io/projected/b4f781b2-7a08-4e5b-b844-40f001caf7da-kube-api-access-tdxx4\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:36:35.629461 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:35.629417 5693 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4f781b2-7a08-4e5b-b844-40f001caf7da-util\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:36:35.629461 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:35.629431 5693 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4f781b2-7a08-4e5b-b844-40f001caf7da-bundle\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:36:35.629587 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:35.629565 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98f41137-2279-4b64-a81f-58adacbc6c0e-bundle" pod "98f41137-2279-4b64-a81f-58adacbc6c0e" (UID: "98f41137-2279-4b64-a81f-58adacbc6c0e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:36:35.631364 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:35.631341 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98f41137-2279-4b64-a81f-58adacbc6c0e-kube-api-access-jcwlf" pod "98f41137-2279-4b64-a81f-58adacbc6c0e" (UID: "98f41137-2279-4b64-a81f-58adacbc6c0e"). InnerVolumeSpecName "kube-api-access-jcwlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:36:35.634175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:35.634155 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98f41137-2279-4b64-a81f-58adacbc6c0e-util" pod "98f41137-2279-4b64-a81f-58adacbc6c0e" (UID: "98f41137-2279-4b64-a81f-58adacbc6c0e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:36:35.730552 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:35.730520 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcwlf\" (UniqueName: \"kubernetes.io/projected/98f41137-2279-4b64-a81f-58adacbc6c0e-kube-api-access-jcwlf\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:36:35.730552 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:35.730546 5693 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98f41137-2279-4b64-a81f-58adacbc6c0e-bundle\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:36:35.730749 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:35.730570 5693 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98f41137-2279-4b64-a81f-58adacbc6c0e-util\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:36:36.300203 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:36.300170 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxt6r" Oct 31 12:36:36.300203 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:36.300186 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxt6r" event={"ID":"98f41137-2279-4b64-a81f-58adacbc6c0e","Type":"ContainerDied","Data":"5ca1b63c41bb3985a5b1eb018d6c0b09258bc080c463f089589bbc5ed1d4c3e8"} Oct 31 12:36:36.300560 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:36.300238 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ca1b63c41bb3985a5b1eb018d6c0b09258bc080c463f089589bbc5ed1d4c3e8" Oct 31 12:36:36.301991 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:36.301968 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716b4pm" event={"ID":"b4f781b2-7a08-4e5b-b844-40f001caf7da","Type":"ContainerDied","Data":"e33b9dba7407606aa0ce9e5f2b022562badf8f74fe8487ce58f8ffdce31ef9c9"} Oct 31 12:36:36.302134 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:36.301999 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e33b9dba7407606aa0ce9e5f2b022562badf8f74fe8487ce58f8ffdce31ef9c9" Oct 31 12:36:36.302134 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:36.302003 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716b4pm" Oct 31 12:36:43.264627 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.264592 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-vd8dj"] Oct 31 12:36:43.265048 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.264886 5693 cpu_manager.go:414] "containerMap: removing container" podUID="b4f781b2-7a08-4e5b-b844-40f001caf7da" containerName="util" Oct 31 12:36:43.265048 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.264899 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4f781b2-7a08-4e5b-b844-40f001caf7da" containerName="util" Oct 31 12:36:43.265048 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.264909 5693 cpu_manager.go:414] "containerMap: removing container" podUID="b4f781b2-7a08-4e5b-b844-40f001caf7da" containerName="pull" Oct 31 12:36:43.265048 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.264915 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4f781b2-7a08-4e5b-b844-40f001caf7da" containerName="pull" Oct 31 12:36:43.265048 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.264923 5693 cpu_manager.go:414] "containerMap: removing container" podUID="b4f781b2-7a08-4e5b-b844-40f001caf7da" containerName="extract" Oct 31 12:36:43.265048 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.264929 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4f781b2-7a08-4e5b-b844-40f001caf7da" containerName="extract" Oct 31 12:36:43.265048 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.264937 5693 cpu_manager.go:414] "containerMap: removing container" podUID="98f41137-2279-4b64-a81f-58adacbc6c0e" containerName="pull" Oct 31 12:36:43.265048 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.264941 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98f41137-2279-4b64-a81f-58adacbc6c0e" containerName="pull" Oct 31 12:36:43.265048 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.264949 5693 cpu_manager.go:414] "containerMap: removing container" podUID="98f41137-2279-4b64-a81f-58adacbc6c0e" containerName="util" Oct 31 12:36:43.265048 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.264955 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98f41137-2279-4b64-a81f-58adacbc6c0e" containerName="util" Oct 31 12:36:43.265048 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.264962 5693 cpu_manager.go:414] "containerMap: removing container" podUID="98f41137-2279-4b64-a81f-58adacbc6c0e" containerName="extract" Oct 31 12:36:43.265048 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.264966 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98f41137-2279-4b64-a81f-58adacbc6c0e" containerName="extract" Oct 31 12:36:43.265048 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.265015 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4f781b2-7a08-4e5b-b844-40f001caf7da" containerName="extract" Oct 31 12:36:43.265048 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.265022 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="98f41137-2279-4b64-a81f-58adacbc6c0e" containerName="extract" Oct 31 12:36:43.266749 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.266734 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-vd8dj" Oct 31 12:36:43.291999 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.291977 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 31 12:36:43.291999 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.291985 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 12:36:43.292739 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.292724 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-96llj\"" Oct 31 12:36:43.292781 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.292765 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 12:36:43.298579 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.298468 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 12:36:43.299923 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.299875 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 12:36:43.390546 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.390515 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3335d434-c4ef-4f47-90d2-d0cc63f3204c-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c44b469d4-vd8dj\" (UID: \"3335d434-c4ef-4f47-90d2-d0cc63f3204c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-vd8dj" Oct 31 12:36:43.390546 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.390550 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3335d434-c4ef-4f47-90d2-d0cc63f3204c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c44b469d4-vd8dj\" (UID: \"3335d434-c4ef-4f47-90d2-d0cc63f3204c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-vd8dj" Oct 31 12:36:43.390766 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.390628 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3335d434-c4ef-4f47-90d2-d0cc63f3204c-manifests\") pod \"opendatahub-operator-controller-manager-5c44b469d4-vd8dj\" (UID: \"3335d434-c4ef-4f47-90d2-d0cc63f3204c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-vd8dj" Oct 31 12:36:43.390766 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.390673 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3335d434-c4ef-4f47-90d2-d0cc63f3204c-tmp\") pod \"opendatahub-operator-controller-manager-5c44b469d4-vd8dj\" (UID: \"3335d434-c4ef-4f47-90d2-d0cc63f3204c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-vd8dj" Oct 31 12:36:43.390766 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.390688 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7fpv\" (UniqueName: \"kubernetes.io/projected/3335d434-c4ef-4f47-90d2-d0cc63f3204c-kube-api-access-v7fpv\") pod \"opendatahub-operator-controller-manager-5c44b469d4-vd8dj\" (UID: \"3335d434-c4ef-4f47-90d2-d0cc63f3204c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-vd8dj" Oct 31 12:36:43.390766 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.390714 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3335d434-c4ef-4f47-90d2-d0cc63f3204c-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c44b469d4-vd8dj\" (UID: \"3335d434-c4ef-4f47-90d2-d0cc63f3204c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-vd8dj" Oct 31 12:36:43.491948 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.491906 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3335d434-c4ef-4f47-90d2-d0cc63f3204c-manifests\") pod \"opendatahub-operator-controller-manager-5c44b469d4-vd8dj\" (UID: \"3335d434-c4ef-4f47-90d2-d0cc63f3204c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-vd8dj" Oct 31 12:36:43.492129 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.491963 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3335d434-c4ef-4f47-90d2-d0cc63f3204c-tmp\") pod \"opendatahub-operator-controller-manager-5c44b469d4-vd8dj\" (UID: \"3335d434-c4ef-4f47-90d2-d0cc63f3204c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-vd8dj" Oct 31 12:36:43.492129 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.492014 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7fpv\" (UniqueName: \"kubernetes.io/projected/3335d434-c4ef-4f47-90d2-d0cc63f3204c-kube-api-access-v7fpv\") pod \"opendatahub-operator-controller-manager-5c44b469d4-vd8dj\" (UID: \"3335d434-c4ef-4f47-90d2-d0cc63f3204c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-vd8dj" Oct 31 12:36:43.492129 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.492054 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3335d434-c4ef-4f47-90d2-d0cc63f3204c-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c44b469d4-vd8dj\" (UID: \"3335d434-c4ef-4f47-90d2-d0cc63f3204c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-vd8dj" Oct 31 12:36:43.492129 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.492103 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3335d434-c4ef-4f47-90d2-d0cc63f3204c-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c44b469d4-vd8dj\" (UID: \"3335d434-c4ef-4f47-90d2-d0cc63f3204c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-vd8dj" Oct 31 12:36:43.492330 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.492136 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3335d434-c4ef-4f47-90d2-d0cc63f3204c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c44b469d4-vd8dj\" (UID: \"3335d434-c4ef-4f47-90d2-d0cc63f3204c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-vd8dj" Oct 31 12:36:43.492423 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.492402 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3335d434-c4ef-4f47-90d2-d0cc63f3204c-manifests\") pod \"opendatahub-operator-controller-manager-5c44b469d4-vd8dj\" (UID: \"3335d434-c4ef-4f47-90d2-d0cc63f3204c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-vd8dj" Oct 31 12:36:43.494591 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.494564 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3335d434-c4ef-4f47-90d2-d0cc63f3204c-tmp\") pod \"opendatahub-operator-controller-manager-5c44b469d4-vd8dj\" (UID: \"3335d434-c4ef-4f47-90d2-d0cc63f3204c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-vd8dj" Oct 31 12:36:43.494797 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.494777 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3335d434-c4ef-4f47-90d2-d0cc63f3204c-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c44b469d4-vd8dj\" (UID: \"3335d434-c4ef-4f47-90d2-d0cc63f3204c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-vd8dj" Oct 31 12:36:43.494863 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.494803 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3335d434-c4ef-4f47-90d2-d0cc63f3204c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c44b469d4-vd8dj\" (UID: \"3335d434-c4ef-4f47-90d2-d0cc63f3204c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-vd8dj" Oct 31 12:36:43.494979 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.494964 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3335d434-c4ef-4f47-90d2-d0cc63f3204c-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c44b469d4-vd8dj\" (UID: \"3335d434-c4ef-4f47-90d2-d0cc63f3204c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-vd8dj" Oct 31 12:36:43.509979 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.509953 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7fpv\" (UniqueName: \"kubernetes.io/projected/3335d434-c4ef-4f47-90d2-d0cc63f3204c-kube-api-access-v7fpv\") pod \"opendatahub-operator-controller-manager-5c44b469d4-vd8dj\" (UID: \"3335d434-c4ef-4f47-90d2-d0cc63f3204c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-vd8dj" Oct 31 12:36:43.579135 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.579048 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-vd8dj" Oct 31 12:36:43.598794 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.598760 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-vd8dj"] Oct 31 12:36:43.886320 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:43.886239 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-vd8dj"] Oct 31 12:36:43.890613 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:36:43.890590 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3335d434_c4ef_4f47_90d2_d0cc63f3204c.slice/crio-1fe2b10de226de053a2424966b392c1a51c10ac55b6f02625cc3faa1670a87cf WatchSource:0}: Error finding container 1fe2b10de226de053a2424966b392c1a51c10ac55b6f02625cc3faa1670a87cf: Status 404 returned error can't find the container with id 1fe2b10de226de053a2424966b392c1a51c10ac55b6f02625cc3faa1670a87cf Oct 31 12:36:44.328972 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:44.328935 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-vd8dj" event={"ID":"3335d434-c4ef-4f47-90d2-d0cc63f3204c","Type":"ContainerStarted","Data":"1fe2b10de226de053a2424966b392c1a51c10ac55b6f02625cc3faa1670a87cf"} Oct 31 12:36:47.346387 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:47.346304 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-vd8dj" event={"ID":"3335d434-c4ef-4f47-90d2-d0cc63f3204c","Type":"ContainerStarted","Data":"9daf833306abd5815769bf82fc550fafcefbef4bea0afce3410e97d06908c655"} Oct 31 12:36:48.351056 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:48.351021 5693 generic.go:356] "Generic (PLEG): container finished" podID="3335d434-c4ef-4f47-90d2-d0cc63f3204c" containerID="9daf833306abd5815769bf82fc550fafcefbef4bea0afce3410e97d06908c655" exitCode=0 Oct 31 12:36:48.351545 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:48.351089 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-vd8dj" event={"ID":"3335d434-c4ef-4f47-90d2-d0cc63f3204c","Type":"ContainerDied","Data":"9daf833306abd5815769bf82fc550fafcefbef4bea0afce3410e97d06908c655"} Oct 31 12:36:48.685358 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:36:48.685321 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:36:49.357723 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:49.357687 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-vd8dj" event={"ID":"3335d434-c4ef-4f47-90d2-d0cc63f3204c","Type":"ContainerStarted","Data":"7496345ea80caf9d9378d161ad7904af00f66b2c646ec52ff37db65bdd89fe00"} Oct 31 12:36:49.358118 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:49.357793 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-vd8dj" Oct 31 12:36:49.409901 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:36:49.409816 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-vd8dj" podStartSLOduration=2.331907636 podStartE2EDuration="6.409797348s" totalImagesPullingTime="3.585622188s" totalInitContainerRuntime="492.267524ms" isStatefulPod=false podCreationTimestamp="2026-10-31 12:36:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:36:49.407807477 +0000 UTC m=+260.115393771" watchObservedRunningTime="2026-10-31 12:36:49.409797348 +0000 UTC m=+260.117383646" Oct 31 12:37:00.363323 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:00.363291 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-vd8dj" Oct 31 12:37:14.551768 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:14.551679 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-dj5tf"] Oct 31 12:37:14.554696 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:14.554679 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-dj5tf" Oct 31 12:37:14.563759 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:14.563735 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 31 12:37:14.564224 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:14.564204 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 31 12:37:14.564928 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:14.564907 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-p585q\"" Oct 31 12:37:14.565232 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:14.565213 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-dj5tf"] Oct 31 12:37:14.623082 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:14.623046 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/068dc7ce-e22f-4d35-b14b-189cdb118723-bound-sa-token\") pod \"cert-manager-597b95cb84-dj5tf\" (UID: \"068dc7ce-e22f-4d35-b14b-189cdb118723\") " pod="cert-manager/cert-manager-597b95cb84-dj5tf" Oct 31 12:37:14.623082 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:14.623081 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnp2x\" (UniqueName: \"kubernetes.io/projected/068dc7ce-e22f-4d35-b14b-189cdb118723-kube-api-access-qnp2x\") pod \"cert-manager-597b95cb84-dj5tf\" (UID: \"068dc7ce-e22f-4d35-b14b-189cdb118723\") " pod="cert-manager/cert-manager-597b95cb84-dj5tf" Oct 31 12:37:14.724129 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:14.724096 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/068dc7ce-e22f-4d35-b14b-189cdb118723-bound-sa-token\") pod \"cert-manager-597b95cb84-dj5tf\" (UID: \"068dc7ce-e22f-4d35-b14b-189cdb118723\") " pod="cert-manager/cert-manager-597b95cb84-dj5tf" Oct 31 12:37:14.724294 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:14.724141 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnp2x\" (UniqueName: \"kubernetes.io/projected/068dc7ce-e22f-4d35-b14b-189cdb118723-kube-api-access-qnp2x\") pod \"cert-manager-597b95cb84-dj5tf\" (UID: \"068dc7ce-e22f-4d35-b14b-189cdb118723\") " pod="cert-manager/cert-manager-597b95cb84-dj5tf" Oct 31 12:37:14.739100 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:14.739061 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/068dc7ce-e22f-4d35-b14b-189cdb118723-bound-sa-token\") pod \"cert-manager-597b95cb84-dj5tf\" (UID: \"068dc7ce-e22f-4d35-b14b-189cdb118723\") " pod="cert-manager/cert-manager-597b95cb84-dj5tf" Oct 31 12:37:14.740189 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:14.740164 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnp2x\" (UniqueName: \"kubernetes.io/projected/068dc7ce-e22f-4d35-b14b-189cdb118723-kube-api-access-qnp2x\") pod \"cert-manager-597b95cb84-dj5tf\" (UID: \"068dc7ce-e22f-4d35-b14b-189cdb118723\") " pod="cert-manager/cert-manager-597b95cb84-dj5tf" Oct 31 12:37:14.874112 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:14.874032 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-dj5tf" Oct 31 12:37:15.154454 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:15.154425 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-dj5tf"] Oct 31 12:37:15.158315 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:37:15.158284 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod068dc7ce_e22f_4d35_b14b_189cdb118723.slice/crio-f4ec45ce684f167541295293298a5d67c482cab5b0fe289c11fe95be310c65dc WatchSource:0}: Error finding container f4ec45ce684f167541295293298a5d67c482cab5b0fe289c11fe95be310c65dc: Status 404 returned error can't find the container with id f4ec45ce684f167541295293298a5d67c482cab5b0fe289c11fe95be310c65dc Oct 31 12:37:15.450756 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:15.450672 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-dj5tf" event={"ID":"068dc7ce-e22f-4d35-b14b-189cdb118723","Type":"ContainerStarted","Data":"f4ec45ce684f167541295293298a5d67c482cab5b0fe289c11fe95be310c65dc"} Oct 31 12:37:18.463959 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:18.463925 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-dj5tf" event={"ID":"068dc7ce-e22f-4d35-b14b-189cdb118723","Type":"ContainerStarted","Data":"78abd478585605c6120b121c7bf95a0bbc12e682be3a855f6383d577032c6b0b"} Oct 31 12:37:18.481810 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:18.481756 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-dj5tf" podStartSLOduration=1.272494763 podStartE2EDuration="4.481741953s" totalImagesPullingTime="3.20924719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:18.48104867 +0000 UTC m=+289.188634964" watchObservedRunningTime="2026-10-31 12:37:18.481741953 +0000 UTC m=+289.189328239" Oct 31 12:37:18.686271 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:37:18.686231 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:37:19.687635 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:19.687599 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-54cf69b775-b227x"] Oct 31 12:37:19.725838 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:19.725798 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54cf69b775-b227x"] Oct 31 12:37:19.726033 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:19.725910 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54cf69b775-b227x" Oct 31 12:37:19.739540 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:19.739515 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 31 12:37:19.739677 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:19.739588 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 31 12:37:19.739677 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:19.739617 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xqbvn\"" Oct 31 12:37:19.765975 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:19.765940 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42b4c824-4d0e-4551-869e-8f369356e2d9-tls-certs\") pod \"kube-auth-proxy-54cf69b775-b227x\" (UID: \"42b4c824-4d0e-4551-869e-8f369356e2d9\") " pod="openshift-ingress/kube-auth-proxy-54cf69b775-b227x" Oct 31 12:37:19.766139 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:19.765978 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42b4c824-4d0e-4551-869e-8f369356e2d9-tmp\") pod \"kube-auth-proxy-54cf69b775-b227x\" (UID: \"42b4c824-4d0e-4551-869e-8f369356e2d9\") " pod="openshift-ingress/kube-auth-proxy-54cf69b775-b227x" Oct 31 12:37:19.766139 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:19.766004 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj46j\" (UniqueName: \"kubernetes.io/projected/42b4c824-4d0e-4551-869e-8f369356e2d9-kube-api-access-xj46j\") pod \"kube-auth-proxy-54cf69b775-b227x\" (UID: \"42b4c824-4d0e-4551-869e-8f369356e2d9\") " pod="openshift-ingress/kube-auth-proxy-54cf69b775-b227x" Oct 31 12:37:19.868655 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:19.868613 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42b4c824-4d0e-4551-869e-8f369356e2d9-tls-certs\") pod \"kube-auth-proxy-54cf69b775-b227x\" (UID: \"42b4c824-4d0e-4551-869e-8f369356e2d9\") " pod="openshift-ingress/kube-auth-proxy-54cf69b775-b227x" Oct 31 12:37:19.868899 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:19.868702 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42b4c824-4d0e-4551-869e-8f369356e2d9-tmp\") pod \"kube-auth-proxy-54cf69b775-b227x\" (UID: \"42b4c824-4d0e-4551-869e-8f369356e2d9\") " pod="openshift-ingress/kube-auth-proxy-54cf69b775-b227x" Oct 31 12:37:19.868899 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:19.868733 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj46j\" (UniqueName: \"kubernetes.io/projected/42b4c824-4d0e-4551-869e-8f369356e2d9-kube-api-access-xj46j\") pod \"kube-auth-proxy-54cf69b775-b227x\" (UID: \"42b4c824-4d0e-4551-869e-8f369356e2d9\") " pod="openshift-ingress/kube-auth-proxy-54cf69b775-b227x" Oct 31 12:37:19.871844 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:19.871809 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42b4c824-4d0e-4551-869e-8f369356e2d9-tmp\") pod \"kube-auth-proxy-54cf69b775-b227x\" (UID: \"42b4c824-4d0e-4551-869e-8f369356e2d9\") " pod="openshift-ingress/kube-auth-proxy-54cf69b775-b227x" Oct 31 12:37:19.872351 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:19.872328 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42b4c824-4d0e-4551-869e-8f369356e2d9-tls-certs\") pod \"kube-auth-proxy-54cf69b775-b227x\" (UID: \"42b4c824-4d0e-4551-869e-8f369356e2d9\") " pod="openshift-ingress/kube-auth-proxy-54cf69b775-b227x" Oct 31 12:37:19.891265 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:19.891239 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj46j\" (UniqueName: \"kubernetes.io/projected/42b4c824-4d0e-4551-869e-8f369356e2d9-kube-api-access-xj46j\") pod \"kube-auth-proxy-54cf69b775-b227x\" (UID: \"42b4c824-4d0e-4551-869e-8f369356e2d9\") " pod="openshift-ingress/kube-auth-proxy-54cf69b775-b227x" Oct 31 12:37:20.047329 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:20.047251 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54cf69b775-b227x" Oct 31 12:37:20.364613 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:20.364538 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54cf69b775-b227x"] Oct 31 12:37:20.367815 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:37:20.367787 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b4c824_4d0e_4551_869e_8f369356e2d9.slice/crio-63f511802358940564213d375744d6d4b51138395303a430d59f03a37298b0ec WatchSource:0}: Error finding container 63f511802358940564213d375744d6d4b51138395303a430d59f03a37298b0ec: Status 404 returned error can't find the container with id 63f511802358940564213d375744d6d4b51138395303a430d59f03a37298b0ec Oct 31 12:37:20.471443 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:20.471410 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54cf69b775-b227x" event={"ID":"42b4c824-4d0e-4551-869e-8f369356e2d9","Type":"ContainerStarted","Data":"63f511802358940564213d375744d6d4b51138395303a430d59f03a37298b0ec"} Oct 31 12:37:23.487086 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:23.487053 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54cf69b775-b227x" event={"ID":"42b4c824-4d0e-4551-869e-8f369356e2d9","Type":"ContainerStarted","Data":"909329a7c25ff0cdd5829339c3f1d479edae50fa2560e6e77119e475a89ba014"} Oct 31 12:37:23.507850 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:23.507793 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-54cf69b775-b227x" podStartSLOduration=2.176921977 podStartE2EDuration="4.507780728s" totalImagesPullingTime="2.330858751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:23.506201962 +0000 UTC m=+294.213788254" watchObservedRunningTime="2026-10-31 12:37:23.507780728 +0000 UTC m=+294.215367017" Oct 31 12:37:48.685727 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:37:48.685692 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:37:49.880239 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:49.880211 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6ff787b849-f7hht"] Oct 31 12:37:49.883585 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:49.883568 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6ff787b849-f7hht" Oct 31 12:37:49.888438 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:49.888418 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-4v5r4\"" Oct 31 12:37:49.893445 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:49.893422 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6ff787b849-f7hht"] Oct 31 12:37:49.897884 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:49.897859 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 31 12:37:49.897884 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:49.897859 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 31 12:37:49.898056 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:49.897882 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 31 12:37:49.954733 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:49.954696 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/036d4bf6-0f78-46e5-8060-452d8ca7b606-cert\") pod \"jobset-controller-manager-6ff787b849-f7hht\" (UID: \"036d4bf6-0f78-46e5-8060-452d8ca7b606\") " pod="openshift-jobset-operator/jobset-controller-manager-6ff787b849-f7hht" Oct 31 12:37:49.954925 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:49.954748 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf6hq\" (UniqueName: \"kubernetes.io/projected/036d4bf6-0f78-46e5-8060-452d8ca7b606-kube-api-access-nf6hq\") pod \"jobset-controller-manager-6ff787b849-f7hht\" (UID: \"036d4bf6-0f78-46e5-8060-452d8ca7b606\") " pod="openshift-jobset-operator/jobset-controller-manager-6ff787b849-f7hht" Oct 31 12:37:49.954925 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:49.954773 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/036d4bf6-0f78-46e5-8060-452d8ca7b606-manager-config\") pod \"jobset-controller-manager-6ff787b849-f7hht\" (UID: \"036d4bf6-0f78-46e5-8060-452d8ca7b606\") " pod="openshift-jobset-operator/jobset-controller-manager-6ff787b849-f7hht" Oct 31 12:37:49.955007 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:49.954920 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/036d4bf6-0f78-46e5-8060-452d8ca7b606-metrics-certs\") pod \"jobset-controller-manager-6ff787b849-f7hht\" (UID: \"036d4bf6-0f78-46e5-8060-452d8ca7b606\") " pod="openshift-jobset-operator/jobset-controller-manager-6ff787b849-f7hht" Oct 31 12:37:49.988412 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:49.988378 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 31 12:37:49.988607 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:49.988430 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 31 12:37:50.056294 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:50.056262 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf6hq\" (UniqueName: \"kubernetes.io/projected/036d4bf6-0f78-46e5-8060-452d8ca7b606-kube-api-access-nf6hq\") pod \"jobset-controller-manager-6ff787b849-f7hht\" (UID: \"036d4bf6-0f78-46e5-8060-452d8ca7b606\") " pod="openshift-jobset-operator/jobset-controller-manager-6ff787b849-f7hht" Oct 31 12:37:50.056479 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:50.056302 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/036d4bf6-0f78-46e5-8060-452d8ca7b606-manager-config\") pod \"jobset-controller-manager-6ff787b849-f7hht\" (UID: \"036d4bf6-0f78-46e5-8060-452d8ca7b606\") " pod="openshift-jobset-operator/jobset-controller-manager-6ff787b849-f7hht" Oct 31 12:37:50.056479 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:50.056338 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/036d4bf6-0f78-46e5-8060-452d8ca7b606-metrics-certs\") pod \"jobset-controller-manager-6ff787b849-f7hht\" (UID: \"036d4bf6-0f78-46e5-8060-452d8ca7b606\") " pod="openshift-jobset-operator/jobset-controller-manager-6ff787b849-f7hht" Oct 31 12:37:50.056479 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:50.056357 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/036d4bf6-0f78-46e5-8060-452d8ca7b606-cert\") pod \"jobset-controller-manager-6ff787b849-f7hht\" (UID: \"036d4bf6-0f78-46e5-8060-452d8ca7b606\") " pod="openshift-jobset-operator/jobset-controller-manager-6ff787b849-f7hht" Oct 31 12:37:50.057074 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:50.057055 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/036d4bf6-0f78-46e5-8060-452d8ca7b606-manager-config\") pod \"jobset-controller-manager-6ff787b849-f7hht\" (UID: \"036d4bf6-0f78-46e5-8060-452d8ca7b606\") " pod="openshift-jobset-operator/jobset-controller-manager-6ff787b849-f7hht" Oct 31 12:37:50.058939 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:50.058918 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/036d4bf6-0f78-46e5-8060-452d8ca7b606-metrics-certs\") pod \"jobset-controller-manager-6ff787b849-f7hht\" (UID: \"036d4bf6-0f78-46e5-8060-452d8ca7b606\") " pod="openshift-jobset-operator/jobset-controller-manager-6ff787b849-f7hht" Oct 31 12:37:50.059023 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:50.058947 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/036d4bf6-0f78-46e5-8060-452d8ca7b606-cert\") pod \"jobset-controller-manager-6ff787b849-f7hht\" (UID: \"036d4bf6-0f78-46e5-8060-452d8ca7b606\") " pod="openshift-jobset-operator/jobset-controller-manager-6ff787b849-f7hht" Oct 31 12:37:50.068531 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:50.068511 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf6hq\" (UniqueName: \"kubernetes.io/projected/036d4bf6-0f78-46e5-8060-452d8ca7b606-kube-api-access-nf6hq\") pod \"jobset-controller-manager-6ff787b849-f7hht\" (UID: \"036d4bf6-0f78-46e5-8060-452d8ca7b606\") " pod="openshift-jobset-operator/jobset-controller-manager-6ff787b849-f7hht" Oct 31 12:37:50.195943 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:50.195916 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6ff787b849-f7hht" Oct 31 12:37:50.467370 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:50.467304 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6ff787b849-f7hht"] Oct 31 12:37:50.470253 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:37:50.470227 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod036d4bf6_0f78_46e5_8060_452d8ca7b606.slice/crio-3c39900fce07b78d3e6237baf4ef2c26d7d753df5c8ab395e7ef6fa3bf64acac WatchSource:0}: Error finding container 3c39900fce07b78d3e6237baf4ef2c26d7d753df5c8ab395e7ef6fa3bf64acac: Status 404 returned error can't find the container with id 3c39900fce07b78d3e6237baf4ef2c26d7d753df5c8ab395e7ef6fa3bf64acac Oct 31 12:37:50.472150 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:50.472131 5693 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 12:37:50.593451 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:50.593413 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6ff787b849-f7hht" event={"ID":"036d4bf6-0f78-46e5-8060-452d8ca7b606","Type":"ContainerStarted","Data":"3c39900fce07b78d3e6237baf4ef2c26d7d753df5c8ab395e7ef6fa3bf64acac"} Oct 31 12:37:54.609965 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:54.609930 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6ff787b849-f7hht" event={"ID":"036d4bf6-0f78-46e5-8060-452d8ca7b606","Type":"ContainerStarted","Data":"144a165c726be07631234db5512cb0f7bd33d88346822cbe75866b5df3a8bbcb"} Oct 31 12:37:54.610457 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:54.610036 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-6ff787b849-f7hht" Oct 31 12:37:54.629396 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:37:54.629350 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-6ff787b849-f7hht" podStartSLOduration=2.335569875 podStartE2EDuration="5.629337734s" totalImagesPullingTime="3.293767859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:54.627888615 +0000 UTC m=+325.335474911" watchObservedRunningTime="2026-10-31 12:37:54.629337734 +0000 UTC m=+325.336924028" Oct 31 12:38:05.619366 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:38:05.619337 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-6ff787b849-f7hht" Oct 31 12:38:18.685407 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:38:18.685372 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:38:23.368480 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:38:23.368442 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-ckmml"] Oct 31 12:38:23.369127 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:38:23.368748 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ckmml" podUID="a8ca5a81-e73c-48e0-b4d6-bc6469003a4d" containerName="discovery" containerID="cri-o://560ad6c17f696d545828691dc26220219f943eeb05f5116cc63be9da58e921e0" gracePeriod=30 Oct 31 12:38:23.604915 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:38:23.604890 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ckmml" Oct 31 12:38:23.720974 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:38:23.720939 5693 generic.go:356] "Generic (PLEG): container finished" podID="a8ca5a81-e73c-48e0-b4d6-bc6469003a4d" containerID="560ad6c17f696d545828691dc26220219f943eeb05f5116cc63be9da58e921e0" exitCode=0 Oct 31 12:38:23.721156 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:38:23.720985 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ckmml" Oct 31 12:38:23.721156 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:38:23.721016 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ckmml" event={"ID":"a8ca5a81-e73c-48e0-b4d6-bc6469003a4d","Type":"ContainerDied","Data":"560ad6c17f696d545828691dc26220219f943eeb05f5116cc63be9da58e921e0"} Oct 31 12:38:23.721156 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:38:23.721064 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ckmml" event={"ID":"a8ca5a81-e73c-48e0-b4d6-bc6469003a4d","Type":"ContainerDied","Data":"6e7643cb30a0d9bcad833248a1d73509ef4bb0fef21072d8a62eda328a2fd724"} Oct 31 12:38:23.721156 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:38:23.721083 5693 scope.go:122] "RemoveContainer" containerID="560ad6c17f696d545828691dc26220219f943eeb05f5116cc63be9da58e921e0" Oct 31 12:38:23.724290 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:38:23.724276 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-cacerts\" (UniqueName: \"kubernetes.io/secret/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-cacerts\") pod \"a8ca5a81-e73c-48e0-b4d6-bc6469003a4d\" (UID: \"a8ca5a81-e73c-48e0-b4d6-bc6469003a4d\") " Oct 31 12:38:23.724413 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:38:23.724325 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-istio-token\" (UniqueName: \"kubernetes.io/projected/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-istio-token\") pod \"a8ca5a81-e73c-48e0-b4d6-bc6469003a4d\" (UID: \"a8ca5a81-e73c-48e0-b4d6-bc6469003a4d\") " Oct 31 12:38:23.724413 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:38:23.724348 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-local-certs\") pod \"a8ca5a81-e73c-48e0-b4d6-bc6469003a4d\" (UID: \"a8ca5a81-e73c-48e0-b4d6-bc6469003a4d\") " Oct 31 12:38:23.724628 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:38:23.724460 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-istio-kubeconfig\") pod \"a8ca5a81-e73c-48e0-b4d6-bc6469003a4d\" (UID: \"a8ca5a81-e73c-48e0-b4d6-bc6469003a4d\") " Oct 31 12:38:23.724628 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:38:23.724540 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-kube-api-access-zcvpc\" (UniqueName: \"kubernetes.io/projected/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-kube-api-access-zcvpc\") pod \"a8ca5a81-e73c-48e0-b4d6-bc6469003a4d\" (UID: \"a8ca5a81-e73c-48e0-b4d6-bc6469003a4d\") " Oct 31 12:38:23.724628 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:38:23.724595 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-istio-csr-ca-configmap\") pod \"a8ca5a81-e73c-48e0-b4d6-bc6469003a4d\" (UID: \"a8ca5a81-e73c-48e0-b4d6-bc6469003a4d\") " Oct 31 12:38:23.724819 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:38:23.724664 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-istio-csr-dns-cert\") pod \"a8ca5a81-e73c-48e0-b4d6-bc6469003a4d\" (UID: \"a8ca5a81-e73c-48e0-b4d6-bc6469003a4d\") " Oct 31 12:38:23.725692 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:38:23.725619 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-istio-csr-ca-configmap" pod "a8ca5a81-e73c-48e0-b4d6-bc6469003a4d" (UID: "a8ca5a81-e73c-48e0-b4d6-bc6469003a4d"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:38:23.727015 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:38:23.726986 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-cacerts" pod "a8ca5a81-e73c-48e0-b4d6-bc6469003a4d" (UID: "a8ca5a81-e73c-48e0-b4d6-bc6469003a4d"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:38:23.727132 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:38:23.727110 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-kube-api-access-zcvpc" pod "a8ca5a81-e73c-48e0-b4d6-bc6469003a4d" (UID: "a8ca5a81-e73c-48e0-b4d6-bc6469003a4d"). InnerVolumeSpecName "kube-api-access-zcvpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:38:23.727266 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:38:23.727242 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-istio-token" pod "a8ca5a81-e73c-48e0-b4d6-bc6469003a4d" (UID: "a8ca5a81-e73c-48e0-b4d6-bc6469003a4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:38:23.727424 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:38:23.727406 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-local-certs" pod "a8ca5a81-e73c-48e0-b4d6-bc6469003a4d" (UID: "a8ca5a81-e73c-48e0-b4d6-bc6469003a4d"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:38:23.727482 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:38:23.727463 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-istio-csr-dns-cert" pod "a8ca5a81-e73c-48e0-b4d6-bc6469003a4d" (UID: "a8ca5a81-e73c-48e0-b4d6-bc6469003a4d"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:38:23.727518 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:38:23.727489 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-istio-kubeconfig" pod "a8ca5a81-e73c-48e0-b4d6-bc6469003a4d" (UID: "a8ca5a81-e73c-48e0-b4d6-bc6469003a4d"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:38:23.741800 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:38:23.741778 5693 scope.go:122] "RemoveContainer" containerID="560ad6c17f696d545828691dc26220219f943eeb05f5116cc63be9da58e921e0" Oct 31 12:38:23.742090 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:38:23.742069 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"560ad6c17f696d545828691dc26220219f943eeb05f5116cc63be9da58e921e0\": container with ID starting with 560ad6c17f696d545828691dc26220219f943eeb05f5116cc63be9da58e921e0 not found: ID does not exist" containerID="560ad6c17f696d545828691dc26220219f943eeb05f5116cc63be9da58e921e0" Oct 31 12:38:23.742144 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:38:23.742102 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"560ad6c17f696d545828691dc26220219f943eeb05f5116cc63be9da58e921e0"} err="failed to get container status \"560ad6c17f696d545828691dc26220219f943eeb05f5116cc63be9da58e921e0\": rpc error: code = NotFound desc = could not find container \"560ad6c17f696d545828691dc26220219f943eeb05f5116cc63be9da58e921e0\": container with ID starting with 560ad6c17f696d545828691dc26220219f943eeb05f5116cc63be9da58e921e0 not found: ID does not exist" Oct 31 12:38:23.826036 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:38:23.826001 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcvpc\" (UniqueName: \"kubernetes.io/projected/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-kube-api-access-zcvpc\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:38:23.826036 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:38:23.826030 5693 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-istio-csr-ca-configmap\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:38:23.826036 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:38:23.826040 5693 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-istio-csr-dns-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:38:23.826239 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:38:23.826049 5693 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-cacerts\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:38:23.826239 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:38:23.826058 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:38:23.826239 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:38:23.826066 5693 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-local-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:38:23.826239 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:38:23.826074 5693 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d-istio-kubeconfig\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:38:24.041282 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:38:24.041252 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-ckmml"] Oct 31 12:38:24.048275 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:38:24.048251 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-ckmml"] Oct 31 12:38:25.405084 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:38:25.405047 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8ca5a81-e73c-48e0-b4d6-bc6469003a4d" path="/var/lib/kubelet/pods/a8ca5a81-e73c-48e0-b4d6-bc6469003a4d/volumes" Oct 31 12:38:48.688021 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:38:48.687935 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:39:18.685375 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:39:18.685328 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:39:48.685314 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:39:48.685279 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:40:18.685347 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:40:18.685260 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:40:48.684992 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:40:48.684956 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:41:18.684969 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:41:18.684932 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:41:48.686191 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:41:48.686108 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:42:18.685350 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:42:18.685307 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:42:48.685425 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:42:48.685389 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:43:18.685754 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:43:18.685673 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:43:48.686225 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:43:48.686190 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:44:18.685831 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:44:18.685795 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:44:48.685873 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:44:48.685794 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:45:18.685363 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:45:18.685325 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:45:48.685440 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:45:48.685402 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:46:18.686191 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:46:18.686090 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:46:48.685802 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:46:48.685768 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:47:18.685616 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:47:18.685578 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:47:48.685314 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:47:48.685231 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:48:00.062344 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.062310 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jb7gv"] Oct 31 12:48:00.062860 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.062584 5693 cpu_manager.go:414] "containerMap: removing container" podUID="a8ca5a81-e73c-48e0-b4d6-bc6469003a4d" containerName="discovery" Oct 31 12:48:00.062860 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.062595 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8ca5a81-e73c-48e0-b4d6-bc6469003a4d" containerName="discovery" Oct 31 12:48:00.062860 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.062671 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8ca5a81-e73c-48e0-b4d6-bc6469003a4d" containerName="discovery" Oct 31 12:48:00.065444 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.065426 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-jb7gv" Oct 31 12:48:00.071633 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.071433 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rdq67\"" Oct 31 12:48:00.074230 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.074210 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 12:48:00.074734 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.074216 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 12:48:00.074734 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.074299 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 31 12:48:00.075846 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.075823 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jb7gv"] Oct 31 12:48:00.132885 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.132842 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/49459373-a78d-4bdb-a207-3df331b6bb18-config-file\") pod \"limitador-limitador-74dfcfb894-jb7gv\" (UID: \"49459373-a78d-4bdb-a207-3df331b6bb18\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jb7gv" Oct 31 12:48:00.133043 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.132982 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6gm7\" (UniqueName: \"kubernetes.io/projected/49459373-a78d-4bdb-a207-3df331b6bb18-kube-api-access-g6gm7\") pod \"limitador-limitador-74dfcfb894-jb7gv\" (UID: \"49459373-a78d-4bdb-a207-3df331b6bb18\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jb7gv" Oct 31 12:48:00.135320 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.135293 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-b2vpf"] Oct 31 12:48:00.143394 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.143361 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-b2vpf" Oct 31 12:48:00.155081 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.155059 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-b2vpf"] Oct 31 12:48:00.163478 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.163453 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jb7gv"] Oct 31 12:48:00.171235 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.171201 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 31 12:48:00.174256 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.174237 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 31 12:48:00.233976 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.233946 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6gm7\" (UniqueName: \"kubernetes.io/projected/49459373-a78d-4bdb-a207-3df331b6bb18-kube-api-access-g6gm7\") pod \"limitador-limitador-74dfcfb894-jb7gv\" (UID: \"49459373-a78d-4bdb-a207-3df331b6bb18\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jb7gv" Oct 31 12:48:00.234135 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.234002 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8p4n\" (UniqueName: \"kubernetes.io/projected/86e98514-c296-467f-ba33-cbcdf689b48b-kube-api-access-g8p4n\") pod \"kuadrant-console-plugin-66c4fb575d-b2vpf\" (UID: \"86e98514-c296-467f-ba33-cbcdf689b48b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-b2vpf" Oct 31 12:48:00.234135 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.234029 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/86e98514-c296-467f-ba33-cbcdf689b48b-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-b2vpf\" (UID: \"86e98514-c296-467f-ba33-cbcdf689b48b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-b2vpf" Oct 31 12:48:00.234135 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.234045 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/86e98514-c296-467f-ba33-cbcdf689b48b-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-b2vpf\" (UID: \"86e98514-c296-467f-ba33-cbcdf689b48b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-b2vpf" Oct 31 12:48:00.234135 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.234069 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/49459373-a78d-4bdb-a207-3df331b6bb18-config-file\") pod \"limitador-limitador-74dfcfb894-jb7gv\" (UID: \"49459373-a78d-4bdb-a207-3df331b6bb18\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jb7gv" Oct 31 12:48:00.234651 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.234621 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/49459373-a78d-4bdb-a207-3df331b6bb18-config-file\") pod \"limitador-limitador-74dfcfb894-jb7gv\" (UID: \"49459373-a78d-4bdb-a207-3df331b6bb18\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jb7gv" Oct 31 12:48:00.246271 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.246246 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6gm7\" (UniqueName: \"kubernetes.io/projected/49459373-a78d-4bdb-a207-3df331b6bb18-kube-api-access-g6gm7\") pod \"limitador-limitador-74dfcfb894-jb7gv\" (UID: \"49459373-a78d-4bdb-a207-3df331b6bb18\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jb7gv" Oct 31 12:48:00.335508 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.335419 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8p4n\" (UniqueName: \"kubernetes.io/projected/86e98514-c296-467f-ba33-cbcdf689b48b-kube-api-access-g8p4n\") pod \"kuadrant-console-plugin-66c4fb575d-b2vpf\" (UID: \"86e98514-c296-467f-ba33-cbcdf689b48b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-b2vpf" Oct 31 12:48:00.335508 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.335460 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/86e98514-c296-467f-ba33-cbcdf689b48b-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-b2vpf\" (UID: \"86e98514-c296-467f-ba33-cbcdf689b48b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-b2vpf" Oct 31 12:48:00.335508 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.335484 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/86e98514-c296-467f-ba33-cbcdf689b48b-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-b2vpf\" (UID: \"86e98514-c296-467f-ba33-cbcdf689b48b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-b2vpf" Oct 31 12:48:00.336299 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.336224 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/86e98514-c296-467f-ba33-cbcdf689b48b-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-b2vpf\" (UID: \"86e98514-c296-467f-ba33-cbcdf689b48b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-b2vpf" Oct 31 12:48:00.337959 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.337937 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/86e98514-c296-467f-ba33-cbcdf689b48b-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-b2vpf\" (UID: \"86e98514-c296-467f-ba33-cbcdf689b48b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-b2vpf" Oct 31 12:48:00.347792 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.347762 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8p4n\" (UniqueName: \"kubernetes.io/projected/86e98514-c296-467f-ba33-cbcdf689b48b-kube-api-access-g8p4n\") pod \"kuadrant-console-plugin-66c4fb575d-b2vpf\" (UID: \"86e98514-c296-467f-ba33-cbcdf689b48b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-b2vpf" Oct 31 12:48:00.379509 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.379484 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-jb7gv" Oct 31 12:48:00.454572 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.454545 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-b2vpf" Oct 31 12:48:00.651475 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.651447 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jb7gv"] Oct 31 12:48:00.654083 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:48:00.654053 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49459373_a78d_4bdb_a207_3df331b6bb18.slice/crio-001ae4b97d74c2489827b7f6b9b2638ec359e4aa7fd6a1b69f336be874dc7000 WatchSource:0}: Error finding container 001ae4b97d74c2489827b7f6b9b2638ec359e4aa7fd6a1b69f336be874dc7000: Status 404 returned error can't find the container with id 001ae4b97d74c2489827b7f6b9b2638ec359e4aa7fd6a1b69f336be874dc7000 Oct 31 12:48:00.655880 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.655865 5693 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 12:48:00.724776 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.724720 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-b2vpf"] Oct 31 12:48:00.727545 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:48:00.727520 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86e98514_c296_467f_ba33_cbcdf689b48b.slice/crio-5295aa82c2c6b98bd6a985c418e8d199307fb29c1f9797301485a30147b34ceb WatchSource:0}: Error finding container 5295aa82c2c6b98bd6a985c418e8d199307fb29c1f9797301485a30147b34ceb: Status 404 returned error can't find the container with id 5295aa82c2c6b98bd6a985c418e8d199307fb29c1f9797301485a30147b34ceb Oct 31 12:48:00.877357 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.877320 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-b2vpf" event={"ID":"86e98514-c296-467f-ba33-cbcdf689b48b","Type":"ContainerStarted","Data":"5295aa82c2c6b98bd6a985c418e8d199307fb29c1f9797301485a30147b34ceb"} Oct 31 12:48:00.878481 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.878456 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-jb7gv" event={"ID":"49459373-a78d-4bdb-a207-3df331b6bb18","Type":"ContainerStarted","Data":"001ae4b97d74c2489827b7f6b9b2638ec359e4aa7fd6a1b69f336be874dc7000"} Oct 31 12:48:00.892178 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.892154 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jlt5d"] Oct 31 12:48:00.896730 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.896713 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-jlt5d" Oct 31 12:48:00.902804 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.902729 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-m6rvq\"" Oct 31 12:48:00.902914 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:00.902862 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jlt5d"] Oct 31 12:48:01.029193 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:01.029156 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-64lq2"] Oct 31 12:48:01.032245 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:01.032224 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-64lq2" Oct 31 12:48:01.038171 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:01.038146 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-64lq2"] Oct 31 12:48:01.042260 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:01.042234 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn9rq\" (UniqueName: \"kubernetes.io/projected/92c41689-5020-4a34-a9e6-1f2883cce375-kube-api-access-nn9rq\") pod \"authorino-57f7f8659f-jlt5d\" (UID: \"92c41689-5020-4a34-a9e6-1f2883cce375\") " pod="kuadrant-system/authorino-57f7f8659f-jlt5d" Oct 31 12:48:01.143215 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:01.143175 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn9rq\" (UniqueName: \"kubernetes.io/projected/92c41689-5020-4a34-a9e6-1f2883cce375-kube-api-access-nn9rq\") pod \"authorino-57f7f8659f-jlt5d\" (UID: \"92c41689-5020-4a34-a9e6-1f2883cce375\") " pod="kuadrant-system/authorino-57f7f8659f-jlt5d" Oct 31 12:48:01.143705 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:01.143224 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfmbx\" (UniqueName: \"kubernetes.io/projected/a6c68f41-432b-425e-8016-76908847a2f9-kube-api-access-nfmbx\") pod \"authorino-56c6c9ddd5-64lq2\" (UID: \"a6c68f41-432b-425e-8016-76908847a2f9\") " pod="kuadrant-system/authorino-56c6c9ddd5-64lq2" Oct 31 12:48:01.155576 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:01.155527 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn9rq\" (UniqueName: \"kubernetes.io/projected/92c41689-5020-4a34-a9e6-1f2883cce375-kube-api-access-nn9rq\") pod \"authorino-57f7f8659f-jlt5d\" (UID: \"92c41689-5020-4a34-a9e6-1f2883cce375\") " pod="kuadrant-system/authorino-57f7f8659f-jlt5d" Oct 31 12:48:01.209925 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:01.209893 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-jlt5d" Oct 31 12:48:01.244531 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:01.244493 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfmbx\" (UniqueName: \"kubernetes.io/projected/a6c68f41-432b-425e-8016-76908847a2f9-kube-api-access-nfmbx\") pod \"authorino-56c6c9ddd5-64lq2\" (UID: \"a6c68f41-432b-425e-8016-76908847a2f9\") " pod="kuadrant-system/authorino-56c6c9ddd5-64lq2" Oct 31 12:48:01.259546 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:01.258807 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfmbx\" (UniqueName: \"kubernetes.io/projected/a6c68f41-432b-425e-8016-76908847a2f9-kube-api-access-nfmbx\") pod \"authorino-56c6c9ddd5-64lq2\" (UID: \"a6c68f41-432b-425e-8016-76908847a2f9\") " pod="kuadrant-system/authorino-56c6c9ddd5-64lq2" Oct 31 12:48:01.343828 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:01.343795 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-64lq2" Oct 31 12:48:01.480428 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:01.480352 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jlt5d"] Oct 31 12:48:01.483118 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:48:01.483092 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92c41689_5020_4a34_a9e6_1f2883cce375.slice/crio-d826aa5634a290fea1e3e750b710c6a4c9660feca0c97db41c3b040e785cfa67 WatchSource:0}: Error finding container d826aa5634a290fea1e3e750b710c6a4c9660feca0c97db41c3b040e785cfa67: Status 404 returned error can't find the container with id d826aa5634a290fea1e3e750b710c6a4c9660feca0c97db41c3b040e785cfa67 Oct 31 12:48:01.615206 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:01.615124 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-64lq2"] Oct 31 12:48:01.617548 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:48:01.617525 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6c68f41_432b_425e_8016_76908847a2f9.slice/crio-29a03f423dcbb6258aa193c5aae22ad442609ded58933f0a02d83581db533c1f WatchSource:0}: Error finding container 29a03f423dcbb6258aa193c5aae22ad442609ded58933f0a02d83581db533c1f: Status 404 returned error can't find the container with id 29a03f423dcbb6258aa193c5aae22ad442609ded58933f0a02d83581db533c1f Oct 31 12:48:01.885883 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:01.885788 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-jlt5d" event={"ID":"92c41689-5020-4a34-a9e6-1f2883cce375","Type":"ContainerStarted","Data":"d826aa5634a290fea1e3e750b710c6a4c9660feca0c97db41c3b040e785cfa67"} Oct 31 12:48:01.887705 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:01.887673 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-64lq2" event={"ID":"a6c68f41-432b-425e-8016-76908847a2f9","Type":"ContainerStarted","Data":"29a03f423dcbb6258aa193c5aae22ad442609ded58933f0a02d83581db533c1f"} Oct 31 12:48:04.904342 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:04.904303 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-jb7gv" event={"ID":"49459373-a78d-4bdb-a207-3df331b6bb18","Type":"ContainerStarted","Data":"3839ebec44dee4cfa731b5456aef7ed8f2659b980564c00416a4f3df6b5d9bd6"} Oct 31 12:48:04.904844 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:04.904443 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-jb7gv" Oct 31 12:48:04.923485 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:04.923426 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-jb7gv" podStartSLOduration=1.004216982 podStartE2EDuration="4.923408717s" totalImagesPullingTime="3.919191735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:48:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:48:04.92283083 +0000 UTC m=+935.630417128" watchObservedRunningTime="2026-10-31 12:48:04.923408717 +0000 UTC m=+935.630995013" Oct 31 12:48:05.911838 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:05.911790 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-64lq2" event={"ID":"a6c68f41-432b-425e-8016-76908847a2f9","Type":"ContainerStarted","Data":"d7e836a1680d75771a66023beabc5a716900f5458fb82f4dcd92187212726320"} Oct 31 12:48:05.929787 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:05.929733 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-64lq2" podStartSLOduration=0.866071001 podStartE2EDuration="4.929717388s" totalImagesPullingTime="4.063646387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:48:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:48:05.92834129 +0000 UTC m=+936.635927585" watchObservedRunningTime="2026-10-31 12:48:05.929717388 +0000 UTC m=+936.637303688" Oct 31 12:48:05.954554 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:05.954480 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jlt5d"] Oct 31 12:48:06.917675 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:06.917634 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-jlt5d" event={"ID":"92c41689-5020-4a34-a9e6-1f2883cce375","Type":"ContainerStarted","Data":"500fed1ee886de5a846a19b0a0bf8fcc13ae031a058c52a5a5de2e7c40f56d61"} Oct 31 12:48:06.917675 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:06.917668 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-jlt5d" podUID="92c41689-5020-4a34-a9e6-1f2883cce375" containerName="authorino" containerID="cri-o://500fed1ee886de5a846a19b0a0bf8fcc13ae031a058c52a5a5de2e7c40f56d61" gracePeriod=30 Oct 31 12:48:06.934717 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:06.934455 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-jlt5d" podStartSLOduration=2.421054872 podStartE2EDuration="6.934437118s" totalImagesPullingTime="4.513382246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:48:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:48:06.934351774 +0000 UTC m=+937.641938071" watchObservedRunningTime="2026-10-31 12:48:06.934437118 +0000 UTC m=+937.642023415" Oct 31 12:48:07.151985 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:07.151963 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-jlt5d" Oct 31 12:48:07.303579 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:07.303545 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92c41689-5020-4a34-a9e6-1f2883cce375-kube-api-access-nn9rq\" (UniqueName: \"kubernetes.io/projected/92c41689-5020-4a34-a9e6-1f2883cce375-kube-api-access-nn9rq\") pod \"92c41689-5020-4a34-a9e6-1f2883cce375\" (UID: \"92c41689-5020-4a34-a9e6-1f2883cce375\") " Oct 31 12:48:07.305885 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:07.305857 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92c41689-5020-4a34-a9e6-1f2883cce375-kube-api-access-nn9rq" pod "92c41689-5020-4a34-a9e6-1f2883cce375" (UID: "92c41689-5020-4a34-a9e6-1f2883cce375"). InnerVolumeSpecName "kube-api-access-nn9rq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:48:07.404696 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:07.404667 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn9rq\" (UniqueName: \"kubernetes.io/projected/92c41689-5020-4a34-a9e6-1f2883cce375-kube-api-access-nn9rq\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:48:07.923434 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:07.923388 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-b2vpf" event={"ID":"86e98514-c296-467f-ba33-cbcdf689b48b","Type":"ContainerStarted","Data":"4b40872783e0b156beab1e597ef0c7b706c8c4e48953525d88388895afbac005"} Oct 31 12:48:07.924765 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:07.924740 5693 generic.go:356] "Generic (PLEG): container finished" podID="92c41689-5020-4a34-a9e6-1f2883cce375" containerID="500fed1ee886de5a846a19b0a0bf8fcc13ae031a058c52a5a5de2e7c40f56d61" exitCode=0 Oct 31 12:48:07.924885 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:07.924774 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-jlt5d" Oct 31 12:48:07.924885 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:07.924789 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-jlt5d" event={"ID":"92c41689-5020-4a34-a9e6-1f2883cce375","Type":"ContainerDied","Data":"500fed1ee886de5a846a19b0a0bf8fcc13ae031a058c52a5a5de2e7c40f56d61"} Oct 31 12:48:07.924885 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:07.924818 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-jlt5d" event={"ID":"92c41689-5020-4a34-a9e6-1f2883cce375","Type":"ContainerDied","Data":"d826aa5634a290fea1e3e750b710c6a4c9660feca0c97db41c3b040e785cfa67"} Oct 31 12:48:07.924885 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:07.924834 5693 scope.go:122] "RemoveContainer" containerID="500fed1ee886de5a846a19b0a0bf8fcc13ae031a058c52a5a5de2e7c40f56d61" Oct 31 12:48:07.932624 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:07.932606 5693 scope.go:122] "RemoveContainer" containerID="500fed1ee886de5a846a19b0a0bf8fcc13ae031a058c52a5a5de2e7c40f56d61" Oct 31 12:48:07.932889 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:48:07.932870 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"500fed1ee886de5a846a19b0a0bf8fcc13ae031a058c52a5a5de2e7c40f56d61\": container with ID starting with 500fed1ee886de5a846a19b0a0bf8fcc13ae031a058c52a5a5de2e7c40f56d61 not found: ID does not exist" containerID="500fed1ee886de5a846a19b0a0bf8fcc13ae031a058c52a5a5de2e7c40f56d61" Oct 31 12:48:07.932949 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:07.932896 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"500fed1ee886de5a846a19b0a0bf8fcc13ae031a058c52a5a5de2e7c40f56d61"} err="failed to get container status \"500fed1ee886de5a846a19b0a0bf8fcc13ae031a058c52a5a5de2e7c40f56d61\": rpc error: code = NotFound desc = could not find container \"500fed1ee886de5a846a19b0a0bf8fcc13ae031a058c52a5a5de2e7c40f56d61\": container with ID starting with 500fed1ee886de5a846a19b0a0bf8fcc13ae031a058c52a5a5de2e7c40f56d61 not found: ID does not exist" Oct 31 12:48:07.941488 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:07.941428 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-b2vpf" podStartSLOduration=1.777400241 podStartE2EDuration="7.941416931s" totalImagesPullingTime="6.16401669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:48:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:48:07.94035756 +0000 UTC m=+938.647943856" watchObservedRunningTime="2026-10-31 12:48:07.941416931 +0000 UTC m=+938.649003225" Oct 31 12:48:07.955248 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:07.955221 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jlt5d"] Oct 31 12:48:07.961574 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:07.961552 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jlt5d"] Oct 31 12:48:09.405584 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:09.405552 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92c41689-5020-4a34-a9e6-1f2883cce375" path="/var/lib/kubelet/pods/92c41689-5020-4a34-a9e6-1f2883cce375/volumes" Oct 31 12:48:13.523764 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:13.523734 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jb7gv"] Oct 31 12:48:13.524247 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:13.523912 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-jb7gv" podUID="49459373-a78d-4bdb-a207-3df331b6bb18" containerName="limitador" containerID="cri-o://3839ebec44dee4cfa731b5456aef7ed8f2659b980564c00416a4f3df6b5d9bd6" gracePeriod=30 Oct 31 12:48:13.524687 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:13.524671 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-jb7gv" Oct 31 12:48:13.948296 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:13.948268 5693 generic.go:356] "Generic (PLEG): container finished" podID="49459373-a78d-4bdb-a207-3df331b6bb18" containerID="3839ebec44dee4cfa731b5456aef7ed8f2659b980564c00416a4f3df6b5d9bd6" exitCode=0 Oct 31 12:48:13.948422 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:13.948336 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-jb7gv" event={"ID":"49459373-a78d-4bdb-a207-3df331b6bb18","Type":"ContainerDied","Data":"3839ebec44dee4cfa731b5456aef7ed8f2659b980564c00416a4f3df6b5d9bd6"} Oct 31 12:48:14.059372 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:14.059349 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-jb7gv" Oct 31 12:48:14.155199 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:14.155170 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49459373-a78d-4bdb-a207-3df331b6bb18-config-file\" (UniqueName: \"kubernetes.io/configmap/49459373-a78d-4bdb-a207-3df331b6bb18-config-file\") pod \"49459373-a78d-4bdb-a207-3df331b6bb18\" (UID: \"49459373-a78d-4bdb-a207-3df331b6bb18\") " Oct 31 12:48:14.155357 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:14.155240 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49459373-a78d-4bdb-a207-3df331b6bb18-kube-api-access-g6gm7\" (UniqueName: \"kubernetes.io/projected/49459373-a78d-4bdb-a207-3df331b6bb18-kube-api-access-g6gm7\") pod \"49459373-a78d-4bdb-a207-3df331b6bb18\" (UID: \"49459373-a78d-4bdb-a207-3df331b6bb18\") " Oct 31 12:48:14.155579 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:14.155556 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49459373-a78d-4bdb-a207-3df331b6bb18-config-file" pod "49459373-a78d-4bdb-a207-3df331b6bb18" (UID: "49459373-a78d-4bdb-a207-3df331b6bb18"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:48:14.157467 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:14.157444 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49459373-a78d-4bdb-a207-3df331b6bb18-kube-api-access-g6gm7" pod "49459373-a78d-4bdb-a207-3df331b6bb18" (UID: "49459373-a78d-4bdb-a207-3df331b6bb18"). InnerVolumeSpecName "kube-api-access-g6gm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:48:14.256091 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:14.256052 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6gm7\" (UniqueName: \"kubernetes.io/projected/49459373-a78d-4bdb-a207-3df331b6bb18-kube-api-access-g6gm7\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:48:14.256091 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:14.256085 5693 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/49459373-a78d-4bdb-a207-3df331b6bb18-config-file\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:48:14.953824 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:14.953792 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-jb7gv" event={"ID":"49459373-a78d-4bdb-a207-3df331b6bb18","Type":"ContainerDied","Data":"001ae4b97d74c2489827b7f6b9b2638ec359e4aa7fd6a1b69f336be874dc7000"} Oct 31 12:48:14.954245 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:14.953847 5693 scope.go:122] "RemoveContainer" containerID="3839ebec44dee4cfa731b5456aef7ed8f2659b980564c00416a4f3df6b5d9bd6" Oct 31 12:48:14.954245 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:14.953806 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-jb7gv" Oct 31 12:48:14.975463 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:14.975437 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jb7gv"] Oct 31 12:48:14.984097 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:14.984066 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jb7gv"] Oct 31 12:48:15.405178 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:15.405146 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49459373-a78d-4bdb-a207-3df331b6bb18" path="/var/lib/kubelet/pods/49459373-a78d-4bdb-a207-3df331b6bb18/volumes" Oct 31 12:48:18.685819 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:48:18.685782 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:48:19.991508 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:19.991480 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-rqmck"] Oct 31 12:48:19.991918 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:19.991826 5693 cpu_manager.go:414] "containerMap: removing container" podUID="92c41689-5020-4a34-a9e6-1f2883cce375" containerName="authorino" Oct 31 12:48:19.991918 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:19.991838 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92c41689-5020-4a34-a9e6-1f2883cce375" containerName="authorino" Oct 31 12:48:19.991918 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:19.991867 5693 cpu_manager.go:414] "containerMap: removing container" podUID="49459373-a78d-4bdb-a207-3df331b6bb18" containerName="limitador" Oct 31 12:48:19.991918 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:19.991873 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49459373-a78d-4bdb-a207-3df331b6bb18" containerName="limitador" Oct 31 12:48:19.992044 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:19.991921 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="49459373-a78d-4bdb-a207-3df331b6bb18" containerName="limitador" Oct 31 12:48:19.992044 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:19.991933 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="92c41689-5020-4a34-a9e6-1f2883cce375" containerName="authorino" Oct 31 12:48:19.995987 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:19.995971 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rqmck" Oct 31 12:48:20.003214 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:20.002526 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 12:48:20.003691 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:20.003667 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-rqmck"] Oct 31 12:48:20.005772 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:20.005752 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 31 12:48:20.005913 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:20.005872 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-7wv2l\"" Oct 31 12:48:20.100005 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:20.099971 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxbvc\" (UniqueName: \"kubernetes.io/projected/b8aff5f5-e566-4b4d-9360-a33187a1119c-kube-api-access-bxbvc\") pod \"postgres-7cdbf6dc44-rqmck\" (UID: \"b8aff5f5-e566-4b4d-9360-a33187a1119c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rqmck" Oct 31 12:48:20.100176 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:20.100041 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b8aff5f5-e566-4b4d-9360-a33187a1119c-data\") pod \"postgres-7cdbf6dc44-rqmck\" (UID: \"b8aff5f5-e566-4b4d-9360-a33187a1119c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rqmck" Oct 31 12:48:20.103029 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:20.103009 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 12:48:20.200601 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:20.200570 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxbvc\" (UniqueName: \"kubernetes.io/projected/b8aff5f5-e566-4b4d-9360-a33187a1119c-kube-api-access-bxbvc\") pod \"postgres-7cdbf6dc44-rqmck\" (UID: \"b8aff5f5-e566-4b4d-9360-a33187a1119c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rqmck" Oct 31 12:48:20.200785 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:20.200628 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b8aff5f5-e566-4b4d-9360-a33187a1119c-data\") pod \"postgres-7cdbf6dc44-rqmck\" (UID: \"b8aff5f5-e566-4b4d-9360-a33187a1119c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rqmck" Oct 31 12:48:20.201001 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:20.200985 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b8aff5f5-e566-4b4d-9360-a33187a1119c-data\") pod \"postgres-7cdbf6dc44-rqmck\" (UID: \"b8aff5f5-e566-4b4d-9360-a33187a1119c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rqmck" Oct 31 12:48:20.213490 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:20.213469 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxbvc\" (UniqueName: \"kubernetes.io/projected/b8aff5f5-e566-4b4d-9360-a33187a1119c-kube-api-access-bxbvc\") pod \"postgres-7cdbf6dc44-rqmck\" (UID: \"b8aff5f5-e566-4b4d-9360-a33187a1119c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rqmck" Oct 31 12:48:20.309185 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:20.309104 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rqmck" Oct 31 12:48:20.585437 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:20.585357 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-rqmck"] Oct 31 12:48:20.588106 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:48:20.588078 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8aff5f5_e566_4b4d_9360_a33187a1119c.slice/crio-824817837bc8ecadcc5e4ae1cae39247f9ccdd813e36bc28bb122da4a7c83099 WatchSource:0}: Error finding container 824817837bc8ecadcc5e4ae1cae39247f9ccdd813e36bc28bb122da4a7c83099: Status 404 returned error can't find the container with id 824817837bc8ecadcc5e4ae1cae39247f9ccdd813e36bc28bb122da4a7c83099 Oct 31 12:48:20.980548 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:20.980513 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rqmck" event={"ID":"b8aff5f5-e566-4b4d-9360-a33187a1119c","Type":"ContainerStarted","Data":"824817837bc8ecadcc5e4ae1cae39247f9ccdd813e36bc28bb122da4a7c83099"} Oct 31 12:48:28.030315 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:28.030275 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rqmck" event={"ID":"b8aff5f5-e566-4b4d-9360-a33187a1119c","Type":"ContainerStarted","Data":"a6d8d71df6fb84b9fc08d3aefc6b863d7a718630490bdb46d7af44326ddf93e5"} Oct 31 12:48:28.030751 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:28.030382 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rqmck" Oct 31 12:48:28.049676 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:28.048565 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rqmck" podStartSLOduration=2.197549161 podStartE2EDuration="9.04854866s" totalImagesPullingTime="6.850999499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:48:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:48:28.047789799 +0000 UTC m=+958.755376072" watchObservedRunningTime="2026-10-31 12:48:28.04854866 +0000 UTC m=+958.756134956" Oct 31 12:48:34.063441 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:34.063411 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rqmck" Oct 31 12:48:41.226347 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:41.226312 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-dc8d885b8-gss4t"] Oct 31 12:48:41.229676 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:41.229660 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-dc8d885b8-gss4t" Oct 31 12:48:41.234183 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:41.234156 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jnfxm\"" Oct 31 12:48:41.234183 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:41.234171 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 31 12:48:41.234183 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:41.234190 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 12:48:41.234380 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:41.234202 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 12:48:41.236340 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:41.236323 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 31 12:48:41.240813 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:41.240794 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-dc8d885b8-gss4t"] Oct 31 12:48:41.269354 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:41.269326 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8fd0a33-2ae9-4606-99f4-05e40fda900c-webhook-cert\") pod \"maas-controller-dc8d885b8-gss4t\" (UID: \"e8fd0a33-2ae9-4606-99f4-05e40fda900c\") " pod="opendatahub/maas-controller-dc8d885b8-gss4t" Oct 31 12:48:41.269499 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:41.269374 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8fd0a33-2ae9-4606-99f4-05e40fda900c-metrics-certs\") pod \"maas-controller-dc8d885b8-gss4t\" (UID: \"e8fd0a33-2ae9-4606-99f4-05e40fda900c\") " pod="opendatahub/maas-controller-dc8d885b8-gss4t" Oct 31 12:48:41.269499 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:41.269485 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96nh4\" (UniqueName: \"kubernetes.io/projected/e8fd0a33-2ae9-4606-99f4-05e40fda900c-kube-api-access-96nh4\") pod \"maas-controller-dc8d885b8-gss4t\" (UID: \"e8fd0a33-2ae9-4606-99f4-05e40fda900c\") " pod="opendatahub/maas-controller-dc8d885b8-gss4t" Oct 31 12:48:41.334104 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:41.334073 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 12:48:41.370119 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:41.370088 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8fd0a33-2ae9-4606-99f4-05e40fda900c-webhook-cert\") pod \"maas-controller-dc8d885b8-gss4t\" (UID: \"e8fd0a33-2ae9-4606-99f4-05e40fda900c\") " pod="opendatahub/maas-controller-dc8d885b8-gss4t" Oct 31 12:48:41.370255 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:41.370139 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8fd0a33-2ae9-4606-99f4-05e40fda900c-metrics-certs\") pod \"maas-controller-dc8d885b8-gss4t\" (UID: \"e8fd0a33-2ae9-4606-99f4-05e40fda900c\") " pod="opendatahub/maas-controller-dc8d885b8-gss4t" Oct 31 12:48:41.370255 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:41.370164 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96nh4\" (UniqueName: \"kubernetes.io/projected/e8fd0a33-2ae9-4606-99f4-05e40fda900c-kube-api-access-96nh4\") pod \"maas-controller-dc8d885b8-gss4t\" (UID: \"e8fd0a33-2ae9-4606-99f4-05e40fda900c\") " pod="opendatahub/maas-controller-dc8d885b8-gss4t" Oct 31 12:48:41.372675 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:41.372634 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8fd0a33-2ae9-4606-99f4-05e40fda900c-metrics-certs\") pod \"maas-controller-dc8d885b8-gss4t\" (UID: \"e8fd0a33-2ae9-4606-99f4-05e40fda900c\") " pod="opendatahub/maas-controller-dc8d885b8-gss4t" Oct 31 12:48:41.372834 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:41.372815 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8fd0a33-2ae9-4606-99f4-05e40fda900c-webhook-cert\") pod \"maas-controller-dc8d885b8-gss4t\" (UID: \"e8fd0a33-2ae9-4606-99f4-05e40fda900c\") " pod="opendatahub/maas-controller-dc8d885b8-gss4t" Oct 31 12:48:41.382870 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:41.382852 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96nh4\" (UniqueName: \"kubernetes.io/projected/e8fd0a33-2ae9-4606-99f4-05e40fda900c-kube-api-access-96nh4\") pod \"maas-controller-dc8d885b8-gss4t\" (UID: \"e8fd0a33-2ae9-4606-99f4-05e40fda900c\") " pod="opendatahub/maas-controller-dc8d885b8-gss4t" Oct 31 12:48:41.542190 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:41.542085 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-dc8d885b8-gss4t" Oct 31 12:48:41.727928 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:41.727896 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7767cbb97d-2mzd8"] Oct 31 12:48:41.732138 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:41.732113 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7767cbb97d-2mzd8" Oct 31 12:48:41.744809 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:41.744780 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7767cbb97d-2mzd8"] Oct 31 12:48:41.773023 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:41.772998 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/052fb6af-f18c-429c-8946-2022c122adfd-webhook-cert\") pod \"maas-controller-7767cbb97d-2mzd8\" (UID: \"052fb6af-f18c-429c-8946-2022c122adfd\") " pod="opendatahub/maas-controller-7767cbb97d-2mzd8" Oct 31 12:48:41.773168 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:41.773032 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grwzv\" (UniqueName: \"kubernetes.io/projected/052fb6af-f18c-429c-8946-2022c122adfd-kube-api-access-grwzv\") pod \"maas-controller-7767cbb97d-2mzd8\" (UID: \"052fb6af-f18c-429c-8946-2022c122adfd\") " pod="opendatahub/maas-controller-7767cbb97d-2mzd8" Oct 31 12:48:41.773168 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:41.773138 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/052fb6af-f18c-429c-8946-2022c122adfd-metrics-certs\") pod \"maas-controller-7767cbb97d-2mzd8\" (UID: \"052fb6af-f18c-429c-8946-2022c122adfd\") " pod="opendatahub/maas-controller-7767cbb97d-2mzd8" Oct 31 12:48:41.816438 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:41.816371 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-dc8d885b8-gss4t"] Oct 31 12:48:41.819559 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:48:41.819538 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8fd0a33_2ae9_4606_99f4_05e40fda900c.slice/crio-5d33a61c36d63c708294336229f66ee7adbd7b308aabc4023501b5c40d49e7fd WatchSource:0}: Error finding container 5d33a61c36d63c708294336229f66ee7adbd7b308aabc4023501b5c40d49e7fd: Status 404 returned error can't find the container with id 5d33a61c36d63c708294336229f66ee7adbd7b308aabc4023501b5c40d49e7fd Oct 31 12:48:41.874416 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:41.874383 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grwzv\" (UniqueName: \"kubernetes.io/projected/052fb6af-f18c-429c-8946-2022c122adfd-kube-api-access-grwzv\") pod \"maas-controller-7767cbb97d-2mzd8\" (UID: \"052fb6af-f18c-429c-8946-2022c122adfd\") " pod="opendatahub/maas-controller-7767cbb97d-2mzd8" Oct 31 12:48:41.874571 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:41.874445 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/052fb6af-f18c-429c-8946-2022c122adfd-metrics-certs\") pod \"maas-controller-7767cbb97d-2mzd8\" (UID: \"052fb6af-f18c-429c-8946-2022c122adfd\") " pod="opendatahub/maas-controller-7767cbb97d-2mzd8" Oct 31 12:48:41.874571 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:41.874477 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/052fb6af-f18c-429c-8946-2022c122adfd-webhook-cert\") pod \"maas-controller-7767cbb97d-2mzd8\" (UID: \"052fb6af-f18c-429c-8946-2022c122adfd\") " pod="opendatahub/maas-controller-7767cbb97d-2mzd8" Oct 31 12:48:41.877109 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:41.877087 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/052fb6af-f18c-429c-8946-2022c122adfd-webhook-cert\") pod \"maas-controller-7767cbb97d-2mzd8\" (UID: \"052fb6af-f18c-429c-8946-2022c122adfd\") " pod="opendatahub/maas-controller-7767cbb97d-2mzd8" Oct 31 12:48:41.877206 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:41.877119 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/052fb6af-f18c-429c-8946-2022c122adfd-metrics-certs\") pod \"maas-controller-7767cbb97d-2mzd8\" (UID: \"052fb6af-f18c-429c-8946-2022c122adfd\") " pod="opendatahub/maas-controller-7767cbb97d-2mzd8" Oct 31 12:48:41.886490 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:41.886471 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grwzv\" (UniqueName: \"kubernetes.io/projected/052fb6af-f18c-429c-8946-2022c122adfd-kube-api-access-grwzv\") pod \"maas-controller-7767cbb97d-2mzd8\" (UID: \"052fb6af-f18c-429c-8946-2022c122adfd\") " pod="opendatahub/maas-controller-7767cbb97d-2mzd8" Oct 31 12:48:42.045112 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:42.045077 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7767cbb97d-2mzd8" Oct 31 12:48:42.088357 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:42.088272 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-dc8d885b8-gss4t" event={"ID":"e8fd0a33-2ae9-4606-99f4-05e40fda900c","Type":"ContainerStarted","Data":"5d33a61c36d63c708294336229f66ee7adbd7b308aabc4023501b5c40d49e7fd"} Oct 31 12:48:42.321681 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:42.321632 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7767cbb97d-2mzd8"] Oct 31 12:48:42.324681 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:48:42.324630 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod052fb6af_f18c_429c_8946_2022c122adfd.slice/crio-4a0cde9a35dcb2a6baf260d405defac000c55dcf34f52515f4e49e7f81084394 WatchSource:0}: Error finding container 4a0cde9a35dcb2a6baf260d405defac000c55dcf34f52515f4e49e7f81084394: Status 404 returned error can't find the container with id 4a0cde9a35dcb2a6baf260d405defac000c55dcf34f52515f4e49e7f81084394 Oct 31 12:48:43.094851 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:43.094813 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7767cbb97d-2mzd8" event={"ID":"052fb6af-f18c-429c-8946-2022c122adfd","Type":"ContainerStarted","Data":"4a0cde9a35dcb2a6baf260d405defac000c55dcf34f52515f4e49e7f81084394"} Oct 31 12:48:47.123877 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:47.123839 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7767cbb97d-2mzd8" event={"ID":"052fb6af-f18c-429c-8946-2022c122adfd","Type":"ContainerStarted","Data":"1b2bf9172fab3059a265f2ad41e1444280d9efe1ec24d57a1654ebeb84874d5b"} Oct 31 12:48:47.124318 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:47.123929 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7767cbb97d-2mzd8" Oct 31 12:48:47.125324 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:47.125298 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-dc8d885b8-gss4t" event={"ID":"e8fd0a33-2ae9-4606-99f4-05e40fda900c","Type":"ContainerStarted","Data":"4f65ea97cd19bf31e81c19bc8303e4f993f4c51059c8c9334a0db53e64376b11"} Oct 31 12:48:47.125438 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:47.125424 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-dc8d885b8-gss4t" Oct 31 12:48:47.180620 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:47.180549 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7767cbb97d-2mzd8" podStartSLOduration=2.44499592 podStartE2EDuration="6.18053016s" totalImagesPullingTime="3.73553424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:48:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:48:47.178997189 +0000 UTC m=+977.886583483" watchObservedRunningTime="2026-10-31 12:48:47.18053016 +0000 UTC m=+977.888116469" Oct 31 12:48:47.207516 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:47.207465 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-dc8d885b8-gss4t" podStartSLOduration=1.9126548140000001 podStartE2EDuration="6.20745062s" totalImagesPullingTime="4.294795806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:48:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:48:47.206546406 +0000 UTC m=+977.914132700" watchObservedRunningTime="2026-10-31 12:48:47.20745062 +0000 UTC m=+977.915036914" Oct 31 12:48:47.788493 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:47.788461 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2wppt"] Oct 31 12:48:47.812231 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:47.812198 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2wppt"] Oct 31 12:48:47.812394 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:47.812305 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2wppt" Oct 31 12:48:47.816034 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:47.816003 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 31 12:48:47.816972 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:47.816938 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-cfxkb\"" Oct 31 12:48:47.816972 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:47.816943 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 31 12:48:47.817214 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:47.816999 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 31 12:48:47.932268 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:47.932219 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75062797-634c-4d93-9c96-7c0060e0f684-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-2wppt\" (UID: \"75062797-634c-4d93-9c96-7c0060e0f684\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2wppt" Oct 31 12:48:47.932441 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:47.932393 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t8ps\" (UniqueName: \"kubernetes.io/projected/75062797-634c-4d93-9c96-7c0060e0f684-kube-api-access-5t8ps\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-2wppt\" (UID: \"75062797-634c-4d93-9c96-7c0060e0f684\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2wppt" Oct 31 12:48:47.932507 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:47.932483 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/75062797-634c-4d93-9c96-7c0060e0f684-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-2wppt\" (UID: \"75062797-634c-4d93-9c96-7c0060e0f684\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2wppt" Oct 31 12:48:47.932566 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:47.932549 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/75062797-634c-4d93-9c96-7c0060e0f684-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-2wppt\" (UID: \"75062797-634c-4d93-9c96-7c0060e0f684\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2wppt" Oct 31 12:48:48.033534 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:48.033499 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t8ps\" (UniqueName: \"kubernetes.io/projected/75062797-634c-4d93-9c96-7c0060e0f684-kube-api-access-5t8ps\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-2wppt\" (UID: \"75062797-634c-4d93-9c96-7c0060e0f684\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2wppt" Oct 31 12:48:48.033534 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:48.033538 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/75062797-634c-4d93-9c96-7c0060e0f684-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-2wppt\" (UID: \"75062797-634c-4d93-9c96-7c0060e0f684\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2wppt" Oct 31 12:48:48.033782 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:48.033562 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/75062797-634c-4d93-9c96-7c0060e0f684-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-2wppt\" (UID: \"75062797-634c-4d93-9c96-7c0060e0f684\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2wppt" Oct 31 12:48:48.033782 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:48.033611 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75062797-634c-4d93-9c96-7c0060e0f684-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-2wppt\" (UID: \"75062797-634c-4d93-9c96-7c0060e0f684\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2wppt" Oct 31 12:48:48.034344 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:48.034288 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75062797-634c-4d93-9c96-7c0060e0f684-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-2wppt\" (UID: \"75062797-634c-4d93-9c96-7c0060e0f684\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2wppt" Oct 31 12:48:48.034560 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:48.034484 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/75062797-634c-4d93-9c96-7c0060e0f684-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-2wppt\" (UID: \"75062797-634c-4d93-9c96-7c0060e0f684\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2wppt" Oct 31 12:48:48.036373 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:48.036348 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/75062797-634c-4d93-9c96-7c0060e0f684-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-2wppt\" (UID: \"75062797-634c-4d93-9c96-7c0060e0f684\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2wppt" Oct 31 12:48:48.045579 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:48.045526 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t8ps\" (UniqueName: \"kubernetes.io/projected/75062797-634c-4d93-9c96-7c0060e0f684-kube-api-access-5t8ps\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-2wppt\" (UID: \"75062797-634c-4d93-9c96-7c0060e0f684\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2wppt" Oct 31 12:48:48.125900 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:48.125863 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2wppt" Oct 31 12:48:48.400493 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:48.400450 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2wppt"] Oct 31 12:48:48.403571 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:48:48.403543 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75062797_634c_4d93_9c96_7c0060e0f684.slice/crio-661e1883a0ded33bce98dc8fc4c5e96cb1d347c0f90f32b971112dbea90f6432 WatchSource:0}: Error finding container 661e1883a0ded33bce98dc8fc4c5e96cb1d347c0f90f32b971112dbea90f6432: Status 404 returned error can't find the container with id 661e1883a0ded33bce98dc8fc4c5e96cb1d347c0f90f32b971112dbea90f6432 Oct 31 12:48:48.685831 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:48:48.685753 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:48:49.133604 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:49.133523 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2wppt" event={"ID":"75062797-634c-4d93-9c96-7c0060e0f684","Type":"ContainerStarted","Data":"661e1883a0ded33bce98dc8fc4c5e96cb1d347c0f90f32b971112dbea90f6432"} Oct 31 12:48:58.132040 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:58.132009 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-dc8d885b8-gss4t" Oct 31 12:48:58.132522 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:58.132094 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7767cbb97d-2mzd8" Oct 31 12:48:58.204816 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:58.204781 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-dc8d885b8-gss4t"] Oct 31 12:48:58.205026 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:58.205002 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-dc8d885b8-gss4t" podUID="e8fd0a33-2ae9-4606-99f4-05e40fda900c" containerName="manager" containerID="cri-o://4f65ea97cd19bf31e81c19bc8303e4f993f4c51059c8c9334a0db53e64376b11" gracePeriod=30 Oct 31 12:48:58.478711 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:58.478424 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-dc8d885b8-gss4t" Oct 31 12:48:58.631612 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:58.631578 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8fd0a33-2ae9-4606-99f4-05e40fda900c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8fd0a33-2ae9-4606-99f4-05e40fda900c-metrics-certs\") pod \"e8fd0a33-2ae9-4606-99f4-05e40fda900c\" (UID: \"e8fd0a33-2ae9-4606-99f4-05e40fda900c\") " Oct 31 12:48:58.631816 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:58.631633 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8fd0a33-2ae9-4606-99f4-05e40fda900c-kube-api-access-96nh4\" (UniqueName: \"kubernetes.io/projected/e8fd0a33-2ae9-4606-99f4-05e40fda900c-kube-api-access-96nh4\") pod \"e8fd0a33-2ae9-4606-99f4-05e40fda900c\" (UID: \"e8fd0a33-2ae9-4606-99f4-05e40fda900c\") " Oct 31 12:48:58.631816 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:58.631800 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8fd0a33-2ae9-4606-99f4-05e40fda900c-webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8fd0a33-2ae9-4606-99f4-05e40fda900c-webhook-cert\") pod \"e8fd0a33-2ae9-4606-99f4-05e40fda900c\" (UID: \"e8fd0a33-2ae9-4606-99f4-05e40fda900c\") " Oct 31 12:48:58.634356 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:58.634329 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8fd0a33-2ae9-4606-99f4-05e40fda900c-metrics-certs" pod "e8fd0a33-2ae9-4606-99f4-05e40fda900c" (UID: "e8fd0a33-2ae9-4606-99f4-05e40fda900c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:48:58.634489 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:58.634466 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8fd0a33-2ae9-4606-99f4-05e40fda900c-kube-api-access-96nh4" pod "e8fd0a33-2ae9-4606-99f4-05e40fda900c" (UID: "e8fd0a33-2ae9-4606-99f4-05e40fda900c"). InnerVolumeSpecName "kube-api-access-96nh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:48:58.634542 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:58.634486 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8fd0a33-2ae9-4606-99f4-05e40fda900c-webhook-cert" pod "e8fd0a33-2ae9-4606-99f4-05e40fda900c" (UID: "e8fd0a33-2ae9-4606-99f4-05e40fda900c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:48:58.732701 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:58.732612 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96nh4\" (UniqueName: \"kubernetes.io/projected/e8fd0a33-2ae9-4606-99f4-05e40fda900c-kube-api-access-96nh4\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:48:58.732701 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:58.732663 5693 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8fd0a33-2ae9-4606-99f4-05e40fda900c-webhook-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:48:58.732701 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:58.732679 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8fd0a33-2ae9-4606-99f4-05e40fda900c-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:48:59.180133 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:59.180085 5693 generic.go:356] "Generic (PLEG): container finished" podID="e8fd0a33-2ae9-4606-99f4-05e40fda900c" containerID="4f65ea97cd19bf31e81c19bc8303e4f993f4c51059c8c9334a0db53e64376b11" exitCode=0 Oct 31 12:48:59.180557 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:59.180138 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-dc8d885b8-gss4t" Oct 31 12:48:59.180557 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:59.180148 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-dc8d885b8-gss4t" event={"ID":"e8fd0a33-2ae9-4606-99f4-05e40fda900c","Type":"ContainerDied","Data":"4f65ea97cd19bf31e81c19bc8303e4f993f4c51059c8c9334a0db53e64376b11"} Oct 31 12:48:59.180557 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:59.180202 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-dc8d885b8-gss4t" event={"ID":"e8fd0a33-2ae9-4606-99f4-05e40fda900c","Type":"ContainerDied","Data":"5d33a61c36d63c708294336229f66ee7adbd7b308aabc4023501b5c40d49e7fd"} Oct 31 12:48:59.180557 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:59.180234 5693 scope.go:122] "RemoveContainer" containerID="4f65ea97cd19bf31e81c19bc8303e4f993f4c51059c8c9334a0db53e64376b11" Oct 31 12:48:59.190224 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:59.190208 5693 scope.go:122] "RemoveContainer" containerID="4f65ea97cd19bf31e81c19bc8303e4f993f4c51059c8c9334a0db53e64376b11" Oct 31 12:48:59.190493 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:48:59.190473 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f65ea97cd19bf31e81c19bc8303e4f993f4c51059c8c9334a0db53e64376b11\": container with ID starting with 4f65ea97cd19bf31e81c19bc8303e4f993f4c51059c8c9334a0db53e64376b11 not found: ID does not exist" containerID="4f65ea97cd19bf31e81c19bc8303e4f993f4c51059c8c9334a0db53e64376b11" Oct 31 12:48:59.190567 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:59.190502 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f65ea97cd19bf31e81c19bc8303e4f993f4c51059c8c9334a0db53e64376b11"} err="failed to get container status \"4f65ea97cd19bf31e81c19bc8303e4f993f4c51059c8c9334a0db53e64376b11\": rpc error: code = NotFound desc = could not find container \"4f65ea97cd19bf31e81c19bc8303e4f993f4c51059c8c9334a0db53e64376b11\": container with ID starting with 4f65ea97cd19bf31e81c19bc8303e4f993f4c51059c8c9334a0db53e64376b11 not found: ID does not exist" Oct 31 12:48:59.201862 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:59.201840 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-dc8d885b8-gss4t"] Oct 31 12:48:59.206334 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:59.206309 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-dc8d885b8-gss4t"] Oct 31 12:48:59.409725 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:48:59.409691 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8fd0a33-2ae9-4606-99f4-05e40fda900c" path="/var/lib/kubelet/pods/e8fd0a33-2ae9-4606-99f4-05e40fda900c/volumes" Oct 31 12:49:03.198933 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:03.198894 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2wppt" event={"ID":"75062797-634c-4d93-9c96-7c0060e0f684","Type":"ContainerStarted","Data":"9ad0d07133f5b25741486d4d21a89b3207ae392eb0acb1c8563c5abe3dbf1e4e"} Oct 31 12:49:03.199437 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:03.199019 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2wppt" Oct 31 12:49:03.219332 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:03.219288 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2wppt" podStartSLOduration=2.364244596 podStartE2EDuration="16.21927446s" totalImagesPullingTime="13.855029864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:49:03.218354387 +0000 UTC m=+993.925940682" watchObservedRunningTime="2026-10-31 12:49:03.21927446 +0000 UTC m=+993.926860754" Oct 31 12:49:14.208551 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:14.208478 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2wppt" Oct 31 12:49:18.685385 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:49:18.685348 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:49:19.434368 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:19.434328 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7c54796cf8-7bkc9"] Oct 31 12:49:19.434688 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:19.434664 5693 cpu_manager.go:414] "containerMap: removing container" podUID="e8fd0a33-2ae9-4606-99f4-05e40fda900c" containerName="manager" Oct 31 12:49:19.434688 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:19.434686 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8fd0a33-2ae9-4606-99f4-05e40fda900c" containerName="manager" Oct 31 12:49:19.434868 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:19.434792 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8fd0a33-2ae9-4606-99f4-05e40fda900c" containerName="manager" Oct 31 12:49:19.440811 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:19.440788 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c54796cf8-7bkc9" Oct 31 12:49:19.450048 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:19.450025 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rhftr\"" Oct 31 12:49:19.450202 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:19.450118 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 12:49:19.450202 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:19.450193 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 31 12:49:19.451221 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:19.451194 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c54796cf8-7bkc9"] Oct 31 12:49:19.501936 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:19.501898 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c76fb2a1-f89a-4614-b1ac-2cea628ce3bb-maas-api-tls\") pod \"maas-api-7c54796cf8-7bkc9\" (UID: \"c76fb2a1-f89a-4614-b1ac-2cea628ce3bb\") " pod="odh-ai-gateway-infra/maas-api-7c54796cf8-7bkc9" Oct 31 12:49:19.502105 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:19.501954 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkl7d\" (UniqueName: \"kubernetes.io/projected/c76fb2a1-f89a-4614-b1ac-2cea628ce3bb-kube-api-access-xkl7d\") pod \"maas-api-7c54796cf8-7bkc9\" (UID: \"c76fb2a1-f89a-4614-b1ac-2cea628ce3bb\") " pod="odh-ai-gateway-infra/maas-api-7c54796cf8-7bkc9" Oct 31 12:49:19.502105 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:19.502047 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c76fb2a1-f89a-4614-b1ac-2cea628ce3bb-metrics-certs\") pod \"maas-api-7c54796cf8-7bkc9\" (UID: \"c76fb2a1-f89a-4614-b1ac-2cea628ce3bb\") " pod="odh-ai-gateway-infra/maas-api-7c54796cf8-7bkc9" Oct 31 12:49:19.502105 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:19.502076 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c76fb2a1-f89a-4614-b1ac-2cea628ce3bb-openshift-service-ca\") pod \"maas-api-7c54796cf8-7bkc9\" (UID: \"c76fb2a1-f89a-4614-b1ac-2cea628ce3bb\") " pod="odh-ai-gateway-infra/maas-api-7c54796cf8-7bkc9" Oct 31 12:49:19.602711 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:19.602671 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c76fb2a1-f89a-4614-b1ac-2cea628ce3bb-maas-api-tls\") pod \"maas-api-7c54796cf8-7bkc9\" (UID: \"c76fb2a1-f89a-4614-b1ac-2cea628ce3bb\") " pod="odh-ai-gateway-infra/maas-api-7c54796cf8-7bkc9" Oct 31 12:49:19.602902 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:19.602766 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkl7d\" (UniqueName: \"kubernetes.io/projected/c76fb2a1-f89a-4614-b1ac-2cea628ce3bb-kube-api-access-xkl7d\") pod \"maas-api-7c54796cf8-7bkc9\" (UID: \"c76fb2a1-f89a-4614-b1ac-2cea628ce3bb\") " pod="odh-ai-gateway-infra/maas-api-7c54796cf8-7bkc9" Oct 31 12:49:19.602902 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:19.602794 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c76fb2a1-f89a-4614-b1ac-2cea628ce3bb-metrics-certs\") pod \"maas-api-7c54796cf8-7bkc9\" (UID: \"c76fb2a1-f89a-4614-b1ac-2cea628ce3bb\") " pod="odh-ai-gateway-infra/maas-api-7c54796cf8-7bkc9" Oct 31 12:49:19.602902 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:19.602822 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c76fb2a1-f89a-4614-b1ac-2cea628ce3bb-openshift-service-ca\") pod \"maas-api-7c54796cf8-7bkc9\" (UID: \"c76fb2a1-f89a-4614-b1ac-2cea628ce3bb\") " pod="odh-ai-gateway-infra/maas-api-7c54796cf8-7bkc9" Oct 31 12:49:19.603434 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:19.603410 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c76fb2a1-f89a-4614-b1ac-2cea628ce3bb-openshift-service-ca\") pod \"maas-api-7c54796cf8-7bkc9\" (UID: \"c76fb2a1-f89a-4614-b1ac-2cea628ce3bb\") " pod="odh-ai-gateway-infra/maas-api-7c54796cf8-7bkc9" Oct 31 12:49:19.605248 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:19.605226 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c76fb2a1-f89a-4614-b1ac-2cea628ce3bb-metrics-certs\") pod \"maas-api-7c54796cf8-7bkc9\" (UID: \"c76fb2a1-f89a-4614-b1ac-2cea628ce3bb\") " pod="odh-ai-gateway-infra/maas-api-7c54796cf8-7bkc9" Oct 31 12:49:19.605403 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:19.605380 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c76fb2a1-f89a-4614-b1ac-2cea628ce3bb-maas-api-tls\") pod \"maas-api-7c54796cf8-7bkc9\" (UID: \"c76fb2a1-f89a-4614-b1ac-2cea628ce3bb\") " pod="odh-ai-gateway-infra/maas-api-7c54796cf8-7bkc9" Oct 31 12:49:19.615430 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:19.615410 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkl7d\" (UniqueName: \"kubernetes.io/projected/c76fb2a1-f89a-4614-b1ac-2cea628ce3bb-kube-api-access-xkl7d\") pod \"maas-api-7c54796cf8-7bkc9\" (UID: \"c76fb2a1-f89a-4614-b1ac-2cea628ce3bb\") " pod="odh-ai-gateway-infra/maas-api-7c54796cf8-7bkc9" Oct 31 12:49:19.758028 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:19.757949 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c54796cf8-7bkc9" Oct 31 12:49:20.037865 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:20.037812 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c54796cf8-7bkc9"] Oct 31 12:49:20.040050 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:49:20.040025 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc76fb2a1_f89a_4614_b1ac_2cea628ce3bb.slice/crio-61bfce49aee9e8efcd827e539d85cf9b3d88b4d3e4bb486dc562b6aa08312d53 WatchSource:0}: Error finding container 61bfce49aee9e8efcd827e539d85cf9b3d88b4d3e4bb486dc562b6aa08312d53: Status 404 returned error can't find the container with id 61bfce49aee9e8efcd827e539d85cf9b3d88b4d3e4bb486dc562b6aa08312d53 Oct 31 12:49:20.267183 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:20.267148 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c54796cf8-7bkc9" event={"ID":"c76fb2a1-f89a-4614-b1ac-2cea628ce3bb","Type":"ContainerStarted","Data":"61bfce49aee9e8efcd827e539d85cf9b3d88b4d3e4bb486dc562b6aa08312d53"} Oct 31 12:49:25.290372 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:25.290336 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c54796cf8-7bkc9" event={"ID":"c76fb2a1-f89a-4614-b1ac-2cea628ce3bb","Type":"ContainerStarted","Data":"f1468b751451b34f4deea7d7eb707714ee765f8f43c03c197ea1ef5b1fc87809"} Oct 31 12:49:25.290810 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:25.290440 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7c54796cf8-7bkc9" Oct 31 12:49:25.312233 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:25.312159 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7c54796cf8-7bkc9" podStartSLOduration=2.002031508 podStartE2EDuration="6.312146296s" totalImagesPullingTime="4.310114788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:49:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:49:25.310394768 +0000 UTC m=+1016.017981060" watchObservedRunningTime="2026-10-31 12:49:25.312146296 +0000 UTC m=+1016.019732590" Oct 31 12:49:31.299378 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:31.299346 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7c54796cf8-7bkc9" Oct 31 12:49:32.055036 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.054986 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-btnvt"] Oct 31 12:49:32.184897 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.184869 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-btnvt"] Oct 31 12:49:32.184897 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.184903 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-btnvt"] Oct 31 12:49:32.185098 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.184988 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-btnvt" Oct 31 12:49:32.192893 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.192871 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2kvfh"] Oct 31 12:49:32.202224 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.202204 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-btnvt" Oct 31 12:49:32.202366 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.202241 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-2kvfh" Oct 31 12:49:32.207114 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.207088 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2kvfh"] Oct 31 12:49:32.208494 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.208465 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 31 12:49:32.248730 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.248694 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-547b4fbdb9-zxdgg"] Oct 31 12:49:32.258103 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.258084 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-zxdgg" Oct 31 12:49:32.264621 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.264455 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-547b4fbdb9-zxdgg"] Oct 31 12:49:32.313700 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.313620 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/796a4469-22c6-4d99-b8e4-34f417c8661d-openshift-service-ca\") pod \"maas-api-547b4fbdb9-zxdgg\" (UID: \"796a4469-22c6-4d99-b8e4-34f417c8661d\") " pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-zxdgg" Oct 31 12:49:32.313700 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.313684 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/796a4469-22c6-4d99-b8e4-34f417c8661d-maas-api-tls\") pod \"maas-api-547b4fbdb9-zxdgg\" (UID: \"796a4469-22c6-4d99-b8e4-34f417c8661d\") " pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-zxdgg" Oct 31 12:49:32.314107 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.313753 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q44vt\" (UniqueName: \"kubernetes.io/projected/b89c8d25-3e85-4fce-ba4d-b53ae261c120-kube-api-access-q44vt\") pod \"authorino-6889cbd8c4-2kvfh\" (UID: \"b89c8d25-3e85-4fce-ba4d-b53ae261c120\") " pod="kuadrant-system/authorino-6889cbd8c4-2kvfh" Oct 31 12:49:32.314107 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.313785 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b89c8d25-3e85-4fce-ba4d-b53ae261c120-tls-cert\") pod \"authorino-6889cbd8c4-2kvfh\" (UID: \"b89c8d25-3e85-4fce-ba4d-b53ae261c120\") " pod="kuadrant-system/authorino-6889cbd8c4-2kvfh" Oct 31 12:49:32.314107 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.313860 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/796a4469-22c6-4d99-b8e4-34f417c8661d-metrics-certs\") pod \"maas-api-547b4fbdb9-zxdgg\" (UID: \"796a4469-22c6-4d99-b8e4-34f417c8661d\") " pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-zxdgg" Oct 31 12:49:32.314107 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.313893 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmq9s\" (UniqueName: \"kubernetes.io/projected/796a4469-22c6-4d99-b8e4-34f417c8661d-kube-api-access-rmq9s\") pod \"maas-api-547b4fbdb9-zxdgg\" (UID: \"796a4469-22c6-4d99-b8e4-34f417c8661d\") " pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-zxdgg" Oct 31 12:49:32.316820 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.316801 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-btnvt" Oct 31 12:49:32.346067 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.346044 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-btnvt"] Oct 31 12:49:32.352261 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.352237 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-btnvt"] Oct 31 12:49:32.415000 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.414963 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/796a4469-22c6-4d99-b8e4-34f417c8661d-openshift-service-ca\") pod \"maas-api-547b4fbdb9-zxdgg\" (UID: \"796a4469-22c6-4d99-b8e4-34f417c8661d\") " pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-zxdgg" Oct 31 12:49:32.415000 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.415002 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/796a4469-22c6-4d99-b8e4-34f417c8661d-maas-api-tls\") pod \"maas-api-547b4fbdb9-zxdgg\" (UID: \"796a4469-22c6-4d99-b8e4-34f417c8661d\") " pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-zxdgg" Oct 31 12:49:32.415237 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.415063 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q44vt\" (UniqueName: \"kubernetes.io/projected/b89c8d25-3e85-4fce-ba4d-b53ae261c120-kube-api-access-q44vt\") pod \"authorino-6889cbd8c4-2kvfh\" (UID: \"b89c8d25-3e85-4fce-ba4d-b53ae261c120\") " pod="kuadrant-system/authorino-6889cbd8c4-2kvfh" Oct 31 12:49:32.415237 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.415115 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b89c8d25-3e85-4fce-ba4d-b53ae261c120-tls-cert\") pod \"authorino-6889cbd8c4-2kvfh\" (UID: \"b89c8d25-3e85-4fce-ba4d-b53ae261c120\") " pod="kuadrant-system/authorino-6889cbd8c4-2kvfh" Oct 31 12:49:32.415237 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.415138 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/796a4469-22c6-4d99-b8e4-34f417c8661d-metrics-certs\") pod \"maas-api-547b4fbdb9-zxdgg\" (UID: \"796a4469-22c6-4d99-b8e4-34f417c8661d\") " pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-zxdgg" Oct 31 12:49:32.415237 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.415153 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmq9s\" (UniqueName: \"kubernetes.io/projected/796a4469-22c6-4d99-b8e4-34f417c8661d-kube-api-access-rmq9s\") pod \"maas-api-547b4fbdb9-zxdgg\" (UID: \"796a4469-22c6-4d99-b8e4-34f417c8661d\") " pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-zxdgg" Oct 31 12:49:32.415955 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.415914 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/796a4469-22c6-4d99-b8e4-34f417c8661d-openshift-service-ca\") pod \"maas-api-547b4fbdb9-zxdgg\" (UID: \"796a4469-22c6-4d99-b8e4-34f417c8661d\") " pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-zxdgg" Oct 31 12:49:32.417789 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.417758 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/796a4469-22c6-4d99-b8e4-34f417c8661d-maas-api-tls\") pod \"maas-api-547b4fbdb9-zxdgg\" (UID: \"796a4469-22c6-4d99-b8e4-34f417c8661d\") " pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-zxdgg" Oct 31 12:49:32.417862 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.417837 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b89c8d25-3e85-4fce-ba4d-b53ae261c120-tls-cert\") pod \"authorino-6889cbd8c4-2kvfh\" (UID: \"b89c8d25-3e85-4fce-ba4d-b53ae261c120\") " pod="kuadrant-system/authorino-6889cbd8c4-2kvfh" Oct 31 12:49:32.417919 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.417905 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/796a4469-22c6-4d99-b8e4-34f417c8661d-metrics-certs\") pod \"maas-api-547b4fbdb9-zxdgg\" (UID: \"796a4469-22c6-4d99-b8e4-34f417c8661d\") " pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-zxdgg" Oct 31 12:49:32.427600 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.427580 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q44vt\" (UniqueName: \"kubernetes.io/projected/b89c8d25-3e85-4fce-ba4d-b53ae261c120-kube-api-access-q44vt\") pod \"authorino-6889cbd8c4-2kvfh\" (UID: \"b89c8d25-3e85-4fce-ba4d-b53ae261c120\") " pod="kuadrant-system/authorino-6889cbd8c4-2kvfh" Oct 31 12:49:32.427819 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.427802 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmq9s\" (UniqueName: \"kubernetes.io/projected/796a4469-22c6-4d99-b8e4-34f417c8661d-kube-api-access-rmq9s\") pod \"maas-api-547b4fbdb9-zxdgg\" (UID: \"796a4469-22c6-4d99-b8e4-34f417c8661d\") " pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-zxdgg" Oct 31 12:49:32.432613 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.432592 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2kvfh"] Oct 31 12:49:32.432823 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.432808 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-2kvfh" Oct 31 12:49:32.461598 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.461575 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-77b4b46987-vxf8x"] Oct 31 12:49:32.484271 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.484245 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77b4b46987-vxf8x" Oct 31 12:49:32.484420 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.484248 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77b4b46987-vxf8x"] Oct 31 12:49:32.515839 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.515814 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bbdd88e1-3077-465d-a515-b8b9f78ca3b7-tls-cert\") pod \"authorino-77b4b46987-vxf8x\" (UID: \"bbdd88e1-3077-465d-a515-b8b9f78ca3b7\") " pod="kuadrant-system/authorino-77b4b46987-vxf8x" Oct 31 12:49:32.515969 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.515854 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfn6z\" (UniqueName: \"kubernetes.io/projected/bbdd88e1-3077-465d-a515-b8b9f78ca3b7-kube-api-access-mfn6z\") pod \"authorino-77b4b46987-vxf8x\" (UID: \"bbdd88e1-3077-465d-a515-b8b9f78ca3b7\") " pod="kuadrant-system/authorino-77b4b46987-vxf8x" Oct 31 12:49:32.594603 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.594521 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-zxdgg" Oct 31 12:49:32.616390 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.616357 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bbdd88e1-3077-465d-a515-b8b9f78ca3b7-tls-cert\") pod \"authorino-77b4b46987-vxf8x\" (UID: \"bbdd88e1-3077-465d-a515-b8b9f78ca3b7\") " pod="kuadrant-system/authorino-77b4b46987-vxf8x" Oct 31 12:49:32.616530 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.616395 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfn6z\" (UniqueName: \"kubernetes.io/projected/bbdd88e1-3077-465d-a515-b8b9f78ca3b7-kube-api-access-mfn6z\") pod \"authorino-77b4b46987-vxf8x\" (UID: \"bbdd88e1-3077-465d-a515-b8b9f78ca3b7\") " pod="kuadrant-system/authorino-77b4b46987-vxf8x" Oct 31 12:49:32.618922 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.618905 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bbdd88e1-3077-465d-a515-b8b9f78ca3b7-tls-cert\") pod \"authorino-77b4b46987-vxf8x\" (UID: \"bbdd88e1-3077-465d-a515-b8b9f78ca3b7\") " pod="kuadrant-system/authorino-77b4b46987-vxf8x" Oct 31 12:49:32.629245 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.629226 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfn6z\" (UniqueName: \"kubernetes.io/projected/bbdd88e1-3077-465d-a515-b8b9f78ca3b7-kube-api-access-mfn6z\") pod \"authorino-77b4b46987-vxf8x\" (UID: \"bbdd88e1-3077-465d-a515-b8b9f78ca3b7\") " pod="kuadrant-system/authorino-77b4b46987-vxf8x" Oct 31 12:49:32.735234 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.735205 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2kvfh"] Oct 31 12:49:32.738164 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:49:32.738138 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb89c8d25_3e85_4fce_ba4d_b53ae261c120.slice/crio-d7bc5b58c20747795f60ed8fe85281b32523455ac355d1e481fffec0a4cb307c WatchSource:0}: Error finding container d7bc5b58c20747795f60ed8fe85281b32523455ac355d1e481fffec0a4cb307c: Status 404 returned error can't find the container with id d7bc5b58c20747795f60ed8fe85281b32523455ac355d1e481fffec0a4cb307c Oct 31 12:49:32.817029 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.817000 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77b4b46987-vxf8x" Oct 31 12:49:32.889083 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:32.889053 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-547b4fbdb9-zxdgg"] Oct 31 12:49:32.897510 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:49:32.897483 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod796a4469_22c6_4d99_b8e4_34f417c8661d.slice/crio-04738d56d09d925a4bc4a1cd8d62b0136eca8c6114357b97ca4ae561df696d5a WatchSource:0}: Error finding container 04738d56d09d925a4bc4a1cd8d62b0136eca8c6114357b97ca4ae561df696d5a: Status 404 returned error can't find the container with id 04738d56d09d925a4bc4a1cd8d62b0136eca8c6114357b97ca4ae561df696d5a Oct 31 12:49:33.108516 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:33.108382 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77b4b46987-vxf8x"] Oct 31 12:49:33.110985 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:49:33.110957 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbbdd88e1_3077_465d_a515_b8b9f78ca3b7.slice/crio-ab26f132cba1a80283b8b426eacdf894a46eea59322d5a5b89b1dbdf5f8eb3b4 WatchSource:0}: Error finding container ab26f132cba1a80283b8b426eacdf894a46eea59322d5a5b89b1dbdf5f8eb3b4: Status 404 returned error can't find the container with id ab26f132cba1a80283b8b426eacdf894a46eea59322d5a5b89b1dbdf5f8eb3b4 Oct 31 12:49:33.322122 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:33.322076 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-zxdgg" event={"ID":"796a4469-22c6-4d99-b8e4-34f417c8661d","Type":"ContainerStarted","Data":"04738d56d09d925a4bc4a1cd8d62b0136eca8c6114357b97ca4ae561df696d5a"} Oct 31 12:49:33.323285 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:33.323258 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77b4b46987-vxf8x" event={"ID":"bbdd88e1-3077-465d-a515-b8b9f78ca3b7","Type":"ContainerStarted","Data":"ab26f132cba1a80283b8b426eacdf894a46eea59322d5a5b89b1dbdf5f8eb3b4"} Oct 31 12:49:33.324335 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:33.324315 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-2kvfh" event={"ID":"b89c8d25-3e85-4fce-ba4d-b53ae261c120","Type":"ContainerStarted","Data":"d7bc5b58c20747795f60ed8fe85281b32523455ac355d1e481fffec0a4cb307c"} Oct 31 12:49:34.333730 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:34.333255 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-zxdgg" event={"ID":"796a4469-22c6-4d99-b8e4-34f417c8661d","Type":"ContainerStarted","Data":"dfe7a513520cc608018c50c5998734f71a6c2f738cfa1ccb9c436d3498cebe5d"} Oct 31 12:49:34.333730 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:34.333557 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-zxdgg" Oct 31 12:49:34.335047 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:34.335003 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77b4b46987-vxf8x" event={"ID":"bbdd88e1-3077-465d-a515-b8b9f78ca3b7","Type":"ContainerStarted","Data":"8155e64362b068f86866b19180d8def2d6f3bf40fe7a9bd4e18d70c6f1d71dae"} Oct 31 12:49:34.348081 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:34.348048 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-2kvfh" event={"ID":"b89c8d25-3e85-4fce-ba4d-b53ae261c120","Type":"ContainerStarted","Data":"03510b36dfee4eacea682dd2d2e11aee3827afdd276442ffeffcef782d04d780"} Oct 31 12:49:34.348251 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:34.348209 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-2kvfh" podUID="b89c8d25-3e85-4fce-ba4d-b53ae261c120" containerName="authorino" containerID="cri-o://03510b36dfee4eacea682dd2d2e11aee3827afdd276442ffeffcef782d04d780" gracePeriod=30 Oct 31 12:49:34.379060 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:34.378880 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-zxdgg" podStartSLOduration=1.8788090849999999 podStartE2EDuration="2.378859557s" totalImagesPullingTime="500.050472ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:49:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:49:34.360515672 +0000 UTC m=+1025.068101969" watchObservedRunningTime="2026-10-31 12:49:34.378859557 +0000 UTC m=+1025.086445853" Oct 31 12:49:34.379060 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:34.379004 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-77b4b46987-vxf8x" podStartSLOduration=1.5669791069999999 podStartE2EDuration="2.37899672s" totalImagesPullingTime="812.017613ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:49:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:49:34.378332627 +0000 UTC m=+1025.085918919" watchObservedRunningTime="2026-10-31 12:49:34.37899672 +0000 UTC m=+1025.086583016" Oct 31 12:49:34.407975 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:34.407133 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6889cbd8c4-2kvfh" podStartSLOduration=1.5445720920000001 podStartE2EDuration="2.407111179s" totalImagesPullingTime="862.539087ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:49:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:49:34.406066702 +0000 UTC m=+1025.113652998" watchObservedRunningTime="2026-10-31 12:49:34.407111179 +0000 UTC m=+1025.114697476" Oct 31 12:49:34.412624 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:34.411979 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-64lq2"] Oct 31 12:49:34.412624 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:34.412163 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-64lq2" podUID="a6c68f41-432b-425e-8016-76908847a2f9" containerName="authorino" containerID="cri-o://d7e836a1680d75771a66023beabc5a716900f5458fb82f4dcd92187212726320" gracePeriod=30 Oct 31 12:49:34.724740 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:34.724712 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-2kvfh" Oct 31 12:49:34.737439 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:34.737413 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-64lq2" Oct 31 12:49:34.835296 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:34.835258 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b89c8d25-3e85-4fce-ba4d-b53ae261c120-tls-cert\" (UniqueName: \"kubernetes.io/secret/b89c8d25-3e85-4fce-ba4d-b53ae261c120-tls-cert\") pod \"b89c8d25-3e85-4fce-ba4d-b53ae261c120\" (UID: \"b89c8d25-3e85-4fce-ba4d-b53ae261c120\") " Oct 31 12:49:34.835296 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:34.835300 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b89c8d25-3e85-4fce-ba4d-b53ae261c120-kube-api-access-q44vt\" (UniqueName: \"kubernetes.io/projected/b89c8d25-3e85-4fce-ba4d-b53ae261c120-kube-api-access-q44vt\") pod \"b89c8d25-3e85-4fce-ba4d-b53ae261c120\" (UID: \"b89c8d25-3e85-4fce-ba4d-b53ae261c120\") " Oct 31 12:49:34.835482 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:34.835338 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6c68f41-432b-425e-8016-76908847a2f9-kube-api-access-nfmbx\" (UniqueName: \"kubernetes.io/projected/a6c68f41-432b-425e-8016-76908847a2f9-kube-api-access-nfmbx\") pod \"a6c68f41-432b-425e-8016-76908847a2f9\" (UID: \"a6c68f41-432b-425e-8016-76908847a2f9\") " Oct 31 12:49:34.837461 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:34.837419 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b89c8d25-3e85-4fce-ba4d-b53ae261c120-kube-api-access-q44vt" pod "b89c8d25-3e85-4fce-ba4d-b53ae261c120" (UID: "b89c8d25-3e85-4fce-ba4d-b53ae261c120"). InnerVolumeSpecName "kube-api-access-q44vt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:49:34.837684 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:34.837663 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6c68f41-432b-425e-8016-76908847a2f9-kube-api-access-nfmbx" pod "a6c68f41-432b-425e-8016-76908847a2f9" (UID: "a6c68f41-432b-425e-8016-76908847a2f9"). InnerVolumeSpecName "kube-api-access-nfmbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:49:34.845527 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:34.845472 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b89c8d25-3e85-4fce-ba4d-b53ae261c120-tls-cert" pod "b89c8d25-3e85-4fce-ba4d-b53ae261c120" (UID: "b89c8d25-3e85-4fce-ba4d-b53ae261c120"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:49:34.935862 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:34.935828 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfmbx\" (UniqueName: \"kubernetes.io/projected/a6c68f41-432b-425e-8016-76908847a2f9-kube-api-access-nfmbx\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:49:34.935862 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:34.935855 5693 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b89c8d25-3e85-4fce-ba4d-b53ae261c120-tls-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:49:34.935862 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:34.935864 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q44vt\" (UniqueName: \"kubernetes.io/projected/b89c8d25-3e85-4fce-ba4d-b53ae261c120-kube-api-access-q44vt\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:49:35.354053 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:35.354014 5693 generic.go:356] "Generic (PLEG): container finished" podID="b89c8d25-3e85-4fce-ba4d-b53ae261c120" containerID="03510b36dfee4eacea682dd2d2e11aee3827afdd276442ffeffcef782d04d780" exitCode=0 Oct 31 12:49:35.354595 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:35.354067 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-2kvfh" Oct 31 12:49:35.354595 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:35.354108 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-2kvfh" event={"ID":"b89c8d25-3e85-4fce-ba4d-b53ae261c120","Type":"ContainerDied","Data":"03510b36dfee4eacea682dd2d2e11aee3827afdd276442ffeffcef782d04d780"} Oct 31 12:49:35.354595 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:35.354153 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-2kvfh" event={"ID":"b89c8d25-3e85-4fce-ba4d-b53ae261c120","Type":"ContainerDied","Data":"d7bc5b58c20747795f60ed8fe85281b32523455ac355d1e481fffec0a4cb307c"} Oct 31 12:49:35.354595 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:35.354175 5693 scope.go:122] "RemoveContainer" containerID="03510b36dfee4eacea682dd2d2e11aee3827afdd276442ffeffcef782d04d780" Oct 31 12:49:35.355477 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:35.355452 5693 generic.go:356] "Generic (PLEG): container finished" podID="a6c68f41-432b-425e-8016-76908847a2f9" containerID="d7e836a1680d75771a66023beabc5a716900f5458fb82f4dcd92187212726320" exitCode=0 Oct 31 12:49:35.355602 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:35.355496 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-64lq2" Oct 31 12:49:35.355602 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:35.355533 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-64lq2" event={"ID":"a6c68f41-432b-425e-8016-76908847a2f9","Type":"ContainerDied","Data":"d7e836a1680d75771a66023beabc5a716900f5458fb82f4dcd92187212726320"} Oct 31 12:49:35.355602 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:35.355565 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-64lq2" event={"ID":"a6c68f41-432b-425e-8016-76908847a2f9","Type":"ContainerDied","Data":"29a03f423dcbb6258aa193c5aae22ad442609ded58933f0a02d83581db533c1f"} Oct 31 12:49:35.364022 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:35.364003 5693 scope.go:122] "RemoveContainer" containerID="03510b36dfee4eacea682dd2d2e11aee3827afdd276442ffeffcef782d04d780" Oct 31 12:49:35.364309 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:49:35.364291 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03510b36dfee4eacea682dd2d2e11aee3827afdd276442ffeffcef782d04d780\": container with ID starting with 03510b36dfee4eacea682dd2d2e11aee3827afdd276442ffeffcef782d04d780 not found: ID does not exist" containerID="03510b36dfee4eacea682dd2d2e11aee3827afdd276442ffeffcef782d04d780" Oct 31 12:49:35.364380 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:35.364316 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03510b36dfee4eacea682dd2d2e11aee3827afdd276442ffeffcef782d04d780"} err="failed to get container status \"03510b36dfee4eacea682dd2d2e11aee3827afdd276442ffeffcef782d04d780\": rpc error: code = NotFound desc = could not find container \"03510b36dfee4eacea682dd2d2e11aee3827afdd276442ffeffcef782d04d780\": container with ID starting with 03510b36dfee4eacea682dd2d2e11aee3827afdd276442ffeffcef782d04d780 not found: ID does not exist" Oct 31 12:49:35.364380 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:35.364334 5693 scope.go:122] "RemoveContainer" containerID="d7e836a1680d75771a66023beabc5a716900f5458fb82f4dcd92187212726320" Oct 31 12:49:35.372043 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:35.372026 5693 scope.go:122] "RemoveContainer" containerID="d7e836a1680d75771a66023beabc5a716900f5458fb82f4dcd92187212726320" Oct 31 12:49:35.372309 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:49:35.372293 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7e836a1680d75771a66023beabc5a716900f5458fb82f4dcd92187212726320\": container with ID starting with d7e836a1680d75771a66023beabc5a716900f5458fb82f4dcd92187212726320 not found: ID does not exist" containerID="d7e836a1680d75771a66023beabc5a716900f5458fb82f4dcd92187212726320" Oct 31 12:49:35.372347 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:35.372317 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7e836a1680d75771a66023beabc5a716900f5458fb82f4dcd92187212726320"} err="failed to get container status \"d7e836a1680d75771a66023beabc5a716900f5458fb82f4dcd92187212726320\": rpc error: code = NotFound desc = could not find container \"d7e836a1680d75771a66023beabc5a716900f5458fb82f4dcd92187212726320\": container with ID starting with d7e836a1680d75771a66023beabc5a716900f5458fb82f4dcd92187212726320 not found: ID does not exist" Oct 31 12:49:35.379795 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:35.379766 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2kvfh"] Oct 31 12:49:35.381863 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:35.381843 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2kvfh"] Oct 31 12:49:35.397912 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:35.397889 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-64lq2"] Oct 31 12:49:35.406155 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:35.406134 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b89c8d25-3e85-4fce-ba4d-b53ae261c120" path="/var/lib/kubelet/pods/b89c8d25-3e85-4fce-ba4d-b53ae261c120/volumes" Oct 31 12:49:35.406392 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:35.406380 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-64lq2"] Oct 31 12:49:37.405224 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:37.405194 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6c68f41-432b-425e-8016-76908847a2f9" path="/var/lib/kubelet/pods/a6c68f41-432b-425e-8016-76908847a2f9/volumes" Oct 31 12:49:40.361328 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:40.361297 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-zxdgg" Oct 31 12:49:40.411497 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:40.411467 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c54796cf8-7bkc9"] Oct 31 12:49:40.411702 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:40.411667 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7c54796cf8-7bkc9" podUID="c76fb2a1-f89a-4614-b1ac-2cea628ce3bb" containerName="maas-api" containerID="cri-o://f1468b751451b34f4deea7d7eb707714ee765f8f43c03c197ea1ef5b1fc87809" gracePeriod=30 Oct 31 12:49:40.645519 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:40.645496 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c54796cf8-7bkc9" Oct 31 12:49:40.684954 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:40.684921 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c76fb2a1-f89a-4614-b1ac-2cea628ce3bb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c76fb2a1-f89a-4614-b1ac-2cea628ce3bb-openshift-service-ca\") pod \"c76fb2a1-f89a-4614-b1ac-2cea628ce3bb\" (UID: \"c76fb2a1-f89a-4614-b1ac-2cea628ce3bb\") " Oct 31 12:49:40.685125 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:40.685022 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c76fb2a1-f89a-4614-b1ac-2cea628ce3bb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c76fb2a1-f89a-4614-b1ac-2cea628ce3bb-maas-api-tls\") pod \"c76fb2a1-f89a-4614-b1ac-2cea628ce3bb\" (UID: \"c76fb2a1-f89a-4614-b1ac-2cea628ce3bb\") " Oct 31 12:49:40.685125 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:40.685067 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c76fb2a1-f89a-4614-b1ac-2cea628ce3bb-kube-api-access-xkl7d\" (UniqueName: \"kubernetes.io/projected/c76fb2a1-f89a-4614-b1ac-2cea628ce3bb-kube-api-access-xkl7d\") pod \"c76fb2a1-f89a-4614-b1ac-2cea628ce3bb\" (UID: \"c76fb2a1-f89a-4614-b1ac-2cea628ce3bb\") " Oct 31 12:49:40.685125 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:40.685094 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c76fb2a1-f89a-4614-b1ac-2cea628ce3bb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c76fb2a1-f89a-4614-b1ac-2cea628ce3bb-metrics-certs\") pod \"c76fb2a1-f89a-4614-b1ac-2cea628ce3bb\" (UID: \"c76fb2a1-f89a-4614-b1ac-2cea628ce3bb\") " Oct 31 12:49:40.685304 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:40.685284 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c76fb2a1-f89a-4614-b1ac-2cea628ce3bb-openshift-service-ca" pod "c76fb2a1-f89a-4614-b1ac-2cea628ce3bb" (UID: "c76fb2a1-f89a-4614-b1ac-2cea628ce3bb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:49:40.685360 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:40.685347 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c76fb2a1-f89a-4614-b1ac-2cea628ce3bb-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:49:40.687470 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:40.687440 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c76fb2a1-f89a-4614-b1ac-2cea628ce3bb-kube-api-access-xkl7d" pod "c76fb2a1-f89a-4614-b1ac-2cea628ce3bb" (UID: "c76fb2a1-f89a-4614-b1ac-2cea628ce3bb"). InnerVolumeSpecName "kube-api-access-xkl7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:49:40.687576 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:40.687467 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c76fb2a1-f89a-4614-b1ac-2cea628ce3bb-metrics-certs" pod "c76fb2a1-f89a-4614-b1ac-2cea628ce3bb" (UID: "c76fb2a1-f89a-4614-b1ac-2cea628ce3bb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:49:40.687576 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:40.687492 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c76fb2a1-f89a-4614-b1ac-2cea628ce3bb-maas-api-tls" pod "c76fb2a1-f89a-4614-b1ac-2cea628ce3bb" (UID: "c76fb2a1-f89a-4614-b1ac-2cea628ce3bb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:49:40.786678 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:40.786626 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c76fb2a1-f89a-4614-b1ac-2cea628ce3bb-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:49:40.786678 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:40.786675 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkl7d\" (UniqueName: \"kubernetes.io/projected/c76fb2a1-f89a-4614-b1ac-2cea628ce3bb-kube-api-access-xkl7d\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:49:40.786860 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:40.786692 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c76fb2a1-f89a-4614-b1ac-2cea628ce3bb-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:49:41.381016 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:41.380980 5693 generic.go:356] "Generic (PLEG): container finished" podID="c76fb2a1-f89a-4614-b1ac-2cea628ce3bb" containerID="f1468b751451b34f4deea7d7eb707714ee765f8f43c03c197ea1ef5b1fc87809" exitCode=0 Oct 31 12:49:41.381449 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:41.381023 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c54796cf8-7bkc9" event={"ID":"c76fb2a1-f89a-4614-b1ac-2cea628ce3bb","Type":"ContainerDied","Data":"f1468b751451b34f4deea7d7eb707714ee765f8f43c03c197ea1ef5b1fc87809"} Oct 31 12:49:41.381449 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:41.381060 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c54796cf8-7bkc9" event={"ID":"c76fb2a1-f89a-4614-b1ac-2cea628ce3bb","Type":"ContainerDied","Data":"61bfce49aee9e8efcd827e539d85cf9b3d88b4d3e4bb486dc562b6aa08312d53"} Oct 31 12:49:41.381449 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:41.381080 5693 scope.go:122] "RemoveContainer" containerID="f1468b751451b34f4deea7d7eb707714ee765f8f43c03c197ea1ef5b1fc87809" Oct 31 12:49:41.381449 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:41.381023 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c54796cf8-7bkc9" Oct 31 12:49:41.390460 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:41.390445 5693 scope.go:122] "RemoveContainer" containerID="f1468b751451b34f4deea7d7eb707714ee765f8f43c03c197ea1ef5b1fc87809" Oct 31 12:49:41.390764 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:49:41.390745 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1468b751451b34f4deea7d7eb707714ee765f8f43c03c197ea1ef5b1fc87809\": container with ID starting with f1468b751451b34f4deea7d7eb707714ee765f8f43c03c197ea1ef5b1fc87809 not found: ID does not exist" containerID="f1468b751451b34f4deea7d7eb707714ee765f8f43c03c197ea1ef5b1fc87809" Oct 31 12:49:41.390835 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:41.390773 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1468b751451b34f4deea7d7eb707714ee765f8f43c03c197ea1ef5b1fc87809"} err="failed to get container status \"f1468b751451b34f4deea7d7eb707714ee765f8f43c03c197ea1ef5b1fc87809\": rpc error: code = NotFound desc = could not find container \"f1468b751451b34f4deea7d7eb707714ee765f8f43c03c197ea1ef5b1fc87809\": container with ID starting with f1468b751451b34f4deea7d7eb707714ee765f8f43c03c197ea1ef5b1fc87809 not found: ID does not exist" Oct 31 12:49:41.405017 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:41.404990 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c54796cf8-7bkc9"] Oct 31 12:49:41.408334 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:41.408311 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c54796cf8-7bkc9"] Oct 31 12:49:43.405902 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:49:43.405872 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c76fb2a1-f89a-4614-b1ac-2cea628ce3bb" path="/var/lib/kubelet/pods/c76fb2a1-f89a-4614-b1ac-2cea628ce3bb/volumes" Oct 31 12:49:48.685913 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:49:48.685873 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:50:09.129198 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:09.129167 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-8cbkh"] Oct 31 12:50:09.129739 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:09.129496 5693 cpu_manager.go:414] "containerMap: removing container" podUID="c76fb2a1-f89a-4614-b1ac-2cea628ce3bb" containerName="maas-api" Oct 31 12:50:09.129739 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:09.129507 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c76fb2a1-f89a-4614-b1ac-2cea628ce3bb" containerName="maas-api" Oct 31 12:50:09.129739 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:09.129516 5693 cpu_manager.go:414] "containerMap: removing container" podUID="b89c8d25-3e85-4fce-ba4d-b53ae261c120" containerName="authorino" Oct 31 12:50:09.129739 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:09.129520 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b89c8d25-3e85-4fce-ba4d-b53ae261c120" containerName="authorino" Oct 31 12:50:09.129739 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:09.129530 5693 cpu_manager.go:414] "containerMap: removing container" podUID="a6c68f41-432b-425e-8016-76908847a2f9" containerName="authorino" Oct 31 12:50:09.129739 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:09.129535 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6c68f41-432b-425e-8016-76908847a2f9" containerName="authorino" Oct 31 12:50:09.129739 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:09.129583 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="c76fb2a1-f89a-4614-b1ac-2cea628ce3bb" containerName="maas-api" Oct 31 12:50:09.129739 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:09.129591 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="b89c8d25-3e85-4fce-ba4d-b53ae261c120" containerName="authorino" Oct 31 12:50:09.129739 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:09.129597 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6c68f41-432b-425e-8016-76908847a2f9" containerName="authorino" Oct 31 12:50:09.133760 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:09.133740 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-8cbkh" Oct 31 12:50:09.137556 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:09.137530 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 31 12:50:09.138754 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:09.138738 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 31 12:50:09.138822 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:09.138752 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-gbgr7\"" Oct 31 12:50:09.141965 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:09.141939 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-8cbkh"] Oct 31 12:50:09.233415 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:09.233373 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d776d943-63d2-4b1c-a8d1-2e018aa16a01-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-8cbkh\" (UID: \"d776d943-63d2-4b1c-a8d1-2e018aa16a01\") " pod="opendatahub/ai-gateway-controller-7d85675f98-8cbkh" Oct 31 12:50:09.233580 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:09.233520 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdh9w\" (UniqueName: \"kubernetes.io/projected/d776d943-63d2-4b1c-a8d1-2e018aa16a01-kube-api-access-xdh9w\") pod \"ai-gateway-controller-7d85675f98-8cbkh\" (UID: \"d776d943-63d2-4b1c-a8d1-2e018aa16a01\") " pod="opendatahub/ai-gateway-controller-7d85675f98-8cbkh" Oct 31 12:50:09.334077 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:09.334043 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdh9w\" (UniqueName: \"kubernetes.io/projected/d776d943-63d2-4b1c-a8d1-2e018aa16a01-kube-api-access-xdh9w\") pod \"ai-gateway-controller-7d85675f98-8cbkh\" (UID: \"d776d943-63d2-4b1c-a8d1-2e018aa16a01\") " pod="opendatahub/ai-gateway-controller-7d85675f98-8cbkh" Oct 31 12:50:09.334250 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:09.334087 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d776d943-63d2-4b1c-a8d1-2e018aa16a01-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-8cbkh\" (UID: \"d776d943-63d2-4b1c-a8d1-2e018aa16a01\") " pod="opendatahub/ai-gateway-controller-7d85675f98-8cbkh" Oct 31 12:50:09.336747 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:09.336717 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d776d943-63d2-4b1c-a8d1-2e018aa16a01-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-8cbkh\" (UID: \"d776d943-63d2-4b1c-a8d1-2e018aa16a01\") " pod="opendatahub/ai-gateway-controller-7d85675f98-8cbkh" Oct 31 12:50:09.347119 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:09.347097 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdh9w\" (UniqueName: \"kubernetes.io/projected/d776d943-63d2-4b1c-a8d1-2e018aa16a01-kube-api-access-xdh9w\") pod \"ai-gateway-controller-7d85675f98-8cbkh\" (UID: \"d776d943-63d2-4b1c-a8d1-2e018aa16a01\") " pod="opendatahub/ai-gateway-controller-7d85675f98-8cbkh" Oct 31 12:50:09.446485 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:09.446451 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-8cbkh" Oct 31 12:50:09.719845 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:09.719754 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-8cbkh"] Oct 31 12:50:09.723075 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:50:09.723050 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd776d943_63d2_4b1c_a8d1_2e018aa16a01.slice/crio-c665de7f414a0607cfab430b0e0b414e3978f72c4d13d1a60c422417dc0c4506 WatchSource:0}: Error finding container c665de7f414a0607cfab430b0e0b414e3978f72c4d13d1a60c422417dc0c4506: Status 404 returned error can't find the container with id c665de7f414a0607cfab430b0e0b414e3978f72c4d13d1a60c422417dc0c4506 Oct 31 12:50:10.500634 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:10.500598 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-8cbkh" event={"ID":"d776d943-63d2-4b1c-a8d1-2e018aa16a01","Type":"ContainerStarted","Data":"c665de7f414a0607cfab430b0e0b414e3978f72c4d13d1a60c422417dc0c4506"} Oct 31 12:50:12.509577 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:12.509542 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-8cbkh" event={"ID":"d776d943-63d2-4b1c-a8d1-2e018aa16a01","Type":"ContainerStarted","Data":"f4df7f7b85d7e51993d3937351751825f786901b6313be6692a184ee118722f2"} Oct 31 12:50:12.509979 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:12.509685 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-8cbkh" Oct 31 12:50:12.528967 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:12.528921 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-8cbkh" podStartSLOduration=1.622680452 podStartE2EDuration="3.52890847s" totalImagesPullingTime="1.906228018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:50:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:50:12.527559009 +0000 UTC m=+1063.235145305" watchObservedRunningTime="2026-10-31 12:50:12.52890847 +0000 UTC m=+1063.236494765" Oct 31 12:50:13.448711 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:13.448675 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-w7mmf"] Oct 31 12:50:13.451996 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:13.451981 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-w7mmf" Oct 31 12:50:13.460014 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:13.459991 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-z7fks\"" Oct 31 12:50:13.462319 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:13.462290 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-w7mmf"] Oct 31 12:50:13.464733 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:13.464713 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 31 12:50:13.465001 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:13.464973 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tdhw\" (UniqueName: \"kubernetes.io/projected/6485629e-cfde-468c-93b0-9f68ca0ded66-kube-api-access-7tdhw\") pod \"payload-processing-55fbbbf9dc-w7mmf\" (UID: \"6485629e-cfde-468c-93b0-9f68ca0ded66\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-w7mmf" Oct 31 12:50:13.465168 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:13.465149 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6485629e-cfde-468c-93b0-9f68ca0ded66-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-w7mmf\" (UID: \"6485629e-cfde-468c-93b0-9f68ca0ded66\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-w7mmf" Oct 31 12:50:13.482271 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:13.482242 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-n4pg9"] Oct 31 12:50:13.486048 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:13.486030 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n4pg9" Oct 31 12:50:13.495461 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:13.495430 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-n4pg9"] Oct 31 12:50:13.566378 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:13.566345 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6485629e-cfde-468c-93b0-9f68ca0ded66-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-w7mmf\" (UID: \"6485629e-cfde-468c-93b0-9f68ca0ded66\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-w7mmf" Oct 31 12:50:13.566893 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:13.566410 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tdhw\" (UniqueName: \"kubernetes.io/projected/6485629e-cfde-468c-93b0-9f68ca0ded66-kube-api-access-7tdhw\") pod \"payload-processing-55fbbbf9dc-w7mmf\" (UID: \"6485629e-cfde-468c-93b0-9f68ca0ded66\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-w7mmf" Oct 31 12:50:13.566893 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:13.566483 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q8z6\" (UniqueName: \"kubernetes.io/projected/3d3532b0-6567-4331-a327-a7b917fceeec-kube-api-access-4q8z6\") pod \"payload-pre-processing-7b98bb678c-n4pg9\" (UID: \"3d3532b0-6567-4331-a327-a7b917fceeec\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-n4pg9" Oct 31 12:50:13.566893 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:13.566604 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d3532b0-6567-4331-a327-a7b917fceeec-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-n4pg9\" (UID: \"3d3532b0-6567-4331-a327-a7b917fceeec\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-n4pg9" Oct 31 12:50:13.567178 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:13.567161 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6485629e-cfde-468c-93b0-9f68ca0ded66-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-w7mmf\" (UID: \"6485629e-cfde-468c-93b0-9f68ca0ded66\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-w7mmf" Oct 31 12:50:13.578401 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:13.578374 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tdhw\" (UniqueName: \"kubernetes.io/projected/6485629e-cfde-468c-93b0-9f68ca0ded66-kube-api-access-7tdhw\") pod \"payload-processing-55fbbbf9dc-w7mmf\" (UID: \"6485629e-cfde-468c-93b0-9f68ca0ded66\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-w7mmf" Oct 31 12:50:13.667814 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:13.667767 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q8z6\" (UniqueName: \"kubernetes.io/projected/3d3532b0-6567-4331-a327-a7b917fceeec-kube-api-access-4q8z6\") pod \"payload-pre-processing-7b98bb678c-n4pg9\" (UID: \"3d3532b0-6567-4331-a327-a7b917fceeec\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-n4pg9" Oct 31 12:50:13.667979 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:13.667820 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d3532b0-6567-4331-a327-a7b917fceeec-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-n4pg9\" (UID: \"3d3532b0-6567-4331-a327-a7b917fceeec\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-n4pg9" Oct 31 12:50:13.668486 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:13.668466 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d3532b0-6567-4331-a327-a7b917fceeec-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-n4pg9\" (UID: \"3d3532b0-6567-4331-a327-a7b917fceeec\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-n4pg9" Oct 31 12:50:13.680757 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:13.680725 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q8z6\" (UniqueName: \"kubernetes.io/projected/3d3532b0-6567-4331-a327-a7b917fceeec-kube-api-access-4q8z6\") pod \"payload-pre-processing-7b98bb678c-n4pg9\" (UID: \"3d3532b0-6567-4331-a327-a7b917fceeec\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-n4pg9" Oct 31 12:50:13.764622 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:13.764546 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-w7mmf" Oct 31 12:50:13.801722 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:13.801683 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n4pg9" Oct 31 12:50:14.040097 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:14.040011 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-w7mmf"] Oct 31 12:50:14.043995 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:50:14.043971 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6485629e_cfde_468c_93b0_9f68ca0ded66.slice/crio-69907cde19ec9a563dcf05df485ea39dd0a6ea2595b1ab049f6fe41bb56e445b WatchSource:0}: Error finding container 69907cde19ec9a563dcf05df485ea39dd0a6ea2595b1ab049f6fe41bb56e445b: Status 404 returned error can't find the container with id 69907cde19ec9a563dcf05df485ea39dd0a6ea2595b1ab049f6fe41bb56e445b Oct 31 12:50:14.086601 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:14.086566 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-n4pg9"] Oct 31 12:50:14.090706 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:50:14.090674 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d3532b0_6567_4331_a327_a7b917fceeec.slice/crio-bff0c6a0fb36d9d7e6f2b3073fcdbf57a94c7c592632466e311fdffa771bb406 WatchSource:0}: Error finding container bff0c6a0fb36d9d7e6f2b3073fcdbf57a94c7c592632466e311fdffa771bb406: Status 404 returned error can't find the container with id bff0c6a0fb36d9d7e6f2b3073fcdbf57a94c7c592632466e311fdffa771bb406 Oct 31 12:50:14.519894 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:14.519855 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n4pg9" event={"ID":"3d3532b0-6567-4331-a327-a7b917fceeec","Type":"ContainerStarted","Data":"bff0c6a0fb36d9d7e6f2b3073fcdbf57a94c7c592632466e311fdffa771bb406"} Oct 31 12:50:14.521009 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:14.520986 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-w7mmf" event={"ID":"6485629e-cfde-468c-93b0-9f68ca0ded66","Type":"ContainerStarted","Data":"69907cde19ec9a563dcf05df485ea39dd0a6ea2595b1ab049f6fe41bb56e445b"} Oct 31 12:50:16.530909 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:16.530882 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n4pg9" event={"ID":"3d3532b0-6567-4331-a327-a7b917fceeec","Type":"ContainerStarted","Data":"92da70c80bf808ad69af71268efcc43f46f0cde4c1ae0fbe23677134cf6e2823"} Oct 31 12:50:16.531254 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:16.530991 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n4pg9" Oct 31 12:50:16.532361 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:16.532340 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-w7mmf" event={"ID":"6485629e-cfde-468c-93b0-9f68ca0ded66","Type":"ContainerStarted","Data":"dbceae1e79dcb72382b653c4cb4a44bb4f272da55ae75e0678ec7551bdcbc2e7"} Oct 31 12:50:16.532453 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:16.532416 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-w7mmf" Oct 31 12:50:16.549137 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:16.549063 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n4pg9" podStartSLOduration=1.189888062 podStartE2EDuration="3.549045427s" totalImagesPullingTime="2.359157365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:50:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:50:16.548753202 +0000 UTC m=+1067.256339497" watchObservedRunningTime="2026-10-31 12:50:16.549045427 +0000 UTC m=+1067.256631721" Oct 31 12:50:16.566990 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:16.566954 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-w7mmf" podStartSLOduration=1.236372024 podStartE2EDuration="3.566938659s" totalImagesPullingTime="2.330566635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:50:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:50:16.56594588 +0000 UTC m=+1067.273532176" watchObservedRunningTime="2026-10-31 12:50:16.566938659 +0000 UTC m=+1067.274524966" Oct 31 12:50:18.685699 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:50:18.685666 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:50:23.516543 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:23.516515 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-8cbkh" Oct 31 12:50:23.781980 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:23.781906 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6df7b9f889-66dn6"] Oct 31 12:50:23.785596 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:23.785574 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6df7b9f889-66dn6" Oct 31 12:50:23.796012 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:23.795987 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6df7b9f889-66dn6"] Oct 31 12:50:23.857969 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:23.857932 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c3a62d1-ddd7-4459-8342-2ed0eedd3d8d-metrics-certs\") pod \"ai-gateway-controller-6df7b9f889-66dn6\" (UID: \"5c3a62d1-ddd7-4459-8342-2ed0eedd3d8d\") " pod="opendatahub/ai-gateway-controller-6df7b9f889-66dn6" Oct 31 12:50:23.858115 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:23.857995 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h9kf\" (UniqueName: \"kubernetes.io/projected/5c3a62d1-ddd7-4459-8342-2ed0eedd3d8d-kube-api-access-7h9kf\") pod \"ai-gateway-controller-6df7b9f889-66dn6\" (UID: \"5c3a62d1-ddd7-4459-8342-2ed0eedd3d8d\") " pod="opendatahub/ai-gateway-controller-6df7b9f889-66dn6" Oct 31 12:50:23.959030 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:23.958996 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h9kf\" (UniqueName: \"kubernetes.io/projected/5c3a62d1-ddd7-4459-8342-2ed0eedd3d8d-kube-api-access-7h9kf\") pod \"ai-gateway-controller-6df7b9f889-66dn6\" (UID: \"5c3a62d1-ddd7-4459-8342-2ed0eedd3d8d\") " pod="opendatahub/ai-gateway-controller-6df7b9f889-66dn6" Oct 31 12:50:23.959190 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:23.959066 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c3a62d1-ddd7-4459-8342-2ed0eedd3d8d-metrics-certs\") pod \"ai-gateway-controller-6df7b9f889-66dn6\" (UID: \"5c3a62d1-ddd7-4459-8342-2ed0eedd3d8d\") " pod="opendatahub/ai-gateway-controller-6df7b9f889-66dn6" Oct 31 12:50:23.961571 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:23.961549 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c3a62d1-ddd7-4459-8342-2ed0eedd3d8d-metrics-certs\") pod \"ai-gateway-controller-6df7b9f889-66dn6\" (UID: \"5c3a62d1-ddd7-4459-8342-2ed0eedd3d8d\") " pod="opendatahub/ai-gateway-controller-6df7b9f889-66dn6" Oct 31 12:50:23.971464 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:23.971442 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h9kf\" (UniqueName: \"kubernetes.io/projected/5c3a62d1-ddd7-4459-8342-2ed0eedd3d8d-kube-api-access-7h9kf\") pod \"ai-gateway-controller-6df7b9f889-66dn6\" (UID: \"5c3a62d1-ddd7-4459-8342-2ed0eedd3d8d\") " pod="opendatahub/ai-gateway-controller-6df7b9f889-66dn6" Oct 31 12:50:24.098324 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:24.098253 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6df7b9f889-66dn6" Oct 31 12:50:24.370935 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:24.370853 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6df7b9f889-66dn6"] Oct 31 12:50:24.373954 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:50:24.373925 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c3a62d1_ddd7_4459_8342_2ed0eedd3d8d.slice/crio-30c677ad069c6da1443899ce9a999830dc8cc68aa2929de58be58b98db40d124 WatchSource:0}: Error finding container 30c677ad069c6da1443899ce9a999830dc8cc68aa2929de58be58b98db40d124: Status 404 returned error can't find the container with id 30c677ad069c6da1443899ce9a999830dc8cc68aa2929de58be58b98db40d124 Oct 31 12:50:24.565465 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:24.565431 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6df7b9f889-66dn6" event={"ID":"5c3a62d1-ddd7-4459-8342-2ed0eedd3d8d","Type":"ContainerStarted","Data":"30c677ad069c6da1443899ce9a999830dc8cc68aa2929de58be58b98db40d124"} Oct 31 12:50:25.571943 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:25.571905 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6df7b9f889-66dn6" event={"ID":"5c3a62d1-ddd7-4459-8342-2ed0eedd3d8d","Type":"ContainerStarted","Data":"37c45251089f903b42526db2433a0ffa83d829b99670af8f3620b5e828abaf38"} Oct 31 12:50:25.572340 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:25.572048 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6df7b9f889-66dn6" Oct 31 12:50:25.591822 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:25.591766 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6df7b9f889-66dn6" podStartSLOduration=2.208437337 podStartE2EDuration="2.591752978s" totalImagesPullingTime="383.315641ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:50:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:50:25.590475674 +0000 UTC m=+1076.298061969" watchObservedRunningTime="2026-10-31 12:50:25.591752978 +0000 UTC m=+1076.299339272" Oct 31 12:50:27.539322 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:27.539299 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n4pg9" Oct 31 12:50:27.539728 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:27.539359 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-w7mmf" Oct 31 12:50:36.579307 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:36.579277 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6df7b9f889-66dn6" Oct 31 12:50:36.628282 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:36.628246 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-8cbkh"] Oct 31 12:50:36.628509 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:36.628463 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-8cbkh" podUID="d776d943-63d2-4b1c-a8d1-2e018aa16a01" containerName="manager" containerID="cri-o://f4df7f7b85d7e51993d3937351751825f786901b6313be6692a184ee118722f2" gracePeriod=30 Oct 31 12:50:36.866933 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:36.866905 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-8cbkh" Oct 31 12:50:36.972710 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:36.972679 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d776d943-63d2-4b1c-a8d1-2e018aa16a01-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d776d943-63d2-4b1c-a8d1-2e018aa16a01-metrics-certs\") pod \"d776d943-63d2-4b1c-a8d1-2e018aa16a01\" (UID: \"d776d943-63d2-4b1c-a8d1-2e018aa16a01\") " Oct 31 12:50:36.972853 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:36.972749 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d776d943-63d2-4b1c-a8d1-2e018aa16a01-kube-api-access-xdh9w\" (UniqueName: \"kubernetes.io/projected/d776d943-63d2-4b1c-a8d1-2e018aa16a01-kube-api-access-xdh9w\") pod \"d776d943-63d2-4b1c-a8d1-2e018aa16a01\" (UID: \"d776d943-63d2-4b1c-a8d1-2e018aa16a01\") " Oct 31 12:50:36.974986 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:36.974961 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d776d943-63d2-4b1c-a8d1-2e018aa16a01-kube-api-access-xdh9w" pod "d776d943-63d2-4b1c-a8d1-2e018aa16a01" (UID: "d776d943-63d2-4b1c-a8d1-2e018aa16a01"). InnerVolumeSpecName "kube-api-access-xdh9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:50:36.975061 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:36.974982 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d776d943-63d2-4b1c-a8d1-2e018aa16a01-metrics-certs" pod "d776d943-63d2-4b1c-a8d1-2e018aa16a01" (UID: "d776d943-63d2-4b1c-a8d1-2e018aa16a01"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:50:37.073701 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:37.073624 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d776d943-63d2-4b1c-a8d1-2e018aa16a01-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:50:37.073701 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:37.073700 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdh9w\" (UniqueName: \"kubernetes.io/projected/d776d943-63d2-4b1c-a8d1-2e018aa16a01-kube-api-access-xdh9w\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:50:37.624425 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:37.624392 5693 generic.go:356] "Generic (PLEG): container finished" podID="d776d943-63d2-4b1c-a8d1-2e018aa16a01" containerID="f4df7f7b85d7e51993d3937351751825f786901b6313be6692a184ee118722f2" exitCode=0 Oct 31 12:50:37.624893 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:37.624439 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-8cbkh" Oct 31 12:50:37.624893 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:37.624457 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-8cbkh" event={"ID":"d776d943-63d2-4b1c-a8d1-2e018aa16a01","Type":"ContainerDied","Data":"f4df7f7b85d7e51993d3937351751825f786901b6313be6692a184ee118722f2"} Oct 31 12:50:37.624893 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:37.624498 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-8cbkh" event={"ID":"d776d943-63d2-4b1c-a8d1-2e018aa16a01","Type":"ContainerDied","Data":"c665de7f414a0607cfab430b0e0b414e3978f72c4d13d1a60c422417dc0c4506"} Oct 31 12:50:37.624893 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:37.624517 5693 scope.go:122] "RemoveContainer" containerID="f4df7f7b85d7e51993d3937351751825f786901b6313be6692a184ee118722f2" Oct 31 12:50:37.633199 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:37.633011 5693 scope.go:122] "RemoveContainer" containerID="f4df7f7b85d7e51993d3937351751825f786901b6313be6692a184ee118722f2" Oct 31 12:50:37.633292 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:50:37.633276 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4df7f7b85d7e51993d3937351751825f786901b6313be6692a184ee118722f2\": container with ID starting with f4df7f7b85d7e51993d3937351751825f786901b6313be6692a184ee118722f2 not found: ID does not exist" containerID="f4df7f7b85d7e51993d3937351751825f786901b6313be6692a184ee118722f2" Oct 31 12:50:37.633336 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:37.633298 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4df7f7b85d7e51993d3937351751825f786901b6313be6692a184ee118722f2"} err="failed to get container status \"f4df7f7b85d7e51993d3937351751825f786901b6313be6692a184ee118722f2\": rpc error: code = NotFound desc = could not find container \"f4df7f7b85d7e51993d3937351751825f786901b6313be6692a184ee118722f2\": container with ID starting with f4df7f7b85d7e51993d3937351751825f786901b6313be6692a184ee118722f2 not found: ID does not exist" Oct 31 12:50:37.641307 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:37.641285 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-8cbkh"] Oct 31 12:50:37.647670 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:37.647631 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-8cbkh"] Oct 31 12:50:39.407964 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:39.407893 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d776d943-63d2-4b1c-a8d1-2e018aa16a01" path="/var/lib/kubelet/pods/d776d943-63d2-4b1c-a8d1-2e018aa16a01/volumes" Oct 31 12:50:43.597745 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:43.597712 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5"] Oct 31 12:50:43.598239 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:43.598179 5693 cpu_manager.go:414] "containerMap: removing container" podUID="d776d943-63d2-4b1c-a8d1-2e018aa16a01" containerName="manager" Oct 31 12:50:43.598239 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:43.598196 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d776d943-63d2-4b1c-a8d1-2e018aa16a01" containerName="manager" Oct 31 12:50:43.598352 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:43.598306 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="d776d943-63d2-4b1c-a8d1-2e018aa16a01" containerName="manager" Oct 31 12:50:43.604262 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:43.604239 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5" Oct 31 12:50:43.613391 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:43.613362 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 31 12:50:43.613391 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:43.613359 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 31 12:50:43.613575 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:43.613558 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-26vsr\"" Oct 31 12:50:43.614194 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:43.614175 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 31 12:50:43.614451 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:43.614433 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5"] Oct 31 12:50:43.734903 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:43.734872 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adaef1d1-7d48-4517-af6b-0a5ec03b958c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5\" (UID: \"adaef1d1-7d48-4517-af6b-0a5ec03b958c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5" Oct 31 12:50:43.734903 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:43.734905 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lv2d\" (UniqueName: \"kubernetes.io/projected/adaef1d1-7d48-4517-af6b-0a5ec03b958c-kube-api-access-6lv2d\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5\" (UID: \"adaef1d1-7d48-4517-af6b-0a5ec03b958c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5" Oct 31 12:50:43.735096 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:43.734936 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adaef1d1-7d48-4517-af6b-0a5ec03b958c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5\" (UID: \"adaef1d1-7d48-4517-af6b-0a5ec03b958c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5" Oct 31 12:50:43.735096 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:43.735033 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adaef1d1-7d48-4517-af6b-0a5ec03b958c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5\" (UID: \"adaef1d1-7d48-4517-af6b-0a5ec03b958c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5" Oct 31 12:50:43.735096 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:43.735068 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adaef1d1-7d48-4517-af6b-0a5ec03b958c-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5\" (UID: \"adaef1d1-7d48-4517-af6b-0a5ec03b958c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5" Oct 31 12:50:43.735096 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:43.735088 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adaef1d1-7d48-4517-af6b-0a5ec03b958c-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5\" (UID: \"adaef1d1-7d48-4517-af6b-0a5ec03b958c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5" Oct 31 12:50:43.836341 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:43.836302 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adaef1d1-7d48-4517-af6b-0a5ec03b958c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5\" (UID: \"adaef1d1-7d48-4517-af6b-0a5ec03b958c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5" Oct 31 12:50:43.836493 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:43.836364 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adaef1d1-7d48-4517-af6b-0a5ec03b958c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5\" (UID: \"adaef1d1-7d48-4517-af6b-0a5ec03b958c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5" Oct 31 12:50:43.836493 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:43.836395 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adaef1d1-7d48-4517-af6b-0a5ec03b958c-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5\" (UID: \"adaef1d1-7d48-4517-af6b-0a5ec03b958c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5" Oct 31 12:50:43.836493 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:43.836414 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adaef1d1-7d48-4517-af6b-0a5ec03b958c-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5\" (UID: \"adaef1d1-7d48-4517-af6b-0a5ec03b958c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5" Oct 31 12:50:43.836493 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:43.836437 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adaef1d1-7d48-4517-af6b-0a5ec03b958c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5\" (UID: \"adaef1d1-7d48-4517-af6b-0a5ec03b958c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5" Oct 31 12:50:43.836493 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:43.836453 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lv2d\" (UniqueName: \"kubernetes.io/projected/adaef1d1-7d48-4517-af6b-0a5ec03b958c-kube-api-access-6lv2d\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5\" (UID: \"adaef1d1-7d48-4517-af6b-0a5ec03b958c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5" Oct 31 12:50:43.836808 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:43.836785 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adaef1d1-7d48-4517-af6b-0a5ec03b958c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5\" (UID: \"adaef1d1-7d48-4517-af6b-0a5ec03b958c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5" Oct 31 12:50:43.836894 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:43.836871 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adaef1d1-7d48-4517-af6b-0a5ec03b958c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5\" (UID: \"adaef1d1-7d48-4517-af6b-0a5ec03b958c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5" Oct 31 12:50:43.836962 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:43.836928 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adaef1d1-7d48-4517-af6b-0a5ec03b958c-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5\" (UID: \"adaef1d1-7d48-4517-af6b-0a5ec03b958c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5" Oct 31 12:50:43.838799 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:43.838776 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adaef1d1-7d48-4517-af6b-0a5ec03b958c-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5\" (UID: \"adaef1d1-7d48-4517-af6b-0a5ec03b958c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5" Oct 31 12:50:43.839023 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:43.839008 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adaef1d1-7d48-4517-af6b-0a5ec03b958c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5\" (UID: \"adaef1d1-7d48-4517-af6b-0a5ec03b958c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5" Oct 31 12:50:43.848525 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:43.848472 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lv2d\" (UniqueName: \"kubernetes.io/projected/adaef1d1-7d48-4517-af6b-0a5ec03b958c-kube-api-access-6lv2d\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5\" (UID: \"adaef1d1-7d48-4517-af6b-0a5ec03b958c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5" Oct 31 12:50:43.918402 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:43.918362 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5" Oct 31 12:50:44.196440 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:44.196403 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5"] Oct 31 12:50:44.199495 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:50:44.199468 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadaef1d1_7d48_4517_af6b_0a5ec03b958c.slice/crio-5906401a6fa8717dead912b6331c4e239abdada946530d457c2d683577e6ba98 WatchSource:0}: Error finding container 5906401a6fa8717dead912b6331c4e239abdada946530d457c2d683577e6ba98: Status 404 returned error can't find the container with id 5906401a6fa8717dead912b6331c4e239abdada946530d457c2d683577e6ba98 Oct 31 12:50:44.660736 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:44.660628 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5" event={"ID":"adaef1d1-7d48-4517-af6b-0a5ec03b958c","Type":"ContainerStarted","Data":"5906401a6fa8717dead912b6331c4e239abdada946530d457c2d683577e6ba98"} Oct 31 12:50:45.666387 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:45.666348 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5" event={"ID":"adaef1d1-7d48-4517-af6b-0a5ec03b958c","Type":"ContainerStarted","Data":"a9f70ecca0ee4907eeadec2af975ffde51118be0f197a9b5e0b0df9e5a25d036"} Oct 31 12:50:45.686715 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:45.686605 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5" podStartSLOduration=1.461423535 podStartE2EDuration="2.686590208s" totalImagesPullingTime="1.225166673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:50:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:50:45.685562897 +0000 UTC m=+1096.393149197" watchObservedRunningTime="2026-10-31 12:50:45.686590208 +0000 UTC m=+1096.394176559" Oct 31 12:50:45.918752 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:45.918719 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5" Oct 31 12:50:47.903499 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:47.903465 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-j8sln"] Oct 31 12:50:47.907530 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:47.907505 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-j8sln" Oct 31 12:50:47.916033 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:47.915766 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 31 12:50:47.917320 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:47.917298 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-j8sln"] Oct 31 12:50:48.070225 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:48.070186 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6597c0b-d1c0-4000-b098-520ea8eae932-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-j8sln\" (UID: \"f6597c0b-d1c0-4000-b098-520ea8eae932\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-j8sln" Oct 31 12:50:48.070409 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:48.070274 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6597c0b-d1c0-4000-b098-520ea8eae932-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-j8sln\" (UID: \"f6597c0b-d1c0-4000-b098-520ea8eae932\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-j8sln" Oct 31 12:50:48.070409 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:48.070335 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6597c0b-d1c0-4000-b098-520ea8eae932-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-j8sln\" (UID: \"f6597c0b-d1c0-4000-b098-520ea8eae932\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-j8sln" Oct 31 12:50:48.070493 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:48.070404 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5sfr\" (UniqueName: \"kubernetes.io/projected/f6597c0b-d1c0-4000-b098-520ea8eae932-kube-api-access-m5sfr\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-j8sln\" (UID: \"f6597c0b-d1c0-4000-b098-520ea8eae932\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-j8sln" Oct 31 12:50:48.070493 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:48.070436 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6597c0b-d1c0-4000-b098-520ea8eae932-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-j8sln\" (UID: \"f6597c0b-d1c0-4000-b098-520ea8eae932\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-j8sln" Oct 31 12:50:48.070493 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:48.070457 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6597c0b-d1c0-4000-b098-520ea8eae932-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-j8sln\" (UID: \"f6597c0b-d1c0-4000-b098-520ea8eae932\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-j8sln" Oct 31 12:50:48.171560 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:48.171478 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6597c0b-d1c0-4000-b098-520ea8eae932-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-j8sln\" (UID: \"f6597c0b-d1c0-4000-b098-520ea8eae932\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-j8sln" Oct 31 12:50:48.171560 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:48.171534 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6597c0b-d1c0-4000-b098-520ea8eae932-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-j8sln\" (UID: \"f6597c0b-d1c0-4000-b098-520ea8eae932\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-j8sln" Oct 31 12:50:48.171773 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:48.171665 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5sfr\" (UniqueName: \"kubernetes.io/projected/f6597c0b-d1c0-4000-b098-520ea8eae932-kube-api-access-m5sfr\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-j8sln\" (UID: \"f6597c0b-d1c0-4000-b098-520ea8eae932\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-j8sln" Oct 31 12:50:48.171773 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:48.171721 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6597c0b-d1c0-4000-b098-520ea8eae932-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-j8sln\" (UID: \"f6597c0b-d1c0-4000-b098-520ea8eae932\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-j8sln" Oct 31 12:50:48.171773 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:48.171744 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6597c0b-d1c0-4000-b098-520ea8eae932-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-j8sln\" (UID: \"f6597c0b-d1c0-4000-b098-520ea8eae932\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-j8sln" Oct 31 12:50:48.171773 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:48.171763 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6597c0b-d1c0-4000-b098-520ea8eae932-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-j8sln\" (UID: \"f6597c0b-d1c0-4000-b098-520ea8eae932\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-j8sln" Oct 31 12:50:48.171932 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:48.171886 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6597c0b-d1c0-4000-b098-520ea8eae932-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-j8sln\" (UID: \"f6597c0b-d1c0-4000-b098-520ea8eae932\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-j8sln" Oct 31 12:50:48.171967 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:48.171942 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6597c0b-d1c0-4000-b098-520ea8eae932-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-j8sln\" (UID: \"f6597c0b-d1c0-4000-b098-520ea8eae932\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-j8sln" Oct 31 12:50:48.172118 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:48.172098 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6597c0b-d1c0-4000-b098-520ea8eae932-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-j8sln\" (UID: \"f6597c0b-d1c0-4000-b098-520ea8eae932\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-j8sln" Oct 31 12:50:48.174073 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:48.174052 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6597c0b-d1c0-4000-b098-520ea8eae932-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-j8sln\" (UID: \"f6597c0b-d1c0-4000-b098-520ea8eae932\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-j8sln" Oct 31 12:50:48.174377 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:48.174362 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6597c0b-d1c0-4000-b098-520ea8eae932-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-j8sln\" (UID: \"f6597c0b-d1c0-4000-b098-520ea8eae932\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-j8sln" Oct 31 12:50:48.184174 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:48.184154 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5sfr\" (UniqueName: \"kubernetes.io/projected/f6597c0b-d1c0-4000-b098-520ea8eae932-kube-api-access-m5sfr\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-j8sln\" (UID: \"f6597c0b-d1c0-4000-b098-520ea8eae932\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-j8sln" Oct 31 12:50:48.222241 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:48.222206 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-j8sln" Oct 31 12:50:48.496766 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:48.496567 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-j8sln"] Oct 31 12:50:48.500051 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:50:48.500015 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6597c0b_d1c0_4000_b098_520ea8eae932.slice/crio-0cc03d3919751642aad36534a6a140a587fa0ee21972345080cdea250b20eec3 WatchSource:0}: Error finding container 0cc03d3919751642aad36534a6a140a587fa0ee21972345080cdea250b20eec3: Status 404 returned error can't find the container with id 0cc03d3919751642aad36534a6a140a587fa0ee21972345080cdea250b20eec3 Oct 31 12:50:48.679387 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:48.679353 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-j8sln" event={"ID":"f6597c0b-d1c0-4000-b098-520ea8eae932","Type":"ContainerStarted","Data":"0cc03d3919751642aad36534a6a140a587fa0ee21972345080cdea250b20eec3"} Oct 31 12:50:48.688148 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:50:48.688117 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:50:49.686073 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:49.686035 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-j8sln" event={"ID":"f6597c0b-d1c0-4000-b098-520ea8eae932","Type":"ContainerStarted","Data":"65a15d106f494bf006930b3bd6084a269759ddeadd786fd51d1185c04a9100d8"} Oct 31 12:50:49.707438 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:49.707383 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-j8sln" podStartSLOduration=2.421541496 podStartE2EDuration="2.707368565s" totalImagesPullingTime="285.827069ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:50:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:50:49.706983544 +0000 UTC m=+1100.414569840" watchObservedRunningTime="2026-10-31 12:50:49.707368565 +0000 UTC m=+1100.414954860" Oct 31 12:50:50.223102 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:50.223071 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-j8sln" Oct 31 12:50:50.725896 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:50.725861 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-lfzgn"] Oct 31 12:50:50.734089 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:50.734057 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lfzgn" Oct 31 12:50:50.740121 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:50.740099 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 31 12:50:50.743700 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:50.741866 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-lfzgn"] Oct 31 12:50:50.893749 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:50.893718 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80aab26c-df22-4263-bc1e-4e066eae2903-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lfzgn\" (UID: \"80aab26c-df22-4263-bc1e-4e066eae2903\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lfzgn" Oct 31 12:50:50.893907 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:50.893764 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xkz9\" (UniqueName: \"kubernetes.io/projected/80aab26c-df22-4263-bc1e-4e066eae2903-kube-api-access-8xkz9\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lfzgn\" (UID: \"80aab26c-df22-4263-bc1e-4e066eae2903\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lfzgn" Oct 31 12:50:50.893907 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:50.893856 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80aab26c-df22-4263-bc1e-4e066eae2903-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lfzgn\" (UID: \"80aab26c-df22-4263-bc1e-4e066eae2903\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lfzgn" Oct 31 12:50:50.893907 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:50.893875 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80aab26c-df22-4263-bc1e-4e066eae2903-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lfzgn\" (UID: \"80aab26c-df22-4263-bc1e-4e066eae2903\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lfzgn" Oct 31 12:50:50.894003 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:50.893958 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80aab26c-df22-4263-bc1e-4e066eae2903-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lfzgn\" (UID: \"80aab26c-df22-4263-bc1e-4e066eae2903\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lfzgn" Oct 31 12:50:50.894003 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:50.893986 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80aab26c-df22-4263-bc1e-4e066eae2903-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lfzgn\" (UID: \"80aab26c-df22-4263-bc1e-4e066eae2903\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lfzgn" Oct 31 12:50:50.995288 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:50.995208 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80aab26c-df22-4263-bc1e-4e066eae2903-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lfzgn\" (UID: \"80aab26c-df22-4263-bc1e-4e066eae2903\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lfzgn" Oct 31 12:50:50.995288 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:50.995266 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80aab26c-df22-4263-bc1e-4e066eae2903-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lfzgn\" (UID: \"80aab26c-df22-4263-bc1e-4e066eae2903\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lfzgn" Oct 31 12:50:50.995482 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:50.995294 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xkz9\" (UniqueName: \"kubernetes.io/projected/80aab26c-df22-4263-bc1e-4e066eae2903-kube-api-access-8xkz9\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lfzgn\" (UID: \"80aab26c-df22-4263-bc1e-4e066eae2903\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lfzgn" Oct 31 12:50:50.995482 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:50.995325 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80aab26c-df22-4263-bc1e-4e066eae2903-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lfzgn\" (UID: \"80aab26c-df22-4263-bc1e-4e066eae2903\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lfzgn" Oct 31 12:50:50.995482 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:50.995342 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80aab26c-df22-4263-bc1e-4e066eae2903-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lfzgn\" (UID: \"80aab26c-df22-4263-bc1e-4e066eae2903\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lfzgn" Oct 31 12:50:50.995482 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:50.995367 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80aab26c-df22-4263-bc1e-4e066eae2903-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lfzgn\" (UID: \"80aab26c-df22-4263-bc1e-4e066eae2903\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lfzgn" Oct 31 12:50:50.995772 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:50.995738 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80aab26c-df22-4263-bc1e-4e066eae2903-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lfzgn\" (UID: \"80aab26c-df22-4263-bc1e-4e066eae2903\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lfzgn" Oct 31 12:50:50.995891 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:50.995752 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80aab26c-df22-4263-bc1e-4e066eae2903-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lfzgn\" (UID: \"80aab26c-df22-4263-bc1e-4e066eae2903\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lfzgn" Oct 31 12:50:50.995891 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:50.995824 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80aab26c-df22-4263-bc1e-4e066eae2903-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lfzgn\" (UID: \"80aab26c-df22-4263-bc1e-4e066eae2903\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lfzgn" Oct 31 12:50:50.997771 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:50.997745 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80aab26c-df22-4263-bc1e-4e066eae2903-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lfzgn\" (UID: \"80aab26c-df22-4263-bc1e-4e066eae2903\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lfzgn" Oct 31 12:50:50.997874 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:50.997858 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80aab26c-df22-4263-bc1e-4e066eae2903-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lfzgn\" (UID: \"80aab26c-df22-4263-bc1e-4e066eae2903\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lfzgn" Oct 31 12:50:51.007386 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:51.007357 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xkz9\" (UniqueName: \"kubernetes.io/projected/80aab26c-df22-4263-bc1e-4e066eae2903-kube-api-access-8xkz9\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lfzgn\" (UID: \"80aab26c-df22-4263-bc1e-4e066eae2903\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lfzgn" Oct 31 12:50:51.049327 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:51.049303 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lfzgn" Oct 31 12:50:51.323921 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:51.323846 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-lfzgn"] Oct 31 12:50:51.327328 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:50:51.327299 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80aab26c_df22_4263_bc1e_4e066eae2903.slice/crio-e3abf1964ec4f0b2a122fc32be5d8db6705630896b08fc3173c43b0b5abf4aa7 WatchSource:0}: Error finding container e3abf1964ec4f0b2a122fc32be5d8db6705630896b08fc3173c43b0b5abf4aa7: Status 404 returned error can't find the container with id e3abf1964ec4f0b2a122fc32be5d8db6705630896b08fc3173c43b0b5abf4aa7 Oct 31 12:50:51.695832 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:51.695792 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lfzgn" event={"ID":"80aab26c-df22-4263-bc1e-4e066eae2903","Type":"ContainerStarted","Data":"e3abf1964ec4f0b2a122fc32be5d8db6705630896b08fc3173c43b0b5abf4aa7"} Oct 31 12:50:52.221355 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:52.221310 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg"] Oct 31 12:50:52.225371 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:52.225347 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg" Oct 31 12:50:52.229207 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:52.229184 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 31 12:50:52.236525 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:52.236494 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg"] Oct 31 12:50:52.314052 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:52.314012 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvwld\" (UniqueName: \"kubernetes.io/projected/61487058-ab9d-4537-ab76-cf431dd17a7f-kube-api-access-xvwld\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg\" (UID: \"61487058-ab9d-4537-ab76-cf431dd17a7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg" Oct 31 12:50:52.314225 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:52.314064 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61487058-ab9d-4537-ab76-cf431dd17a7f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg\" (UID: \"61487058-ab9d-4537-ab76-cf431dd17a7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg" Oct 31 12:50:52.314225 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:52.314126 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61487058-ab9d-4537-ab76-cf431dd17a7f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg\" (UID: \"61487058-ab9d-4537-ab76-cf431dd17a7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg" Oct 31 12:50:52.314225 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:52.314177 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61487058-ab9d-4537-ab76-cf431dd17a7f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg\" (UID: \"61487058-ab9d-4537-ab76-cf431dd17a7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg" Oct 31 12:50:52.314325 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:52.314274 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61487058-ab9d-4537-ab76-cf431dd17a7f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg\" (UID: \"61487058-ab9d-4537-ab76-cf431dd17a7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg" Oct 31 12:50:52.314325 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:52.314301 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61487058-ab9d-4537-ab76-cf431dd17a7f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg\" (UID: \"61487058-ab9d-4537-ab76-cf431dd17a7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg" Oct 31 12:50:52.415620 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:52.415586 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61487058-ab9d-4537-ab76-cf431dd17a7f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg\" (UID: \"61487058-ab9d-4537-ab76-cf431dd17a7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg" Oct 31 12:50:52.415620 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:52.415619 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61487058-ab9d-4537-ab76-cf431dd17a7f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg\" (UID: \"61487058-ab9d-4537-ab76-cf431dd17a7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg" Oct 31 12:50:52.415854 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:52.415764 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvwld\" (UniqueName: \"kubernetes.io/projected/61487058-ab9d-4537-ab76-cf431dd17a7f-kube-api-access-xvwld\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg\" (UID: \"61487058-ab9d-4537-ab76-cf431dd17a7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg" Oct 31 12:50:52.415854 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:52.415828 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61487058-ab9d-4537-ab76-cf431dd17a7f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg\" (UID: \"61487058-ab9d-4537-ab76-cf431dd17a7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg" Oct 31 12:50:52.415998 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:52.415977 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61487058-ab9d-4537-ab76-cf431dd17a7f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg\" (UID: \"61487058-ab9d-4537-ab76-cf431dd17a7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg" Oct 31 12:50:52.416067 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:52.416050 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61487058-ab9d-4537-ab76-cf431dd17a7f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg\" (UID: \"61487058-ab9d-4537-ab76-cf431dd17a7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg" Oct 31 12:50:52.416121 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:52.416054 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61487058-ab9d-4537-ab76-cf431dd17a7f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg\" (UID: \"61487058-ab9d-4537-ab76-cf431dd17a7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg" Oct 31 12:50:52.416121 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:52.416099 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61487058-ab9d-4537-ab76-cf431dd17a7f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg\" (UID: \"61487058-ab9d-4537-ab76-cf431dd17a7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg" Oct 31 12:50:52.416225 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:52.416137 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61487058-ab9d-4537-ab76-cf431dd17a7f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg\" (UID: \"61487058-ab9d-4537-ab76-cf431dd17a7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg" Oct 31 12:50:52.418160 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:52.418137 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61487058-ab9d-4537-ab76-cf431dd17a7f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg\" (UID: \"61487058-ab9d-4537-ab76-cf431dd17a7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg" Oct 31 12:50:52.418585 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:52.418568 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61487058-ab9d-4537-ab76-cf431dd17a7f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg\" (UID: \"61487058-ab9d-4537-ab76-cf431dd17a7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg" Oct 31 12:50:52.427956 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:52.427928 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvwld\" (UniqueName: \"kubernetes.io/projected/61487058-ab9d-4537-ab76-cf431dd17a7f-kube-api-access-xvwld\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg\" (UID: \"61487058-ab9d-4537-ab76-cf431dd17a7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg" Oct 31 12:50:52.541305 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:52.541219 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg" Oct 31 12:50:52.702124 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:52.702088 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lfzgn" event={"ID":"80aab26c-df22-4263-bc1e-4e066eae2903","Type":"ContainerStarted","Data":"96fbaea9b619d5230a85de020ae5758976ba3493feb2b3594b0faf7b44a71b07"} Oct 31 12:50:52.723029 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:52.722980 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lfzgn" podStartSLOduration=2.404612225 podStartE2EDuration="2.722967474s" totalImagesPullingTime="318.355249ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:50:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:50:52.720918256 +0000 UTC m=+1103.428504551" watchObservedRunningTime="2026-10-31 12:50:52.722967474 +0000 UTC m=+1103.430553769" Oct 31 12:50:52.815357 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:52.815285 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg"] Oct 31 12:50:52.817860 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:50:52.817833 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61487058_ab9d_4537_ab76_cf431dd17a7f.slice/crio-5e42e03767bf3aebd14fd4e62a1fa9d8969dbb26cec8d1c915ffb084b0516097 WatchSource:0}: Error finding container 5e42e03767bf3aebd14fd4e62a1fa9d8969dbb26cec8d1c915ffb084b0516097: Status 404 returned error can't find the container with id 5e42e03767bf3aebd14fd4e62a1fa9d8969dbb26cec8d1c915ffb084b0516097 Oct 31 12:50:53.049464 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:53.049428 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lfzgn" Oct 31 12:50:53.708694 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:53.708654 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg" event={"ID":"61487058-ab9d-4537-ab76-cf431dd17a7f","Type":"ContainerStarted","Data":"7be8fe56939c46ee1b3789f878a1b828fb8ae92f278ca8326368c3f5a68d49c3"} Oct 31 12:50:53.709150 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:53.708723 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg" event={"ID":"61487058-ab9d-4537-ab76-cf431dd17a7f","Type":"ContainerStarted","Data":"5e42e03767bf3aebd14fd4e62a1fa9d8969dbb26cec8d1c915ffb084b0516097"} Oct 31 12:50:53.730438 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:53.730390 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg" podStartSLOduration=1.446263422 podStartE2EDuration="1.730375239s" totalImagesPullingTime="284.111817ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:50:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:50:53.729392347 +0000 UTC m=+1104.436978645" watchObservedRunningTime="2026-10-31 12:50:53.730375239 +0000 UTC m=+1104.437961558" Oct 31 12:50:53.919519 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:53.919481 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5" Oct 31 12:50:53.932204 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:53.932179 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5" Oct 31 12:50:54.541622 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:54.541590 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg" Oct 31 12:50:54.724544 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:54.724516 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5" Oct 31 12:50:57.575878 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:57.575818 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm"] Oct 31 12:50:57.580369 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:57.580347 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" Oct 31 12:50:57.585014 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:57.584992 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 31 12:50:57.589730 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:57.589707 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm"] Oct 31 12:50:57.674033 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:57.673996 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb46t\" (UniqueName: \"kubernetes.io/projected/66c6ec5a-c0da-4bd7-a314-0c90ccec2425-kube-api-access-qb46t\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm\" (UID: \"66c6ec5a-c0da-4bd7-a314-0c90ccec2425\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" Oct 31 12:50:57.674226 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:57.674109 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66c6ec5a-c0da-4bd7-a314-0c90ccec2425-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm\" (UID: \"66c6ec5a-c0da-4bd7-a314-0c90ccec2425\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" Oct 31 12:50:57.674226 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:57.674150 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66c6ec5a-c0da-4bd7-a314-0c90ccec2425-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm\" (UID: \"66c6ec5a-c0da-4bd7-a314-0c90ccec2425\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" Oct 31 12:50:57.674335 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:57.674240 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66c6ec5a-c0da-4bd7-a314-0c90ccec2425-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm\" (UID: \"66c6ec5a-c0da-4bd7-a314-0c90ccec2425\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" Oct 31 12:50:57.674383 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:57.674332 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66c6ec5a-c0da-4bd7-a314-0c90ccec2425-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm\" (UID: \"66c6ec5a-c0da-4bd7-a314-0c90ccec2425\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" Oct 31 12:50:57.674383 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:57.674350 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66c6ec5a-c0da-4bd7-a314-0c90ccec2425-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm\" (UID: \"66c6ec5a-c0da-4bd7-a314-0c90ccec2425\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" Oct 31 12:50:57.774820 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:57.774781 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66c6ec5a-c0da-4bd7-a314-0c90ccec2425-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm\" (UID: \"66c6ec5a-c0da-4bd7-a314-0c90ccec2425\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" Oct 31 12:50:57.774820 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:57.774823 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66c6ec5a-c0da-4bd7-a314-0c90ccec2425-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm\" (UID: \"66c6ec5a-c0da-4bd7-a314-0c90ccec2425\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" Oct 31 12:50:57.775084 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:57.774985 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66c6ec5a-c0da-4bd7-a314-0c90ccec2425-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm\" (UID: \"66c6ec5a-c0da-4bd7-a314-0c90ccec2425\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" Oct 31 12:50:57.775084 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:57.775029 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66c6ec5a-c0da-4bd7-a314-0c90ccec2425-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm\" (UID: \"66c6ec5a-c0da-4bd7-a314-0c90ccec2425\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" Oct 31 12:50:57.775084 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:57.775070 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb46t\" (UniqueName: \"kubernetes.io/projected/66c6ec5a-c0da-4bd7-a314-0c90ccec2425-kube-api-access-qb46t\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm\" (UID: \"66c6ec5a-c0da-4bd7-a314-0c90ccec2425\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" Oct 31 12:50:57.775237 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:57.775154 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66c6ec5a-c0da-4bd7-a314-0c90ccec2425-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm\" (UID: \"66c6ec5a-c0da-4bd7-a314-0c90ccec2425\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" Oct 31 12:50:57.775306 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:57.775282 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66c6ec5a-c0da-4bd7-a314-0c90ccec2425-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm\" (UID: \"66c6ec5a-c0da-4bd7-a314-0c90ccec2425\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" Oct 31 12:50:57.775366 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:57.775340 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66c6ec5a-c0da-4bd7-a314-0c90ccec2425-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm\" (UID: \"66c6ec5a-c0da-4bd7-a314-0c90ccec2425\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" Oct 31 12:50:57.775529 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:57.775491 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66c6ec5a-c0da-4bd7-a314-0c90ccec2425-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm\" (UID: \"66c6ec5a-c0da-4bd7-a314-0c90ccec2425\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" Oct 31 12:50:57.777158 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:57.777139 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66c6ec5a-c0da-4bd7-a314-0c90ccec2425-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm\" (UID: \"66c6ec5a-c0da-4bd7-a314-0c90ccec2425\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" Oct 31 12:50:57.777531 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:57.777511 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66c6ec5a-c0da-4bd7-a314-0c90ccec2425-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm\" (UID: \"66c6ec5a-c0da-4bd7-a314-0c90ccec2425\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" Oct 31 12:50:57.787511 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:57.787489 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb46t\" (UniqueName: \"kubernetes.io/projected/66c6ec5a-c0da-4bd7-a314-0c90ccec2425-kube-api-access-qb46t\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm\" (UID: \"66c6ec5a-c0da-4bd7-a314-0c90ccec2425\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" Oct 31 12:50:57.894568 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:57.894526 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" Oct 31 12:50:58.169601 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:58.169527 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm"] Oct 31 12:50:58.172903 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:50:58.172878 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66c6ec5a_c0da_4bd7_a314_0c90ccec2425.slice/crio-6cb4587d67ba943256ff2e766d767b6d3e7db8f1e5da96844bfacbe7a7c89bec WatchSource:0}: Error finding container 6cb4587d67ba943256ff2e766d767b6d3e7db8f1e5da96844bfacbe7a7c89bec: Status 404 returned error can't find the container with id 6cb4587d67ba943256ff2e766d767b6d3e7db8f1e5da96844bfacbe7a7c89bec Oct 31 12:50:58.223240 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:58.223211 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-j8sln" Oct 31 12:50:58.235928 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:58.235905 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-j8sln" Oct 31 12:50:58.740580 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:58.740493 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" event={"ID":"66c6ec5a-c0da-4bd7-a314-0c90ccec2425","Type":"ContainerStarted","Data":"09639c131e70ac499625f6055d799904a1b5ad88d936437286f6ac472a8c53ab"} Oct 31 12:50:58.740580 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:58.740533 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" event={"ID":"66c6ec5a-c0da-4bd7-a314-0c90ccec2425","Type":"ContainerStarted","Data":"6cb4587d67ba943256ff2e766d767b6d3e7db8f1e5da96844bfacbe7a7c89bec"} Oct 31 12:50:58.752302 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:58.752273 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-j8sln" Oct 31 12:50:58.760513 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:58.760470 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" podStartSLOduration=1.491160689 podStartE2EDuration="1.760455925s" totalImagesPullingTime="269.295236ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:50:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:50:58.759601877 +0000 UTC m=+1109.467188175" watchObservedRunningTime="2026-10-31 12:50:58.760455925 +0000 UTC m=+1109.468042219" Oct 31 12:50:58.895513 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:50:58.895479 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" Oct 31 12:51:01.049863 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:01.049832 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lfzgn" Oct 31 12:51:01.062339 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:01.062313 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lfzgn" Oct 31 12:51:01.765823 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:01.765795 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lfzgn" Oct 31 12:51:02.541784 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:02.541751 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg" Oct 31 12:51:02.554408 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:02.554384 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg" Oct 31 12:51:02.770572 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:02.770547 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg" Oct 31 12:51:07.894852 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:07.894799 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" Oct 31 12:51:07.907546 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:07.907516 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" Oct 31 12:51:08.797833 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:08.797804 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" Oct 31 12:51:11.470415 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:11.470380 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-9c9ccc756-vngkm"] Oct 31 12:51:11.473947 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:11.473929 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9c9ccc756-vngkm" Oct 31 12:51:11.485003 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:11.484982 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-9c9ccc756-vngkm"] Oct 31 12:51:11.496874 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:11.496844 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv29h\" (UniqueName: \"kubernetes.io/projected/56840e04-370d-44ef-92a1-46d4bb15f9ec-kube-api-access-pv29h\") pod \"authorino-9c9ccc756-vngkm\" (UID: \"56840e04-370d-44ef-92a1-46d4bb15f9ec\") " pod="kuadrant-system/authorino-9c9ccc756-vngkm" Oct 31 12:51:11.496993 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:11.496968 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/56840e04-370d-44ef-92a1-46d4bb15f9ec-tls-cert\") pod \"authorino-9c9ccc756-vngkm\" (UID: \"56840e04-370d-44ef-92a1-46d4bb15f9ec\") " pod="kuadrant-system/authorino-9c9ccc756-vngkm" Oct 31 12:51:11.598152 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:11.598115 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv29h\" (UniqueName: \"kubernetes.io/projected/56840e04-370d-44ef-92a1-46d4bb15f9ec-kube-api-access-pv29h\") pod \"authorino-9c9ccc756-vngkm\" (UID: \"56840e04-370d-44ef-92a1-46d4bb15f9ec\") " pod="kuadrant-system/authorino-9c9ccc756-vngkm" Oct 31 12:51:11.598386 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:11.598177 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/56840e04-370d-44ef-92a1-46d4bb15f9ec-tls-cert\") pod \"authorino-9c9ccc756-vngkm\" (UID: \"56840e04-370d-44ef-92a1-46d4bb15f9ec\") " pod="kuadrant-system/authorino-9c9ccc756-vngkm" Oct 31 12:51:11.600729 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:11.600710 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/56840e04-370d-44ef-92a1-46d4bb15f9ec-tls-cert\") pod \"authorino-9c9ccc756-vngkm\" (UID: \"56840e04-370d-44ef-92a1-46d4bb15f9ec\") " pod="kuadrant-system/authorino-9c9ccc756-vngkm" Oct 31 12:51:11.610718 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:11.610692 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv29h\" (UniqueName: \"kubernetes.io/projected/56840e04-370d-44ef-92a1-46d4bb15f9ec-kube-api-access-pv29h\") pod \"authorino-9c9ccc756-vngkm\" (UID: \"56840e04-370d-44ef-92a1-46d4bb15f9ec\") " pod="kuadrant-system/authorino-9c9ccc756-vngkm" Oct 31 12:51:11.785492 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:11.785414 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9c9ccc756-vngkm" Oct 31 12:51:12.060023 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:12.059997 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-9c9ccc756-vngkm"] Oct 31 12:51:12.062270 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:51:12.062245 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56840e04_370d_44ef_92a1_46d4bb15f9ec.slice/crio-1a3d350f041e89a3d70a5d625d37b842bee889366b9286f53442979c75519a3b WatchSource:0}: Error finding container 1a3d350f041e89a3d70a5d625d37b842bee889366b9286f53442979c75519a3b: Status 404 returned error can't find the container with id 1a3d350f041e89a3d70a5d625d37b842bee889366b9286f53442979c75519a3b Oct 31 12:51:12.806435 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:12.806398 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9c9ccc756-vngkm" event={"ID":"56840e04-370d-44ef-92a1-46d4bb15f9ec","Type":"ContainerStarted","Data":"1a3d350f041e89a3d70a5d625d37b842bee889366b9286f53442979c75519a3b"} Oct 31 12:51:13.814154 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:13.814112 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9c9ccc756-vngkm" event={"ID":"56840e04-370d-44ef-92a1-46d4bb15f9ec","Type":"ContainerStarted","Data":"b4cc1b41da67a29b6d1c3967c5833dfa2c871c15d2b89f205ec62442f376e71c"} Oct 31 12:51:13.835351 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:13.835291 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-9c9ccc756-vngkm" podStartSLOduration=1.741813872 podStartE2EDuration="2.835275749s" totalImagesPullingTime="1.093461877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:51:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:51:13.832691648 +0000 UTC m=+1124.540277944" watchObservedRunningTime="2026-10-31 12:51:13.835275749 +0000 UTC m=+1124.542862046" Oct 31 12:51:13.867551 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:13.867503 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-77b4b46987-vxf8x"] Oct 31 12:51:13.869391 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:13.869359 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-77b4b46987-vxf8x" podUID="bbdd88e1-3077-465d-a515-b8b9f78ca3b7" containerName="authorino" containerID="cri-o://8155e64362b068f86866b19180d8def2d6f3bf40fe7a9bd4e18d70c6f1d71dae" gracePeriod=30 Oct 31 12:51:14.133019 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:14.132990 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77b4b46987-vxf8x" Oct 31 12:51:14.221298 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:14.221267 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbdd88e1-3077-465d-a515-b8b9f78ca3b7-kube-api-access-mfn6z\" (UniqueName: \"kubernetes.io/projected/bbdd88e1-3077-465d-a515-b8b9f78ca3b7-kube-api-access-mfn6z\") pod \"bbdd88e1-3077-465d-a515-b8b9f78ca3b7\" (UID: \"bbdd88e1-3077-465d-a515-b8b9f78ca3b7\") " Oct 31 12:51:14.221484 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:14.221314 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbdd88e1-3077-465d-a515-b8b9f78ca3b7-tls-cert\" (UniqueName: \"kubernetes.io/secret/bbdd88e1-3077-465d-a515-b8b9f78ca3b7-tls-cert\") pod \"bbdd88e1-3077-465d-a515-b8b9f78ca3b7\" (UID: \"bbdd88e1-3077-465d-a515-b8b9f78ca3b7\") " Oct 31 12:51:14.223525 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:14.223496 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbdd88e1-3077-465d-a515-b8b9f78ca3b7-kube-api-access-mfn6z" pod "bbdd88e1-3077-465d-a515-b8b9f78ca3b7" (UID: "bbdd88e1-3077-465d-a515-b8b9f78ca3b7"). InnerVolumeSpecName "kube-api-access-mfn6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:51:14.253670 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:14.250315 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbdd88e1-3077-465d-a515-b8b9f78ca3b7-tls-cert" pod "bbdd88e1-3077-465d-a515-b8b9f78ca3b7" (UID: "bbdd88e1-3077-465d-a515-b8b9f78ca3b7"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:51:14.322747 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:14.322708 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfn6z\" (UniqueName: \"kubernetes.io/projected/bbdd88e1-3077-465d-a515-b8b9f78ca3b7-kube-api-access-mfn6z\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:51:14.322747 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:14.322748 5693 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bbdd88e1-3077-465d-a515-b8b9f78ca3b7-tls-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:51:14.821148 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:14.821114 5693 generic.go:356] "Generic (PLEG): container finished" podID="bbdd88e1-3077-465d-a515-b8b9f78ca3b7" containerID="8155e64362b068f86866b19180d8def2d6f3bf40fe7a9bd4e18d70c6f1d71dae" exitCode=0 Oct 31 12:51:14.821665 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:14.821159 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77b4b46987-vxf8x" Oct 31 12:51:14.821665 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:14.821235 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77b4b46987-vxf8x" event={"ID":"bbdd88e1-3077-465d-a515-b8b9f78ca3b7","Type":"ContainerDied","Data":"8155e64362b068f86866b19180d8def2d6f3bf40fe7a9bd4e18d70c6f1d71dae"} Oct 31 12:51:14.821665 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:14.821275 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77b4b46987-vxf8x" event={"ID":"bbdd88e1-3077-465d-a515-b8b9f78ca3b7","Type":"ContainerDied","Data":"ab26f132cba1a80283b8b426eacdf894a46eea59322d5a5b89b1dbdf5f8eb3b4"} Oct 31 12:51:14.821665 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:14.821292 5693 scope.go:122] "RemoveContainer" containerID="8155e64362b068f86866b19180d8def2d6f3bf40fe7a9bd4e18d70c6f1d71dae" Oct 31 12:51:14.837682 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:14.837632 5693 scope.go:122] "RemoveContainer" containerID="8155e64362b068f86866b19180d8def2d6f3bf40fe7a9bd4e18d70c6f1d71dae" Oct 31 12:51:14.837986 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:51:14.837966 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8155e64362b068f86866b19180d8def2d6f3bf40fe7a9bd4e18d70c6f1d71dae\": container with ID starting with 8155e64362b068f86866b19180d8def2d6f3bf40fe7a9bd4e18d70c6f1d71dae not found: ID does not exist" containerID="8155e64362b068f86866b19180d8def2d6f3bf40fe7a9bd4e18d70c6f1d71dae" Oct 31 12:51:14.838061 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:14.837993 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8155e64362b068f86866b19180d8def2d6f3bf40fe7a9bd4e18d70c6f1d71dae"} err="failed to get container status \"8155e64362b068f86866b19180d8def2d6f3bf40fe7a9bd4e18d70c6f1d71dae\": rpc error: code = NotFound desc = could not find container \"8155e64362b068f86866b19180d8def2d6f3bf40fe7a9bd4e18d70c6f1d71dae\": container with ID starting with 8155e64362b068f86866b19180d8def2d6f3bf40fe7a9bd4e18d70c6f1d71dae not found: ID does not exist" Oct 31 12:51:14.862266 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:14.862234 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-77b4b46987-vxf8x"] Oct 31 12:51:14.871284 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:14.871259 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-77b4b46987-vxf8x"] Oct 31 12:51:15.407269 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:15.407234 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbdd88e1-3077-465d-a515-b8b9f78ca3b7" path="/var/lib/kubelet/pods/bbdd88e1-3077-465d-a515-b8b9f78ca3b7/volumes" Oct 31 12:51:16.876050 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:16.876021 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2"] Oct 31 12:51:16.876513 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:16.876374 5693 cpu_manager.go:414] "containerMap: removing container" podUID="bbdd88e1-3077-465d-a515-b8b9f78ca3b7" containerName="authorino" Oct 31 12:51:16.876513 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:16.876385 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbdd88e1-3077-465d-a515-b8b9f78ca3b7" containerName="authorino" Oct 31 12:51:16.876513 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:16.876476 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbdd88e1-3077-465d-a515-b8b9f78ca3b7" containerName="authorino" Oct 31 12:51:16.880956 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:16.880939 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" Oct 31 12:51:16.894544 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:16.894519 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 12:51:16.894673 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:16.894574 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-2bzjw\"" Oct 31 12:51:16.902779 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:16.902749 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2"] Oct 31 12:51:16.955986 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:16.955948 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f942c1e3-8982-4e26-9c30-82ac4265a1eb-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-95zp2\" (UID: \"f942c1e3-8982-4e26-9c30-82ac4265a1eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" Oct 31 12:51:16.956180 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:16.956002 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f942c1e3-8982-4e26-9c30-82ac4265a1eb-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-95zp2\" (UID: \"f942c1e3-8982-4e26-9c30-82ac4265a1eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" Oct 31 12:51:16.956180 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:16.956137 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f942c1e3-8982-4e26-9c30-82ac4265a1eb-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-95zp2\" (UID: \"f942c1e3-8982-4e26-9c30-82ac4265a1eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" Oct 31 12:51:16.956180 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:16.956164 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f942c1e3-8982-4e26-9c30-82ac4265a1eb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-95zp2\" (UID: \"f942c1e3-8982-4e26-9c30-82ac4265a1eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" Oct 31 12:51:16.956348 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:16.956192 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl46z\" (UniqueName: \"kubernetes.io/projected/f942c1e3-8982-4e26-9c30-82ac4265a1eb-kube-api-access-vl46z\") pod \"maas-default-gateway-openshift-default-fc8c56547-95zp2\" (UID: \"f942c1e3-8982-4e26-9c30-82ac4265a1eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" Oct 31 12:51:16.956348 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:16.956271 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f942c1e3-8982-4e26-9c30-82ac4265a1eb-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-95zp2\" (UID: \"f942c1e3-8982-4e26-9c30-82ac4265a1eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" Oct 31 12:51:16.956348 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:16.956289 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f942c1e3-8982-4e26-9c30-82ac4265a1eb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-95zp2\" (UID: \"f942c1e3-8982-4e26-9c30-82ac4265a1eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" Oct 31 12:51:16.956348 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:16.956305 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f942c1e3-8982-4e26-9c30-82ac4265a1eb-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-95zp2\" (UID: \"f942c1e3-8982-4e26-9c30-82ac4265a1eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" Oct 31 12:51:16.956348 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:16.956344 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f942c1e3-8982-4e26-9c30-82ac4265a1eb-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-95zp2\" (UID: \"f942c1e3-8982-4e26-9c30-82ac4265a1eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" Oct 31 12:51:17.057327 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:17.057286 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f942c1e3-8982-4e26-9c30-82ac4265a1eb-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-95zp2\" (UID: \"f942c1e3-8982-4e26-9c30-82ac4265a1eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" Oct 31 12:51:17.057327 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:17.057326 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f942c1e3-8982-4e26-9c30-82ac4265a1eb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-95zp2\" (UID: \"f942c1e3-8982-4e26-9c30-82ac4265a1eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" Oct 31 12:51:17.057569 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:17.057344 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f942c1e3-8982-4e26-9c30-82ac4265a1eb-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-95zp2\" (UID: \"f942c1e3-8982-4e26-9c30-82ac4265a1eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" Oct 31 12:51:17.057569 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:17.057494 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f942c1e3-8982-4e26-9c30-82ac4265a1eb-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-95zp2\" (UID: \"f942c1e3-8982-4e26-9c30-82ac4265a1eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" Oct 31 12:51:17.057726 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:17.057631 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f942c1e3-8982-4e26-9c30-82ac4265a1eb-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-95zp2\" (UID: \"f942c1e3-8982-4e26-9c30-82ac4265a1eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" Oct 31 12:51:17.057726 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:17.057697 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f942c1e3-8982-4e26-9c30-82ac4265a1eb-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-95zp2\" (UID: \"f942c1e3-8982-4e26-9c30-82ac4265a1eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" Oct 31 12:51:17.057836 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:17.057742 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f942c1e3-8982-4e26-9c30-82ac4265a1eb-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-95zp2\" (UID: \"f942c1e3-8982-4e26-9c30-82ac4265a1eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" Oct 31 12:51:17.057836 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:17.057774 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f942c1e3-8982-4e26-9c30-82ac4265a1eb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-95zp2\" (UID: \"f942c1e3-8982-4e26-9c30-82ac4265a1eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" Oct 31 12:51:17.057836 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:17.057806 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl46z\" (UniqueName: \"kubernetes.io/projected/f942c1e3-8982-4e26-9c30-82ac4265a1eb-kube-api-access-vl46z\") pod \"maas-default-gateway-openshift-default-fc8c56547-95zp2\" (UID: \"f942c1e3-8982-4e26-9c30-82ac4265a1eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" Oct 31 12:51:17.057985 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:17.057856 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f942c1e3-8982-4e26-9c30-82ac4265a1eb-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-95zp2\" (UID: \"f942c1e3-8982-4e26-9c30-82ac4265a1eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" Oct 31 12:51:17.058116 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:17.058087 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f942c1e3-8982-4e26-9c30-82ac4265a1eb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-95zp2\" (UID: \"f942c1e3-8982-4e26-9c30-82ac4265a1eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" Oct 31 12:51:17.058200 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:17.058100 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f942c1e3-8982-4e26-9c30-82ac4265a1eb-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-95zp2\" (UID: \"f942c1e3-8982-4e26-9c30-82ac4265a1eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" Oct 31 12:51:17.058367 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:17.058340 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f942c1e3-8982-4e26-9c30-82ac4265a1eb-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-95zp2\" (UID: \"f942c1e3-8982-4e26-9c30-82ac4265a1eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" Oct 31 12:51:17.058558 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:17.058539 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f942c1e3-8982-4e26-9c30-82ac4265a1eb-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-95zp2\" (UID: \"f942c1e3-8982-4e26-9c30-82ac4265a1eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" Oct 31 12:51:17.060355 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:17.060334 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f942c1e3-8982-4e26-9c30-82ac4265a1eb-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-95zp2\" (UID: \"f942c1e3-8982-4e26-9c30-82ac4265a1eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" Oct 31 12:51:17.060469 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:17.060447 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f942c1e3-8982-4e26-9c30-82ac4265a1eb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-95zp2\" (UID: \"f942c1e3-8982-4e26-9c30-82ac4265a1eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" Oct 31 12:51:17.070704 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:17.070675 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f942c1e3-8982-4e26-9c30-82ac4265a1eb-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-95zp2\" (UID: \"f942c1e3-8982-4e26-9c30-82ac4265a1eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" Oct 31 12:51:17.070704 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:17.070698 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl46z\" (UniqueName: \"kubernetes.io/projected/f942c1e3-8982-4e26-9c30-82ac4265a1eb-kube-api-access-vl46z\") pod \"maas-default-gateway-openshift-default-fc8c56547-95zp2\" (UID: \"f942c1e3-8982-4e26-9c30-82ac4265a1eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" Oct 31 12:51:17.193833 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:17.193793 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" Oct 31 12:51:17.476172 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:17.476092 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2"] Oct 31 12:51:17.479544 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:51:17.479516 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf942c1e3_8982_4e26_9c30_82ac4265a1eb.slice/crio-5b85839dc84f55a4fc166cdc413ff73099acf703a0a14828d5a0d0b31a38b927 WatchSource:0}: Error finding container 5b85839dc84f55a4fc166cdc413ff73099acf703a0a14828d5a0d0b31a38b927: Status 404 returned error can't find the container with id 5b85839dc84f55a4fc166cdc413ff73099acf703a0a14828d5a0d0b31a38b927 Oct 31 12:51:17.839084 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:17.838994 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" event={"ID":"f942c1e3-8982-4e26-9c30-82ac4265a1eb","Type":"ContainerStarted","Data":"5b85839dc84f55a4fc166cdc413ff73099acf703a0a14828d5a0d0b31a38b927"} Oct 31 12:51:18.686826 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:51:18.686732 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:51:22.853223 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:22.853190 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:51:22.853517 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:22.853259 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:51:22.853517 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:22.853293 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:51:23.871702 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:23.871669 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" event={"ID":"f942c1e3-8982-4e26-9c30-82ac4265a1eb","Type":"ContainerStarted","Data":"bc869e7d816188134bfbdd752555eb99c453aab2851a958f9a6655add35e239f"} Oct 31 12:51:23.918227 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:23.918171 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" podStartSLOduration=2.5473976069999997 podStartE2EDuration="7.918159387s" totalImagesPullingTime="5.37076178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:51:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:51:23.916750254 +0000 UTC m=+1134.624336570" watchObservedRunningTime="2026-10-31 12:51:23.918159387 +0000 UTC m=+1134.625745681" Oct 31 12:51:24.197686 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:24.197630 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" Oct 31 12:51:24.198046 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:24.198023 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-95zp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 31 12:51:24.198248 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:24.198224 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" podUID="f942c1e3-8982-4e26-9c30-82ac4265a1eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 31 12:51:25.196136 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:25.196092 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-95zp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 31 12:51:25.196797 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:25.196766 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" podUID="f942c1e3-8982-4e26-9c30-82ac4265a1eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 31 12:51:26.197132 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:26.197089 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-95zp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 31 12:51:26.197771 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:26.197736 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" podUID="f942c1e3-8982-4e26-9c30-82ac4265a1eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 31 12:51:27.196665 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:27.196109 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-95zp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 31 12:51:27.196665 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:27.196186 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" podUID="f942c1e3-8982-4e26-9c30-82ac4265a1eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 31 12:51:28.199667 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:28.195149 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-95zp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 31 12:51:28.199667 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:28.195215 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" podUID="f942c1e3-8982-4e26-9c30-82ac4265a1eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 31 12:51:29.194821 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:29.194788 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-95zp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 31 12:51:29.195006 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:29.194866 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" podUID="f942c1e3-8982-4e26-9c30-82ac4265a1eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 31 12:51:30.195277 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:30.195234 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-95zp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 31 12:51:30.195694 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:30.195307 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" podUID="f942c1e3-8982-4e26-9c30-82ac4265a1eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 31 12:51:31.198560 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:31.198532 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" Oct 31 12:51:31.198982 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:31.198832 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" Oct 31 12:51:31.199683 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:31.199664 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" Oct 31 12:51:31.944074 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:31.944039 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv"] Oct 31 12:51:31.949672 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:31.949013 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" Oct 31 12:51:31.977294 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:31.977264 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv"] Oct 31 12:51:32.007361 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.007325 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vjvb\" (UniqueName: \"kubernetes.io/projected/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-kube-api-access-5vjvb\") pod \"maas-default-gateway-openshift-default-fc8c56547-rskhv\" (UID: \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" Oct 31 12:51:32.007612 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.007410 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-rskhv\" (UID: \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" Oct 31 12:51:32.007612 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.007451 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-rskhv\" (UID: \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" Oct 31 12:51:32.007612 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.007481 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-rskhv\" (UID: \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" Oct 31 12:51:32.007612 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.007511 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-rskhv\" (UID: \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" Oct 31 12:51:32.007612 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.007570 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-rskhv\" (UID: \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" Oct 31 12:51:32.007612 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.007608 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-rskhv\" (UID: \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" Oct 31 12:51:32.007999 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.007723 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-rskhv\" (UID: \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" Oct 31 12:51:32.008044 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.007995 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-rskhv\" (UID: \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" Oct 31 12:51:32.010731 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.010709 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482"] Oct 31 12:51:32.014881 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.014864 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" Oct 31 12:51:32.028698 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.028674 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482"] Oct 31 12:51:32.109270 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.109237 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vjvb\" (UniqueName: \"kubernetes.io/projected/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-kube-api-access-5vjvb\") pod \"maas-default-gateway-openshift-default-fc8c56547-rskhv\" (UID: \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" Oct 31 12:51:32.109477 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.109277 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0bdd69c-691b-47b2-ba17-2e0ac7185643-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-fv482\" (UID: \"c0bdd69c-691b-47b2-ba17-2e0ac7185643\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" Oct 31 12:51:32.109477 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.109302 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-rskhv\" (UID: \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" Oct 31 12:51:32.109477 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.109410 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-rskhv\" (UID: \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" Oct 31 12:51:32.109477 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.109434 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0bdd69c-691b-47b2-ba17-2e0ac7185643-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-fv482\" (UID: \"c0bdd69c-691b-47b2-ba17-2e0ac7185643\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" Oct 31 12:51:32.109477 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.109462 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-rskhv\" (UID: \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" Oct 31 12:51:32.109767 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.109495 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-rskhv\" (UID: \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" Oct 31 12:51:32.109767 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.109595 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-rskhv\" (UID: \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" Oct 31 12:51:32.109767 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.109683 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0bdd69c-691b-47b2-ba17-2e0ac7185643-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-fv482\" (UID: \"c0bdd69c-691b-47b2-ba17-2e0ac7185643\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" Oct 31 12:51:32.109767 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.109719 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-rskhv\" (UID: \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" Oct 31 12:51:32.109767 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.109757 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0bdd69c-691b-47b2-ba17-2e0ac7185643-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-fv482\" (UID: \"c0bdd69c-691b-47b2-ba17-2e0ac7185643\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" Oct 31 12:51:32.110014 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.109775 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-rskhv\" (UID: \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" Oct 31 12:51:32.110014 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.109820 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-rskhv\" (UID: \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" Oct 31 12:51:32.110014 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.109888 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-rskhv\" (UID: \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" Oct 31 12:51:32.110014 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.109900 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnvpf\" (UniqueName: \"kubernetes.io/projected/c0bdd69c-691b-47b2-ba17-2e0ac7185643-kube-api-access-gnvpf\") pod \"maas-default-gateway-openshift-default-fc8c56547-fv482\" (UID: \"c0bdd69c-691b-47b2-ba17-2e0ac7185643\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" Oct 31 12:51:32.110014 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.109948 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0bdd69c-691b-47b2-ba17-2e0ac7185643-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-fv482\" (UID: \"c0bdd69c-691b-47b2-ba17-2e0ac7185643\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" Oct 31 12:51:32.110014 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.109982 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0bdd69c-691b-47b2-ba17-2e0ac7185643-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-fv482\" (UID: \"c0bdd69c-691b-47b2-ba17-2e0ac7185643\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" Oct 31 12:51:32.110232 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.110035 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0bdd69c-691b-47b2-ba17-2e0ac7185643-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-fv482\" (UID: \"c0bdd69c-691b-47b2-ba17-2e0ac7185643\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" Oct 31 12:51:32.110232 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.110075 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-rskhv\" (UID: \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" Oct 31 12:51:32.110232 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.110110 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0bdd69c-691b-47b2-ba17-2e0ac7185643-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-fv482\" (UID: \"c0bdd69c-691b-47b2-ba17-2e0ac7185643\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" Oct 31 12:51:32.110232 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.110155 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-rskhv\" (UID: \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" Oct 31 12:51:32.110232 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.110206 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-rskhv\" (UID: \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" Oct 31 12:51:32.110412 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.110336 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-rskhv\" (UID: \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" Oct 31 12:51:32.112049 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.112024 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-rskhv\" (UID: \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" Oct 31 12:51:32.112494 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.112475 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-rskhv\" (UID: \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" Oct 31 12:51:32.122781 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.122760 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-rskhv\" (UID: \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" Oct 31 12:51:32.122916 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.122805 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vjvb\" (UniqueName: \"kubernetes.io/projected/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-kube-api-access-5vjvb\") pod \"maas-default-gateway-openshift-default-fc8c56547-rskhv\" (UID: \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" Oct 31 12:51:32.211465 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.211377 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0bdd69c-691b-47b2-ba17-2e0ac7185643-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-fv482\" (UID: \"c0bdd69c-691b-47b2-ba17-2e0ac7185643\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" Oct 31 12:51:32.211465 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.211446 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0bdd69c-691b-47b2-ba17-2e0ac7185643-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-fv482\" (UID: \"c0bdd69c-691b-47b2-ba17-2e0ac7185643\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" Oct 31 12:51:32.212005 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.211479 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0bdd69c-691b-47b2-ba17-2e0ac7185643-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-fv482\" (UID: \"c0bdd69c-691b-47b2-ba17-2e0ac7185643\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" Oct 31 12:51:32.212005 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.211543 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0bdd69c-691b-47b2-ba17-2e0ac7185643-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-fv482\" (UID: \"c0bdd69c-691b-47b2-ba17-2e0ac7185643\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" Oct 31 12:51:32.212005 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.211574 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0bdd69c-691b-47b2-ba17-2e0ac7185643-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-fv482\" (UID: \"c0bdd69c-691b-47b2-ba17-2e0ac7185643\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" Oct 31 12:51:32.212005 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.211613 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnvpf\" (UniqueName: \"kubernetes.io/projected/c0bdd69c-691b-47b2-ba17-2e0ac7185643-kube-api-access-gnvpf\") pod \"maas-default-gateway-openshift-default-fc8c56547-fv482\" (UID: \"c0bdd69c-691b-47b2-ba17-2e0ac7185643\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" Oct 31 12:51:32.212005 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.211825 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0bdd69c-691b-47b2-ba17-2e0ac7185643-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-fv482\" (UID: \"c0bdd69c-691b-47b2-ba17-2e0ac7185643\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" Oct 31 12:51:32.212005 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.211854 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0bdd69c-691b-47b2-ba17-2e0ac7185643-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-fv482\" (UID: \"c0bdd69c-691b-47b2-ba17-2e0ac7185643\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" Oct 31 12:51:32.212005 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.211907 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0bdd69c-691b-47b2-ba17-2e0ac7185643-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-fv482\" (UID: \"c0bdd69c-691b-47b2-ba17-2e0ac7185643\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" Oct 31 12:51:32.212005 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.211857 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0bdd69c-691b-47b2-ba17-2e0ac7185643-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-fv482\" (UID: \"c0bdd69c-691b-47b2-ba17-2e0ac7185643\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" Oct 31 12:51:32.212005 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.211861 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0bdd69c-691b-47b2-ba17-2e0ac7185643-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-fv482\" (UID: \"c0bdd69c-691b-47b2-ba17-2e0ac7185643\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" Oct 31 12:51:32.212005 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.211991 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0bdd69c-691b-47b2-ba17-2e0ac7185643-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-fv482\" (UID: \"c0bdd69c-691b-47b2-ba17-2e0ac7185643\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" Oct 31 12:51:32.212351 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.212237 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0bdd69c-691b-47b2-ba17-2e0ac7185643-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-fv482\" (UID: \"c0bdd69c-691b-47b2-ba17-2e0ac7185643\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" Oct 31 12:51:32.212351 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.212304 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0bdd69c-691b-47b2-ba17-2e0ac7185643-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-fv482\" (UID: \"c0bdd69c-691b-47b2-ba17-2e0ac7185643\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" Oct 31 12:51:32.214090 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.214073 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0bdd69c-691b-47b2-ba17-2e0ac7185643-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-fv482\" (UID: \"c0bdd69c-691b-47b2-ba17-2e0ac7185643\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" Oct 31 12:51:32.214338 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.214322 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0bdd69c-691b-47b2-ba17-2e0ac7185643-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-fv482\" (UID: \"c0bdd69c-691b-47b2-ba17-2e0ac7185643\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" Oct 31 12:51:32.224675 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.224631 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnvpf\" (UniqueName: \"kubernetes.io/projected/c0bdd69c-691b-47b2-ba17-2e0ac7185643-kube-api-access-gnvpf\") pod \"maas-default-gateway-openshift-default-fc8c56547-fv482\" (UID: \"c0bdd69c-691b-47b2-ba17-2e0ac7185643\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" Oct 31 12:51:32.224675 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.224670 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0bdd69c-691b-47b2-ba17-2e0ac7185643-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-fv482\" (UID: \"c0bdd69c-691b-47b2-ba17-2e0ac7185643\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" Oct 31 12:51:32.262750 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.262724 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" Oct 31 12:51:32.327903 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.327701 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" Oct 31 12:51:32.541585 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.541511 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv"] Oct 31 12:51:32.545302 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:51:32.545273 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae02ff0_1e33_4007_a97a_63a61e8b4bf4.slice/crio-d4d44ad53932b3b1f32b24881998a24fa335ca0dcd161bcade3ba935f1e36732 WatchSource:0}: Error finding container d4d44ad53932b3b1f32b24881998a24fa335ca0dcd161bcade3ba935f1e36732: Status 404 returned error can't find the container with id d4d44ad53932b3b1f32b24881998a24fa335ca0dcd161bcade3ba935f1e36732 Oct 31 12:51:32.547524 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.547494 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:51:32.547586 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.547557 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:51:32.547622 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.547585 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:51:32.610135 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.610107 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482"] Oct 31 12:51:32.612854 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:51:32.612833 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0bdd69c_691b_47b2_ba17_2e0ac7185643.slice/crio-7f74f3fb08843b2d36f5e005778864fd966f4bf47408b9b1410fe66e724a0517 WatchSource:0}: Error finding container 7f74f3fb08843b2d36f5e005778864fd966f4bf47408b9b1410fe66e724a0517: Status 404 returned error can't find the container with id 7f74f3fb08843b2d36f5e005778864fd966f4bf47408b9b1410fe66e724a0517 Oct 31 12:51:32.615610 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.615580 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:51:32.615746 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.615673 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:51:32.615746 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.615722 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:51:32.942974 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.942937 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" event={"ID":"c0bdd69c-691b-47b2-ba17-2e0ac7185643","Type":"ContainerStarted","Data":"a8dee1033738d29599f24851479d95d461665391b87fb7d5aa8de6e7e3609bed"} Oct 31 12:51:32.943172 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.943001 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" event={"ID":"c0bdd69c-691b-47b2-ba17-2e0ac7185643","Type":"ContainerStarted","Data":"7f74f3fb08843b2d36f5e005778864fd966f4bf47408b9b1410fe66e724a0517"} Oct 31 12:51:32.944688 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.944658 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" event={"ID":"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4","Type":"ContainerStarted","Data":"98ad7ee33e380455fc042048fdba1e1b9f1c9e0ff0aa2f2ee06f8385255781d0"} Oct 31 12:51:32.944822 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.944712 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" event={"ID":"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4","Type":"ContainerStarted","Data":"d4d44ad53932b3b1f32b24881998a24fa335ca0dcd161bcade3ba935f1e36732"} Oct 31 12:51:32.967551 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:32.967485 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" podStartSLOduration=1.9674700920000001 podStartE2EDuration="1.967470092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:51:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:51:32.96682488 +0000 UTC m=+1143.674411176" watchObservedRunningTime="2026-10-31 12:51:32.967470092 +0000 UTC m=+1143.675056385" Oct 31 12:51:33.263554 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:33.263456 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" Oct 31 12:51:33.265095 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:33.265071 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-rskhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 31 12:51:33.265217 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:33.265118 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" podUID="2ae02ff0-1e33-4007-a97a-63a61e8b4bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 31 12:51:33.329008 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:33.328959 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" Oct 31 12:51:33.330447 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:33.330425 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fv482 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 31 12:51:33.330532 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:33.330472 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" podUID="c0bdd69c-691b-47b2-ba17-2e0ac7185643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 31 12:51:34.263993 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:34.263954 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-rskhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 31 12:51:34.264491 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:34.264032 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" podUID="2ae02ff0-1e33-4007-a97a-63a61e8b4bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 31 12:51:34.329096 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:34.329056 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fv482 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 31 12:51:34.329274 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:34.329121 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" podUID="c0bdd69c-691b-47b2-ba17-2e0ac7185643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 31 12:51:35.263436 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:35.263398 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-rskhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 31 12:51:35.263595 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:35.263459 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" podUID="2ae02ff0-1e33-4007-a97a-63a61e8b4bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 31 12:51:35.328412 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:35.328372 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fv482 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 31 12:51:35.328821 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:35.328424 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" podUID="c0bdd69c-691b-47b2-ba17-2e0ac7185643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 31 12:51:36.264022 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:36.263986 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-rskhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 31 12:51:36.264180 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:36.264037 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" podUID="2ae02ff0-1e33-4007-a97a-63a61e8b4bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 31 12:51:36.331040 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:36.330998 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fv482 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 31 12:51:36.331433 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:36.331064 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" podUID="c0bdd69c-691b-47b2-ba17-2e0ac7185643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 31 12:51:37.264134 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:37.264096 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-rskhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 31 12:51:37.264321 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:37.264161 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" podUID="2ae02ff0-1e33-4007-a97a-63a61e8b4bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 31 12:51:37.328370 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:37.328320 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fv482 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 31 12:51:37.328546 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:37.328388 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" podUID="c0bdd69c-691b-47b2-ba17-2e0ac7185643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 31 12:51:38.263509 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:38.263468 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-rskhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 31 12:51:38.264025 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:38.263523 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" podUID="2ae02ff0-1e33-4007-a97a-63a61e8b4bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 31 12:51:38.328318 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:38.328273 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fv482 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 31 12:51:38.328499 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:38.328326 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" podUID="c0bdd69c-691b-47b2-ba17-2e0ac7185643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 31 12:51:39.264106 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:39.264070 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-rskhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 31 12:51:39.264618 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:39.264125 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" podUID="2ae02ff0-1e33-4007-a97a-63a61e8b4bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 31 12:51:39.328978 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:39.328936 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fv482 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 31 12:51:39.329160 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:39.329001 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" podUID="c0bdd69c-691b-47b2-ba17-2e0ac7185643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 31 12:51:40.802958 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:40.802928 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" Oct 31 12:51:40.803421 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:40.803226 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" Oct 31 12:51:40.804235 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:40.804216 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" Oct 31 12:51:40.827540 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:40.827471 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" podStartSLOduration=9.827454743 podStartE2EDuration="9.827454743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:51:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:51:32.996820626 +0000 UTC m=+1143.704406921" watchObservedRunningTime="2026-10-31 12:51:40.827454743 +0000 UTC m=+1151.535041040" Oct 31 12:51:40.920390 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:40.920348 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" Oct 31 12:51:40.921687 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:40.921662 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" Oct 31 12:51:40.922408 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:40.922387 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" Oct 31 12:51:42.829285 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:42.829249 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck"] Oct 31 12:51:42.833679 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:42.833657 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" Oct 31 12:51:42.839798 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:42.839777 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-cf7e6604-gw-openshift-default-dockercfg-rkv8g\"" Oct 31 12:51:42.862304 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:42.862281 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck"] Oct 31 12:51:42.920075 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:42.920046 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-istio-data\") pod \"e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck\" (UID: \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\") " pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" Oct 31 12:51:42.920260 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:42.920095 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-istiod-ca-cert\") pod \"e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck\" (UID: \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\") " pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" Oct 31 12:51:42.920260 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:42.920119 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shl86\" (UniqueName: \"kubernetes.io/projected/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-kube-api-access-shl86\") pod \"e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck\" (UID: \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\") " pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" Oct 31 12:51:42.920260 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:42.920220 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-istio-envoy\") pod \"e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck\" (UID: \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\") " pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" Oct 31 12:51:42.920423 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:42.920265 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-credential-socket\") pod \"e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck\" (UID: \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\") " pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" Oct 31 12:51:42.920423 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:42.920290 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-istio-podinfo\") pod \"e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck\" (UID: \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\") " pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" Oct 31 12:51:42.920423 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:42.920327 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-istio-token\") pod \"e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck\" (UID: \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\") " pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" Oct 31 12:51:42.920423 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:42.920379 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-workload-socket\") pod \"e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck\" (UID: \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\") " pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" Oct 31 12:51:42.920423 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:42.920403 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-workload-certs\") pod \"e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck\" (UID: \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\") " pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" Oct 31 12:51:43.021226 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:43.021194 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-workload-socket\") pod \"e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck\" (UID: \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\") " pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" Oct 31 12:51:43.021226 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:43.021233 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-workload-certs\") pod \"e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck\" (UID: \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\") " pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" Oct 31 12:51:43.021476 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:43.021291 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-istio-data\") pod \"e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck\" (UID: \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\") " pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" Oct 31 12:51:43.021476 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:43.021321 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-istiod-ca-cert\") pod \"e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck\" (UID: \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\") " pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" Oct 31 12:51:43.021476 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:43.021337 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shl86\" (UniqueName: \"kubernetes.io/projected/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-kube-api-access-shl86\") pod \"e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck\" (UID: \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\") " pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" Oct 31 12:51:43.021476 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:43.021355 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-istio-envoy\") pod \"e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck\" (UID: \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\") " pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" Oct 31 12:51:43.021476 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:43.021372 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-credential-socket\") pod \"e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck\" (UID: \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\") " pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" Oct 31 12:51:43.021761 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:43.021493 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-istio-podinfo\") pod \"e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck\" (UID: \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\") " pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" Oct 31 12:51:43.021761 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:43.021545 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-istio-token\") pod \"e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck\" (UID: \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\") " pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" Oct 31 12:51:43.021761 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:43.021731 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-workload-socket\") pod \"e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck\" (UID: \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\") " pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" Oct 31 12:51:43.021929 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:43.021765 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-workload-certs\") pod \"e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck\" (UID: \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\") " pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" Oct 31 12:51:43.021995 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:43.021975 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-credential-socket\") pod \"e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck\" (UID: \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\") " pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" Oct 31 12:51:43.022094 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:43.022075 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-istiod-ca-cert\") pod \"e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck\" (UID: \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\") " pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" Oct 31 12:51:43.022222 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:43.022204 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-istio-data\") pod \"e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck\" (UID: \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\") " pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" Oct 31 12:51:43.024016 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:43.023996 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-istio-envoy\") pod \"e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck\" (UID: \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\") " pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" Oct 31 12:51:43.024079 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:43.024063 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-istio-podinfo\") pod \"e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck\" (UID: \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\") " pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" Oct 31 12:51:43.039680 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:43.039634 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-istio-token\") pod \"e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck\" (UID: \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\") " pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" Oct 31 12:51:43.039798 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:43.039783 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shl86\" (UniqueName: \"kubernetes.io/projected/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-kube-api-access-shl86\") pod \"e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck\" (UID: \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\") " pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" Oct 31 12:51:43.146839 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:43.146808 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" Oct 31 12:51:43.434205 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:43.434054 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck"] Oct 31 12:51:43.436761 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:51:43.436733 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc14cc0b_8b70_4999_b4b4_f9a0aa1f5d5c.slice/crio-4a2eaa4b25c059a7b9270f6c771e9c9671a6acea70cf483dd30df6ee9316c13d WatchSource:0}: Error finding container 4a2eaa4b25c059a7b9270f6c771e9c9671a6acea70cf483dd30df6ee9316c13d: Status 404 returned error can't find the container with id 4a2eaa4b25c059a7b9270f6c771e9c9671a6acea70cf483dd30df6ee9316c13d Oct 31 12:51:43.438883 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:43.438849 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:51:43.438937 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:43.438916 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:51:43.438968 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:43.438944 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:51:43.999365 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:43.999334 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" event={"ID":"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c","Type":"ContainerStarted","Data":"84ecd081bcf7eacf2460045d2e9335c3865d981dce932ba67c9668bbb9472e82"} Oct 31 12:51:43.999903 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:43.999397 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" event={"ID":"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c","Type":"ContainerStarted","Data":"4a2eaa4b25c059a7b9270f6c771e9c9671a6acea70cf483dd30df6ee9316c13d"} Oct 31 12:51:44.026123 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:44.025923 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" podStartSLOduration=2.025906121 podStartE2EDuration="2.025906121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:51:44.025349698 +0000 UTC m=+1154.732935996" watchObservedRunningTime="2026-10-31 12:51:44.025906121 +0000 UTC m=+1154.733492417" Oct 31 12:51:44.078368 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:44.078334 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn"] Oct 31 12:51:44.082715 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:44.082691 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" Oct 31 12:51:44.090916 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:44.090895 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-1abc54b4-openshift-default-dockercfg-sw4hj\"" Oct 31 12:51:44.095676 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:44.095632 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn"] Oct 31 12:51:44.134029 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:44.133997 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48005f27-24cb-4a42-8b06-51a2254c3473-workload-socket\") pod \"e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn\" (UID: \"48005f27-24cb-4a42-8b06-51a2254c3473\") " pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" Oct 31 12:51:44.134241 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:44.134225 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48005f27-24cb-4a42-8b06-51a2254c3473-istiod-ca-cert\") pod \"e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn\" (UID: \"48005f27-24cb-4a42-8b06-51a2254c3473\") " pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" Oct 31 12:51:44.134310 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:44.134273 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48005f27-24cb-4a42-8b06-51a2254c3473-istio-token\") pod \"e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn\" (UID: \"48005f27-24cb-4a42-8b06-51a2254c3473\") " pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" Oct 31 12:51:44.134401 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:44.134382 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48005f27-24cb-4a42-8b06-51a2254c3473-istio-podinfo\") pod \"e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn\" (UID: \"48005f27-24cb-4a42-8b06-51a2254c3473\") " pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" Oct 31 12:51:44.134455 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:44.134412 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48005f27-24cb-4a42-8b06-51a2254c3473-workload-certs\") pod \"e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn\" (UID: \"48005f27-24cb-4a42-8b06-51a2254c3473\") " pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" Oct 31 12:51:44.134560 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:44.134544 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48005f27-24cb-4a42-8b06-51a2254c3473-istio-envoy\") pod \"e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn\" (UID: \"48005f27-24cb-4a42-8b06-51a2254c3473\") " pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" Oct 31 12:51:44.134626 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:44.134580 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48005f27-24cb-4a42-8b06-51a2254c3473-credential-socket\") pod \"e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn\" (UID: \"48005f27-24cb-4a42-8b06-51a2254c3473\") " pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" Oct 31 12:51:44.134714 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:44.134622 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48005f27-24cb-4a42-8b06-51a2254c3473-istio-data\") pod \"e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn\" (UID: \"48005f27-24cb-4a42-8b06-51a2254c3473\") " pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" Oct 31 12:51:44.134714 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:44.134677 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwqx6\" (UniqueName: \"kubernetes.io/projected/48005f27-24cb-4a42-8b06-51a2254c3473-kube-api-access-qwqx6\") pod \"e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn\" (UID: \"48005f27-24cb-4a42-8b06-51a2254c3473\") " pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" Oct 31 12:51:44.147860 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:44.147834 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" Oct 31 12:51:44.152782 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:44.152761 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" Oct 31 12:51:44.235854 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:44.235818 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48005f27-24cb-4a42-8b06-51a2254c3473-istio-podinfo\") pod \"e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn\" (UID: \"48005f27-24cb-4a42-8b06-51a2254c3473\") " pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" Oct 31 12:51:44.235854 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:44.235859 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48005f27-24cb-4a42-8b06-51a2254c3473-workload-certs\") pod \"e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn\" (UID: \"48005f27-24cb-4a42-8b06-51a2254c3473\") " pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" Oct 31 12:51:44.236114 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:44.235951 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48005f27-24cb-4a42-8b06-51a2254c3473-istio-envoy\") pod \"e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn\" (UID: \"48005f27-24cb-4a42-8b06-51a2254c3473\") " pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" Oct 31 12:51:44.236114 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:44.235981 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48005f27-24cb-4a42-8b06-51a2254c3473-credential-socket\") pod \"e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn\" (UID: \"48005f27-24cb-4a42-8b06-51a2254c3473\") " pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" Oct 31 12:51:44.236114 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:44.236017 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48005f27-24cb-4a42-8b06-51a2254c3473-istio-data\") pod \"e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn\" (UID: \"48005f27-24cb-4a42-8b06-51a2254c3473\") " pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" Oct 31 12:51:44.236114 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:44.236039 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwqx6\" (UniqueName: \"kubernetes.io/projected/48005f27-24cb-4a42-8b06-51a2254c3473-kube-api-access-qwqx6\") pod \"e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn\" (UID: \"48005f27-24cb-4a42-8b06-51a2254c3473\") " pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" Oct 31 12:51:44.236114 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:44.236083 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48005f27-24cb-4a42-8b06-51a2254c3473-workload-socket\") pod \"e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn\" (UID: \"48005f27-24cb-4a42-8b06-51a2254c3473\") " pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" Oct 31 12:51:44.236362 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:44.236161 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48005f27-24cb-4a42-8b06-51a2254c3473-istiod-ca-cert\") pod \"e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn\" (UID: \"48005f27-24cb-4a42-8b06-51a2254c3473\") " pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" Oct 31 12:51:44.236362 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:44.236204 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48005f27-24cb-4a42-8b06-51a2254c3473-istio-token\") pod \"e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn\" (UID: \"48005f27-24cb-4a42-8b06-51a2254c3473\") " pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" Oct 31 12:51:44.236462 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:44.236354 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48005f27-24cb-4a42-8b06-51a2254c3473-workload-certs\") pod \"e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn\" (UID: \"48005f27-24cb-4a42-8b06-51a2254c3473\") " pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" Oct 31 12:51:44.236462 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:44.236382 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48005f27-24cb-4a42-8b06-51a2254c3473-credential-socket\") pod \"e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn\" (UID: \"48005f27-24cb-4a42-8b06-51a2254c3473\") " pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" Oct 31 12:51:44.236462 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:44.236434 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48005f27-24cb-4a42-8b06-51a2254c3473-istio-data\") pod \"e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn\" (UID: \"48005f27-24cb-4a42-8b06-51a2254c3473\") " pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" Oct 31 12:51:44.236682 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:44.236635 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48005f27-24cb-4a42-8b06-51a2254c3473-workload-socket\") pod \"e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn\" (UID: \"48005f27-24cb-4a42-8b06-51a2254c3473\") " pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" Oct 31 12:51:44.236956 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:44.236936 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48005f27-24cb-4a42-8b06-51a2254c3473-istiod-ca-cert\") pod \"e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn\" (UID: \"48005f27-24cb-4a42-8b06-51a2254c3473\") " pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" Oct 31 12:51:44.238313 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:44.238293 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48005f27-24cb-4a42-8b06-51a2254c3473-istio-envoy\") pod \"e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn\" (UID: \"48005f27-24cb-4a42-8b06-51a2254c3473\") " pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" Oct 31 12:51:44.238471 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:44.238453 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48005f27-24cb-4a42-8b06-51a2254c3473-istio-podinfo\") pod \"e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn\" (UID: \"48005f27-24cb-4a42-8b06-51a2254c3473\") " pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" Oct 31 12:51:44.249606 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:44.249548 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48005f27-24cb-4a42-8b06-51a2254c3473-istio-token\") pod \"e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn\" (UID: \"48005f27-24cb-4a42-8b06-51a2254c3473\") " pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" Oct 31 12:51:44.262982 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:44.262958 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwqx6\" (UniqueName: \"kubernetes.io/projected/48005f27-24cb-4a42-8b06-51a2254c3473-kube-api-access-qwqx6\") pod \"e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn\" (UID: \"48005f27-24cb-4a42-8b06-51a2254c3473\") " pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" Oct 31 12:51:44.399503 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:44.399468 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" Oct 31 12:51:44.685964 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:44.685918 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn"] Oct 31 12:51:44.688158 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:51:44.688127 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48005f27_24cb_4a42_8b06_51a2254c3473.slice/crio-86b24648ee7948ce4f24637c82b41677182e281db6fc2e0849032eb6303ec426 WatchSource:0}: Error finding container 86b24648ee7948ce4f24637c82b41677182e281db6fc2e0849032eb6303ec426: Status 404 returned error can't find the container with id 86b24648ee7948ce4f24637c82b41677182e281db6fc2e0849032eb6303ec426 Oct 31 12:51:44.690852 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:44.690820 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:51:44.690939 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:44.690877 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:51:44.690939 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:44.690907 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:51:45.007759 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:45.007673 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" event={"ID":"48005f27-24cb-4a42-8b06-51a2254c3473","Type":"ContainerStarted","Data":"14c0ec5e0c98aaf83cfbadb497ce2a77443d5fdf97d11398cb6ae7de709e3699"} Oct 31 12:51:45.007759 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:45.007729 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" event={"ID":"48005f27-24cb-4a42-8b06-51a2254c3473","Type":"ContainerStarted","Data":"86b24648ee7948ce4f24637c82b41677182e281db6fc2e0849032eb6303ec426"} Oct 31 12:51:45.008225 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:45.007897 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" Oct 31 12:51:45.008893 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:45.008871 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" Oct 31 12:51:45.036506 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:45.036455 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" podStartSLOduration=1.036437006 podStartE2EDuration="1.036437006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:51:45.033833196 +0000 UTC m=+1155.741419508" watchObservedRunningTime="2026-10-31 12:51:45.036437006 +0000 UTC m=+1155.744023303" Oct 31 12:51:45.401861 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:45.401831 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" Oct 31 12:51:45.410972 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:45.410936 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" Oct 31 12:51:46.012977 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:46.012947 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" Oct 31 12:51:46.013955 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:46.013932 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" Oct 31 12:51:47.005694 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:47.005625 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb"] Oct 31 12:51:47.010115 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:47.010089 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb" Oct 31 12:51:47.030940 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:47.030919 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-cf7e6604-serving-cert\"" Oct 31 12:51:47.038448 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:47.038422 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb"] Oct 31 12:51:47.064260 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:47.064229 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac16366-4faa-45e8-89a2-8b5c82f4bf29-maas-api-tls\") pod \"maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb\" (UID: \"0ac16366-4faa-45e8-89a2-8b5c82f4bf29\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb" Oct 31 12:51:47.064408 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:47.064287 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ac16366-4faa-45e8-89a2-8b5c82f4bf29-metrics-certs\") pod \"maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb\" (UID: \"0ac16366-4faa-45e8-89a2-8b5c82f4bf29\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb" Oct 31 12:51:47.064445 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:47.064412 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac16366-4faa-45e8-89a2-8b5c82f4bf29-openshift-service-ca\") pod \"maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb\" (UID: \"0ac16366-4faa-45e8-89a2-8b5c82f4bf29\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb" Oct 31 12:51:47.064476 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:47.064452 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqdt5\" (UniqueName: \"kubernetes.io/projected/0ac16366-4faa-45e8-89a2-8b5c82f4bf29-kube-api-access-tqdt5\") pod \"maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb\" (UID: \"0ac16366-4faa-45e8-89a2-8b5c82f4bf29\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb" Oct 31 12:51:47.166221 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:47.166181 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqdt5\" (UniqueName: \"kubernetes.io/projected/0ac16366-4faa-45e8-89a2-8b5c82f4bf29-kube-api-access-tqdt5\") pod \"maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb\" (UID: \"0ac16366-4faa-45e8-89a2-8b5c82f4bf29\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb" Oct 31 12:51:47.167233 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:47.167202 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac16366-4faa-45e8-89a2-8b5c82f4bf29-maas-api-tls\") pod \"maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb\" (UID: \"0ac16366-4faa-45e8-89a2-8b5c82f4bf29\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb" Oct 31 12:51:47.168000 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:47.167289 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ac16366-4faa-45e8-89a2-8b5c82f4bf29-metrics-certs\") pod \"maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb\" (UID: \"0ac16366-4faa-45e8-89a2-8b5c82f4bf29\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb" Oct 31 12:51:47.168000 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:47.167421 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac16366-4faa-45e8-89a2-8b5c82f4bf29-openshift-service-ca\") pod \"maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb\" (UID: \"0ac16366-4faa-45e8-89a2-8b5c82f4bf29\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb" Oct 31 12:51:47.168000 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:51:47.167740 5693 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-cf7e6604-serving-cert: secret "maas-api-e2e-migrate-cf7e6604-serving-cert" not found Oct 31 12:51:47.168000 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:51:47.167819 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ac16366-4faa-45e8-89a2-8b5c82f4bf29-maas-api-tls podName:0ac16366-4faa-45e8-89a2-8b5c82f4bf29 nodeName:}" failed. No retries permitted until 2026-10-31 12:51:47.667791603 +0000 UTC m=+1158.375377887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0ac16366-4faa-45e8-89a2-8b5c82f4bf29-maas-api-tls") pod "maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb" (UID: "0ac16366-4faa-45e8-89a2-8b5c82f4bf29") : secret "maas-api-e2e-migrate-cf7e6604-serving-cert" not found Oct 31 12:51:47.168428 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:47.168357 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac16366-4faa-45e8-89a2-8b5c82f4bf29-openshift-service-ca\") pod \"maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb\" (UID: \"0ac16366-4faa-45e8-89a2-8b5c82f4bf29\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb" Oct 31 12:51:47.171154 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:47.171124 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ac16366-4faa-45e8-89a2-8b5c82f4bf29-metrics-certs\") pod \"maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb\" (UID: \"0ac16366-4faa-45e8-89a2-8b5c82f4bf29\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb" Oct 31 12:51:47.204417 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:47.204393 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqdt5\" (UniqueName: \"kubernetes.io/projected/0ac16366-4faa-45e8-89a2-8b5c82f4bf29-kube-api-access-tqdt5\") pod \"maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb\" (UID: \"0ac16366-4faa-45e8-89a2-8b5c82f4bf29\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb" Oct 31 12:51:47.659796 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:47.658233 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b03bbf98-77cd8cfdbc-gl7bg"] Oct 31 12:51:47.662328 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:47.662305 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b03bbf98-77cd8cfdbc-gl7bg" Oct 31 12:51:47.672409 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:47.672374 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac16366-4faa-45e8-89a2-8b5c82f4bf29-maas-api-tls\") pod \"maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb\" (UID: \"0ac16366-4faa-45e8-89a2-8b5c82f4bf29\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb" Oct 31 12:51:47.677117 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:47.677086 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac16366-4faa-45e8-89a2-8b5c82f4bf29-maas-api-tls\") pod \"maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb\" (UID: \"0ac16366-4faa-45e8-89a2-8b5c82f4bf29\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb" Oct 31 12:51:47.677292 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:47.677270 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b03bbf98-77cd8cfdbc-gl7bg"] Oct 31 12:51:47.765170 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:47.765138 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b03bbf98\"" Oct 31 12:51:47.765942 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:47.765920 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b03bbf98-dockercfg-9zvz6\"" Oct 31 12:51:47.773972 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:47.773951 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb36a5c9-b8b9-4b0d-a247-6ba4f35a8b34-plugins-config-volume\") pod \"payload-processing-e2e-ait-b03bbf98-77cd8cfdbc-gl7bg\" (UID: \"bb36a5c9-b8b9-4b0d-a247-6ba4f35a8b34\") " pod="openshift-ingress/payload-processing-e2e-ait-b03bbf98-77cd8cfdbc-gl7bg" Oct 31 12:51:47.774066 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:47.774019 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzpfr\" (UniqueName: \"kubernetes.io/projected/bb36a5c9-b8b9-4b0d-a247-6ba4f35a8b34-kube-api-access-qzpfr\") pod \"payload-processing-e2e-ait-b03bbf98-77cd8cfdbc-gl7bg\" (UID: \"bb36a5c9-b8b9-4b0d-a247-6ba4f35a8b34\") " pod="openshift-ingress/payload-processing-e2e-ait-b03bbf98-77cd8cfdbc-gl7bg" Oct 31 12:51:47.875685 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:47.875622 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb36a5c9-b8b9-4b0d-a247-6ba4f35a8b34-plugins-config-volume\") pod \"payload-processing-e2e-ait-b03bbf98-77cd8cfdbc-gl7bg\" (UID: \"bb36a5c9-b8b9-4b0d-a247-6ba4f35a8b34\") " pod="openshift-ingress/payload-processing-e2e-ait-b03bbf98-77cd8cfdbc-gl7bg" Oct 31 12:51:47.875871 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:47.875734 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzpfr\" (UniqueName: \"kubernetes.io/projected/bb36a5c9-b8b9-4b0d-a247-6ba4f35a8b34-kube-api-access-qzpfr\") pod \"payload-processing-e2e-ait-b03bbf98-77cd8cfdbc-gl7bg\" (UID: \"bb36a5c9-b8b9-4b0d-a247-6ba4f35a8b34\") " pod="openshift-ingress/payload-processing-e2e-ait-b03bbf98-77cd8cfdbc-gl7bg" Oct 31 12:51:47.876314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:47.876293 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb36a5c9-b8b9-4b0d-a247-6ba4f35a8b34-plugins-config-volume\") pod \"payload-processing-e2e-ait-b03bbf98-77cd8cfdbc-gl7bg\" (UID: \"bb36a5c9-b8b9-4b0d-a247-6ba4f35a8b34\") " pod="openshift-ingress/payload-processing-e2e-ait-b03bbf98-77cd8cfdbc-gl7bg" Oct 31 12:51:47.898571 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:47.898541 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzpfr\" (UniqueName: \"kubernetes.io/projected/bb36a5c9-b8b9-4b0d-a247-6ba4f35a8b34-kube-api-access-qzpfr\") pod \"payload-processing-e2e-ait-b03bbf98-77cd8cfdbc-gl7bg\" (UID: \"bb36a5c9-b8b9-4b0d-a247-6ba4f35a8b34\") " pod="openshift-ingress/payload-processing-e2e-ait-b03bbf98-77cd8cfdbc-gl7bg" Oct 31 12:51:47.929258 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:47.929173 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb" Oct 31 12:51:47.990625 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:47.990590 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b03bbf98-77cd8cfdbc-gl7bg" Oct 31 12:51:48.139486 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:48.139454 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d48f454a-67bddb4b87-75psm"] Oct 31 12:51:48.144002 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:48.143985 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d48f454a-67bddb4b87-75psm" Oct 31 12:51:48.154990 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:48.154970 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d48f454a\"" Oct 31 12:51:48.157094 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:48.155752 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d48f454a-dockercfg-fnvqk\"" Oct 31 12:51:48.166827 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:48.166782 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d48f454a-67bddb4b87-75psm"] Oct 31 12:51:48.219104 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:48.218903 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb"] Oct 31 12:51:48.253704 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:51:48.223039 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ac16366_4faa_45e8_89a2_8b5c82f4bf29.slice/crio-7ee756083f9af455decba360143674c0ee196bbc9b1949491330d5a71ef5b329 WatchSource:0}: Error finding container 7ee756083f9af455decba360143674c0ee196bbc9b1949491330d5a71ef5b329: Status 404 returned error can't find the container with id 7ee756083f9af455decba360143674c0ee196bbc9b1949491330d5a71ef5b329 Oct 31 12:51:48.280046 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:48.280008 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/657aca5d-eafd-4554-80ae-a2b6d1b01987-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d48f454a-67bddb4b87-75psm\" (UID: \"657aca5d-eafd-4554-80ae-a2b6d1b01987\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d48f454a-67bddb4b87-75psm" Oct 31 12:51:48.280207 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:48.280070 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjg9n\" (UniqueName: \"kubernetes.io/projected/657aca5d-eafd-4554-80ae-a2b6d1b01987-kube-api-access-gjg9n\") pod \"payload-pre-processing-e2e-ait-d48f454a-67bddb4b87-75psm\" (UID: \"657aca5d-eafd-4554-80ae-a2b6d1b01987\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d48f454a-67bddb4b87-75psm" Oct 31 12:51:48.293328 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:48.293305 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b03bbf98-77cd8cfdbc-gl7bg"] Oct 31 12:51:48.295204 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:51:48.295181 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb36a5c9_b8b9_4b0d_a247_6ba4f35a8b34.slice/crio-024b60898384fb10e6b85a5508d9a282a5be5a92e7030d703578d33e1f429a62 WatchSource:0}: Error finding container 024b60898384fb10e6b85a5508d9a282a5be5a92e7030d703578d33e1f429a62: Status 404 returned error can't find the container with id 024b60898384fb10e6b85a5508d9a282a5be5a92e7030d703578d33e1f429a62 Oct 31 12:51:48.381085 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:48.381056 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/657aca5d-eafd-4554-80ae-a2b6d1b01987-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d48f454a-67bddb4b87-75psm\" (UID: \"657aca5d-eafd-4554-80ae-a2b6d1b01987\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d48f454a-67bddb4b87-75psm" Oct 31 12:51:48.381206 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:48.381112 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjg9n\" (UniqueName: \"kubernetes.io/projected/657aca5d-eafd-4554-80ae-a2b6d1b01987-kube-api-access-gjg9n\") pod \"payload-pre-processing-e2e-ait-d48f454a-67bddb4b87-75psm\" (UID: \"657aca5d-eafd-4554-80ae-a2b6d1b01987\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d48f454a-67bddb4b87-75psm" Oct 31 12:51:48.382015 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:48.381995 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/657aca5d-eafd-4554-80ae-a2b6d1b01987-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d48f454a-67bddb4b87-75psm\" (UID: \"657aca5d-eafd-4554-80ae-a2b6d1b01987\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d48f454a-67bddb4b87-75psm" Oct 31 12:51:48.401151 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:48.401127 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjg9n\" (UniqueName: \"kubernetes.io/projected/657aca5d-eafd-4554-80ae-a2b6d1b01987-kube-api-access-gjg9n\") pod \"payload-pre-processing-e2e-ait-d48f454a-67bddb4b87-75psm\" (UID: \"657aca5d-eafd-4554-80ae-a2b6d1b01987\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d48f454a-67bddb4b87-75psm" Oct 31 12:51:48.463941 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:48.463906 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d48f454a-67bddb4b87-75psm" Oct 31 12:51:48.469207 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:48.469132 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b03bbf98-6844c68f55-k967j"] Oct 31 12:51:48.472880 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:48.472861 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b03bbf98-6844c68f55-k967j" Oct 31 12:51:48.482277 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:48.482252 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b03bbf98-serving-cert\"" Oct 31 12:51:48.484801 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:48.484737 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b03bbf98-6844c68f55-k967j"] Oct 31 12:51:48.583454 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:48.583412 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf9bb\" (UniqueName: \"kubernetes.io/projected/c81fe16d-d0a1-4976-8801-b84f46cccaed-kube-api-access-rf9bb\") pod \"maas-api-e2e-ait-b03bbf98-6844c68f55-k967j\" (UID: \"c81fe16d-d0a1-4976-8801-b84f46cccaed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b03bbf98-6844c68f55-k967j" Oct 31 12:51:48.583621 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:48.583468 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c81fe16d-d0a1-4976-8801-b84f46cccaed-openshift-service-ca\") pod \"maas-api-e2e-ait-b03bbf98-6844c68f55-k967j\" (UID: \"c81fe16d-d0a1-4976-8801-b84f46cccaed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b03bbf98-6844c68f55-k967j" Oct 31 12:51:48.583621 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:48.583548 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c81fe16d-d0a1-4976-8801-b84f46cccaed-metrics-certs\") pod \"maas-api-e2e-ait-b03bbf98-6844c68f55-k967j\" (UID: \"c81fe16d-d0a1-4976-8801-b84f46cccaed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b03bbf98-6844c68f55-k967j" Oct 31 12:51:48.583621 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:48.583602 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c81fe16d-d0a1-4976-8801-b84f46cccaed-maas-api-tls\") pod \"maas-api-e2e-ait-b03bbf98-6844c68f55-k967j\" (UID: \"c81fe16d-d0a1-4976-8801-b84f46cccaed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b03bbf98-6844c68f55-k967j" Oct 31 12:51:48.685121 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:48.685080 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c81fe16d-d0a1-4976-8801-b84f46cccaed-openshift-service-ca\") pod \"maas-api-e2e-ait-b03bbf98-6844c68f55-k967j\" (UID: \"c81fe16d-d0a1-4976-8801-b84f46cccaed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b03bbf98-6844c68f55-k967j" Oct 31 12:51:48.685307 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:48.685131 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c81fe16d-d0a1-4976-8801-b84f46cccaed-metrics-certs\") pod \"maas-api-e2e-ait-b03bbf98-6844c68f55-k967j\" (UID: \"c81fe16d-d0a1-4976-8801-b84f46cccaed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b03bbf98-6844c68f55-k967j" Oct 31 12:51:48.685307 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:48.685195 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c81fe16d-d0a1-4976-8801-b84f46cccaed-maas-api-tls\") pod \"maas-api-e2e-ait-b03bbf98-6844c68f55-k967j\" (UID: \"c81fe16d-d0a1-4976-8801-b84f46cccaed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b03bbf98-6844c68f55-k967j" Oct 31 12:51:48.685415 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:48.685344 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf9bb\" (UniqueName: \"kubernetes.io/projected/c81fe16d-d0a1-4976-8801-b84f46cccaed-kube-api-access-rf9bb\") pod \"maas-api-e2e-ait-b03bbf98-6844c68f55-k967j\" (UID: \"c81fe16d-d0a1-4976-8801-b84f46cccaed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b03bbf98-6844c68f55-k967j" Oct 31 12:51:48.685916 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:48.685894 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c81fe16d-d0a1-4976-8801-b84f46cccaed-openshift-service-ca\") pod \"maas-api-e2e-ait-b03bbf98-6844c68f55-k967j\" (UID: \"c81fe16d-d0a1-4976-8801-b84f46cccaed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b03bbf98-6844c68f55-k967j" Oct 31 12:51:48.687993 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:48.687970 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c81fe16d-d0a1-4976-8801-b84f46cccaed-maas-api-tls\") pod \"maas-api-e2e-ait-b03bbf98-6844c68f55-k967j\" (UID: \"c81fe16d-d0a1-4976-8801-b84f46cccaed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b03bbf98-6844c68f55-k967j" Oct 31 12:51:48.688144 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:48.688124 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c81fe16d-d0a1-4976-8801-b84f46cccaed-metrics-certs\") pod \"maas-api-e2e-ait-b03bbf98-6844c68f55-k967j\" (UID: \"c81fe16d-d0a1-4976-8801-b84f46cccaed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b03bbf98-6844c68f55-k967j" Oct 31 12:51:48.690513 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:51:48.690485 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:51:48.697399 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:48.697375 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf9bb\" (UniqueName: \"kubernetes.io/projected/c81fe16d-d0a1-4976-8801-b84f46cccaed-kube-api-access-rf9bb\") pod \"maas-api-e2e-ait-b03bbf98-6844c68f55-k967j\" (UID: \"c81fe16d-d0a1-4976-8801-b84f46cccaed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b03bbf98-6844c68f55-k967j" Oct 31 12:51:48.761933 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:48.761907 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d48f454a-67bddb4b87-75psm"] Oct 31 12:51:48.766713 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:51:48.766671 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod657aca5d_eafd_4554_80ae_a2b6d1b01987.slice/crio-b2c4f3b10c1b298b4e60ac776e3adafd83a756f79a7d58358080a1bda0c14537 WatchSource:0}: Error finding container b2c4f3b10c1b298b4e60ac776e3adafd83a756f79a7d58358080a1bda0c14537: Status 404 returned error can't find the container with id b2c4f3b10c1b298b4e60ac776e3adafd83a756f79a7d58358080a1bda0c14537 Oct 31 12:51:48.795468 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:48.795445 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b03bbf98-6844c68f55-k967j" Oct 31 12:51:49.035253 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:49.035168 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d48f454a-67bddb4b87-75psm" event={"ID":"657aca5d-eafd-4554-80ae-a2b6d1b01987","Type":"ContainerStarted","Data":"d5736d32537b2d9a7298c0ddf7d50e1c1b21aac49e73ce28da7840fd040508aa"} Oct 31 12:51:49.035253 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:49.035229 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d48f454a-67bddb4b87-75psm" event={"ID":"657aca5d-eafd-4554-80ae-a2b6d1b01987","Type":"ContainerStarted","Data":"b2c4f3b10c1b298b4e60ac776e3adafd83a756f79a7d58358080a1bda0c14537"} Oct 31 12:51:49.035490 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:49.035317 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d48f454a-67bddb4b87-75psm" Oct 31 12:51:49.036889 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:49.036856 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b03bbf98-77cd8cfdbc-gl7bg" event={"ID":"bb36a5c9-b8b9-4b0d-a247-6ba4f35a8b34","Type":"ContainerStarted","Data":"a49a4f13779cd621162937d4bb4503bbb511523357c7639c1afeda9bb3e253c0"} Oct 31 12:51:49.037043 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:49.036923 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b03bbf98-77cd8cfdbc-gl7bg" event={"ID":"bb36a5c9-b8b9-4b0d-a247-6ba4f35a8b34","Type":"ContainerStarted","Data":"024b60898384fb10e6b85a5508d9a282a5be5a92e7030d703578d33e1f429a62"} Oct 31 12:51:49.037043 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:49.036969 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b03bbf98-77cd8cfdbc-gl7bg" Oct 31 12:51:49.038606 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:49.038585 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb" event={"ID":"0ac16366-4faa-45e8-89a2-8b5c82f4bf29","Type":"ContainerStarted","Data":"b1d2c6b8c3f2866ccf9723b865100aa045fad33566b8e33b3d3a882e2cc7b8ad"} Oct 31 12:51:49.038755 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:49.038625 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb" event={"ID":"0ac16366-4faa-45e8-89a2-8b5c82f4bf29","Type":"ContainerStarted","Data":"7ee756083f9af455decba360143674c0ee196bbc9b1949491330d5a71ef5b329"} Oct 31 12:51:49.038755 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:49.038697 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb" Oct 31 12:51:49.060450 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:49.060396 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d48f454a-67bddb4b87-75psm" podStartSLOduration=1.060377012 podStartE2EDuration="1.060377012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:51:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:51:49.056337838 +0000 UTC m=+1159.763924135" watchObservedRunningTime="2026-10-31 12:51:49.060377012 +0000 UTC m=+1159.767963307" Oct 31 12:51:49.076094 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:49.076049 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b03bbf98-77cd8cfdbc-gl7bg" podStartSLOduration=2.076035457 podStartE2EDuration="2.076035457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:51:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:51:49.074160756 +0000 UTC m=+1159.781747051" watchObservedRunningTime="2026-10-31 12:51:49.076035457 +0000 UTC m=+1159.783621752" Oct 31 12:51:49.081669 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:49.081624 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b03bbf98-6844c68f55-k967j"] Oct 31 12:51:49.084905 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:51:49.084882 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc81fe16d_d0a1_4976_8801_b84f46cccaed.slice/crio-2e9197703be483a1a3abfdcdb2084ce811ac8502893d35847234678ccfff37a4 WatchSource:0}: Error finding container 2e9197703be483a1a3abfdcdb2084ce811ac8502893d35847234678ccfff37a4: Status 404 returned error can't find the container with id 2e9197703be483a1a3abfdcdb2084ce811ac8502893d35847234678ccfff37a4 Oct 31 12:51:49.098546 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:49.098500 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb" podStartSLOduration=2.587920691 podStartE2EDuration="3.098485217s" totalImagesPullingTime="510.564526ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:51:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:51:49.097773716 +0000 UTC m=+1159.805360051" watchObservedRunningTime="2026-10-31 12:51:49.098485217 +0000 UTC m=+1159.806071512" Oct 31 12:51:50.045095 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:50.045051 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b03bbf98-6844c68f55-k967j" event={"ID":"c81fe16d-d0a1-4976-8801-b84f46cccaed","Type":"ContainerStarted","Data":"e0a24ade385c970ad834fedee142d550c27af2fac9f93a6cf024aaf66c7f06b1"} Oct 31 12:51:50.045704 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:50.045129 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b03bbf98-6844c68f55-k967j" Oct 31 12:51:50.045704 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:50.045171 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b03bbf98-6844c68f55-k967j" event={"ID":"c81fe16d-d0a1-4976-8801-b84f46cccaed","Type":"ContainerStarted","Data":"2e9197703be483a1a3abfdcdb2084ce811ac8502893d35847234678ccfff37a4"} Oct 31 12:51:50.068841 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:50.068788 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b03bbf98-6844c68f55-k967j" podStartSLOduration=1.7004637759999999 podStartE2EDuration="2.068771188s" totalImagesPullingTime="368.307412ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:51:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:51:50.0677508 +0000 UTC m=+1160.775337096" watchObservedRunningTime="2026-10-31 12:51:50.068771188 +0000 UTC m=+1160.776357486" Oct 31 12:51:52.482104 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:52.482067 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m"] Oct 31 12:51:52.486217 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:52.486195 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m" Oct 31 12:51:52.503712 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:52.503685 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m"] Oct 31 12:51:52.543045 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:52.543015 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1abc54b4-serving-cert\"" Oct 31 12:51:52.654212 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:52.654125 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72445e01-d558-4fd1-80fd-05a86379726a-openshift-service-ca\") pod \"maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m\" (UID: \"72445e01-d558-4fd1-80fd-05a86379726a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m" Oct 31 12:51:52.654212 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:52.654186 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72445e01-d558-4fd1-80fd-05a86379726a-metrics-certs\") pod \"maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m\" (UID: \"72445e01-d558-4fd1-80fd-05a86379726a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m" Oct 31 12:51:52.654414 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:52.654368 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwwvk\" (UniqueName: \"kubernetes.io/projected/72445e01-d558-4fd1-80fd-05a86379726a-kube-api-access-mwwvk\") pod \"maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m\" (UID: \"72445e01-d558-4fd1-80fd-05a86379726a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m" Oct 31 12:51:52.654523 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:52.654494 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72445e01-d558-4fd1-80fd-05a86379726a-maas-api-tls\") pod \"maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m\" (UID: \"72445e01-d558-4fd1-80fd-05a86379726a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m" Oct 31 12:51:52.756296 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:52.756194 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72445e01-d558-4fd1-80fd-05a86379726a-metrics-certs\") pod \"maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m\" (UID: \"72445e01-d558-4fd1-80fd-05a86379726a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m" Oct 31 12:51:52.756486 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:52.756307 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwwvk\" (UniqueName: \"kubernetes.io/projected/72445e01-d558-4fd1-80fd-05a86379726a-kube-api-access-mwwvk\") pod \"maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m\" (UID: \"72445e01-d558-4fd1-80fd-05a86379726a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m" Oct 31 12:51:52.756486 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:52.756351 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72445e01-d558-4fd1-80fd-05a86379726a-maas-api-tls\") pod \"maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m\" (UID: \"72445e01-d558-4fd1-80fd-05a86379726a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m" Oct 31 12:51:52.756486 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:52.756381 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72445e01-d558-4fd1-80fd-05a86379726a-openshift-service-ca\") pod \"maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m\" (UID: \"72445e01-d558-4fd1-80fd-05a86379726a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m" Oct 31 12:51:52.756486 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:51:52.756469 5693 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-1abc54b4-serving-cert: secret "maas-api-e2e-ait-1abc54b4-serving-cert" not found Oct 31 12:51:52.756710 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:51:52.756534 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72445e01-d558-4fd1-80fd-05a86379726a-maas-api-tls podName:72445e01-d558-4fd1-80fd-05a86379726a nodeName:}" failed. No retries permitted until 2026-10-31 12:51:53.256515492 +0000 UTC m=+1163.964101787 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/72445e01-d558-4fd1-80fd-05a86379726a-maas-api-tls") pod "maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m" (UID: "72445e01-d558-4fd1-80fd-05a86379726a") : secret "maas-api-e2e-ait-1abc54b4-serving-cert" not found Oct 31 12:51:52.757202 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:52.757109 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72445e01-d558-4fd1-80fd-05a86379726a-openshift-service-ca\") pod \"maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m\" (UID: \"72445e01-d558-4fd1-80fd-05a86379726a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m" Oct 31 12:51:52.759116 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:52.759090 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72445e01-d558-4fd1-80fd-05a86379726a-metrics-certs\") pod \"maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m\" (UID: \"72445e01-d558-4fd1-80fd-05a86379726a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m" Oct 31 12:51:52.770300 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:52.770278 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwwvk\" (UniqueName: \"kubernetes.io/projected/72445e01-d558-4fd1-80fd-05a86379726a-kube-api-access-mwwvk\") pod \"maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m\" (UID: \"72445e01-d558-4fd1-80fd-05a86379726a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m" Oct 31 12:51:53.261846 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:53.261810 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72445e01-d558-4fd1-80fd-05a86379726a-maas-api-tls\") pod \"maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m\" (UID: \"72445e01-d558-4fd1-80fd-05a86379726a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m" Oct 31 12:51:53.264342 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:53.264318 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72445e01-d558-4fd1-80fd-05a86379726a-maas-api-tls\") pod \"maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m\" (UID: \"72445e01-d558-4fd1-80fd-05a86379726a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m" Oct 31 12:51:53.401984 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:53.401943 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m" Oct 31 12:51:53.700700 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:53.683955 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m"] Oct 31 12:51:53.704689 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:51:53.704610 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72445e01_d558_4fd1_80fd_05a86379726a.slice/crio-78e7785325f6d3338594623ef10711630e76a3dab16968fb11f67b73e1946a14 WatchSource:0}: Error finding container 78e7785325f6d3338594623ef10711630e76a3dab16968fb11f67b73e1946a14: Status 404 returned error can't find the container with id 78e7785325f6d3338594623ef10711630e76a3dab16968fb11f67b73e1946a14 Oct 31 12:51:54.069758 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:54.066629 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m" event={"ID":"72445e01-d558-4fd1-80fd-05a86379726a","Type":"ContainerStarted","Data":"78e7785325f6d3338594623ef10711630e76a3dab16968fb11f67b73e1946a14"} Oct 31 12:51:55.051569 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:55.051537 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb" Oct 31 12:51:55.073123 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:55.073083 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m" event={"ID":"72445e01-d558-4fd1-80fd-05a86379726a","Type":"ContainerStarted","Data":"ac642ad678b3b95dbb240f072c674327ac13c6300c77fa0ce6202e0936745f02"} Oct 31 12:51:55.073288 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:55.073201 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m" Oct 31 12:51:55.132383 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:55.132334 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m" podStartSLOduration=2.722342596 podStartE2EDuration="3.132320203s" totalImagesPullingTime="409.977607ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:51:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:51:55.129476676 +0000 UTC m=+1165.837062968" watchObservedRunningTime="2026-10-31 12:51:55.132320203 +0000 UTC m=+1165.839906498" Oct 31 12:51:56.057476 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:51:56.057449 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b03bbf98-6844c68f55-k967j" Oct 31 12:52:00.047819 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:00.047782 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d48f454a-67bddb4b87-75psm" Oct 31 12:52:00.048350 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:00.047959 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b03bbf98-77cd8cfdbc-gl7bg" Oct 31 12:52:01.049816 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:01.049777 5693 patch_prober.go:29] interesting pod/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:52:01.050314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:01.049849 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" podUID="bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:52:01.097750 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:01.093984 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4"] Oct 31 12:52:01.118136 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:01.115351 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4" Oct 31 12:52:01.126291 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:01.126043 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4"] Oct 31 12:52:01.128354 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:01.128327 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m" Oct 31 12:52:01.132021 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:01.131147 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-785c04b9-serving-cert\"" Oct 31 12:52:01.224833 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:01.224805 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrcgx\" (UniqueName: \"kubernetes.io/projected/bee8161c-23f9-4563-a8c7-506fcf777846-kube-api-access-rrcgx\") pod \"maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4\" (UID: \"bee8161c-23f9-4563-a8c7-506fcf777846\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4" Oct 31 12:52:01.225027 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:01.225013 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bee8161c-23f9-4563-a8c7-506fcf777846-maas-api-tls\") pod \"maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4\" (UID: \"bee8161c-23f9-4563-a8c7-506fcf777846\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4" Oct 31 12:52:01.225134 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:01.225126 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bee8161c-23f9-4563-a8c7-506fcf777846-metrics-certs\") pod \"maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4\" (UID: \"bee8161c-23f9-4563-a8c7-506fcf777846\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4" Oct 31 12:52:01.225311 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:01.225302 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bee8161c-23f9-4563-a8c7-506fcf777846-openshift-service-ca\") pod \"maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4\" (UID: \"bee8161c-23f9-4563-a8c7-506fcf777846\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4" Oct 31 12:52:01.326754 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:01.326670 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bee8161c-23f9-4563-a8c7-506fcf777846-openshift-service-ca\") pod \"maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4\" (UID: \"bee8161c-23f9-4563-a8c7-506fcf777846\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4" Oct 31 12:52:01.326754 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:01.326716 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrcgx\" (UniqueName: \"kubernetes.io/projected/bee8161c-23f9-4563-a8c7-506fcf777846-kube-api-access-rrcgx\") pod \"maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4\" (UID: \"bee8161c-23f9-4563-a8c7-506fcf777846\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4" Oct 31 12:52:01.326754 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:01.326737 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bee8161c-23f9-4563-a8c7-506fcf777846-maas-api-tls\") pod \"maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4\" (UID: \"bee8161c-23f9-4563-a8c7-506fcf777846\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4" Oct 31 12:52:01.327069 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:01.326787 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bee8161c-23f9-4563-a8c7-506fcf777846-metrics-certs\") pod \"maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4\" (UID: \"bee8161c-23f9-4563-a8c7-506fcf777846\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4" Oct 31 12:52:01.327435 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:01.327407 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bee8161c-23f9-4563-a8c7-506fcf777846-openshift-service-ca\") pod \"maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4\" (UID: \"bee8161c-23f9-4563-a8c7-506fcf777846\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4" Oct 31 12:52:01.329526 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:01.329504 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bee8161c-23f9-4563-a8c7-506fcf777846-maas-api-tls\") pod \"maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4\" (UID: \"bee8161c-23f9-4563-a8c7-506fcf777846\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4" Oct 31 12:52:01.329629 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:01.329592 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bee8161c-23f9-4563-a8c7-506fcf777846-metrics-certs\") pod \"maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4\" (UID: \"bee8161c-23f9-4563-a8c7-506fcf777846\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4" Oct 31 12:52:01.339713 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:01.339633 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrcgx\" (UniqueName: \"kubernetes.io/projected/bee8161c-23f9-4563-a8c7-506fcf777846-kube-api-access-rrcgx\") pod \"maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4\" (UID: \"bee8161c-23f9-4563-a8c7-506fcf777846\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4" Oct 31 12:52:01.439611 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:01.439583 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4" Oct 31 12:52:01.756316 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:01.756262 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4"] Oct 31 12:52:01.759839 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:52:01.759785 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee8161c_23f9_4563_a8c7_506fcf777846.slice/crio-b7fc87183e507c01b4f8ecb8cbe3c8b8100524ea93113ae1404f77df50cf17af WatchSource:0}: Error finding container b7fc87183e507c01b4f8ecb8cbe3c8b8100524ea93113ae1404f77df50cf17af: Status 404 returned error can't find the container with id b7fc87183e507c01b4f8ecb8cbe3c8b8100524ea93113ae1404f77df50cf17af Oct 31 12:52:02.153851 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:02.153817 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4" event={"ID":"bee8161c-23f9-4563-a8c7-506fcf777846","Type":"ContainerStarted","Data":"b7fc87183e507c01b4f8ecb8cbe3c8b8100524ea93113ae1404f77df50cf17af"} Oct 31 12:52:03.162238 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:03.162202 5693 generic.go:356] "Generic (PLEG): container finished" podID="bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c" containerID="84ecd081bcf7eacf2460045d2e9335c3865d981dce932ba67c9668bbb9472e82" exitCode=0 Oct 31 12:52:03.162766 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:03.162276 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" event={"ID":"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c","Type":"ContainerDied","Data":"84ecd081bcf7eacf2460045d2e9335c3865d981dce932ba67c9668bbb9472e82"} Oct 31 12:52:03.162766 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:03.162742 5693 scope.go:122] "RemoveContainer" containerID="84ecd081bcf7eacf2460045d2e9335c3865d981dce932ba67c9668bbb9472e82" Oct 31 12:52:03.164201 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:03.164162 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:52:03.164314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:03.164245 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4" event={"ID":"bee8161c-23f9-4563-a8c7-506fcf777846","Type":"ContainerStarted","Data":"773c318899981743e1e028098b32f0545c9dea00ec0b010b07a748a8f08cc1c6"} Oct 31 12:52:03.164314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:03.164253 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:52:03.164420 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:03.164349 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:52:03.164786 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:03.164768 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4" Oct 31 12:52:03.241385 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:03.241337 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4" podStartSLOduration=1.841271023 podStartE2EDuration="2.241316324s" totalImagesPullingTime="400.045301ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:52:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:52:03.238727735 +0000 UTC m=+1173.946314032" watchObservedRunningTime="2026-10-31 12:52:03.241316324 +0000 UTC m=+1173.948902619" Oct 31 12:52:04.172122 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:04.172087 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" event={"ID":"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c","Type":"ContainerStarted","Data":"92a28f21572fcf617065aac57f1530c1157cb289a064ec37f1d4591236124691"} Oct 31 12:52:05.147501 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:05.147467 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" Oct 31 12:52:05.147769 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:05.147739 5693 patch_prober.go:29] interesting pod/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 31 12:52:05.147898 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:05.147823 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" podUID="bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 31 12:52:05.190670 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:05.190189 5693 generic.go:356] "Generic (PLEG): container finished" podID="48005f27-24cb-4a42-8b06-51a2254c3473" containerID="14c0ec5e0c98aaf83cfbadb497ce2a77443d5fdf97d11398cb6ae7de709e3699" exitCode=0 Oct 31 12:52:05.191189 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:05.191058 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" event={"ID":"48005f27-24cb-4a42-8b06-51a2254c3473","Type":"ContainerDied","Data":"14c0ec5e0c98aaf83cfbadb497ce2a77443d5fdf97d11398cb6ae7de709e3699"} Oct 31 12:52:05.210667 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:05.191412 5693 scope.go:122] "RemoveContainer" containerID="14c0ec5e0c98aaf83cfbadb497ce2a77443d5fdf97d11398cb6ae7de709e3699" Oct 31 12:52:05.210667 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:05.199034 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:52:05.210667 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:05.199127 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:52:05.210667 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:05.199175 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:52:06.147753 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:06.147710 5693 patch_prober.go:29] interesting pod/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 31 12:52:06.147986 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:06.147771 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" podUID="bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 31 12:52:06.214131 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:06.214098 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" event={"ID":"48005f27-24cb-4a42-8b06-51a2254c3473","Type":"ContainerStarted","Data":"2314a90358a3e21a8d2295847000c3db99c131b54dccd7490d292a28a0f3ece2"} Oct 31 12:52:06.400136 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:06.400029 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" Oct 31 12:52:06.401253 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:06.400413 5693 patch_prober.go:29] interesting pod/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 31 12:52:06.401476 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:06.401455 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" podUID="48005f27-24cb-4a42-8b06-51a2254c3473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 31 12:52:07.148191 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:07.148151 5693 patch_prober.go:29] interesting pod/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 31 12:52:07.148366 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:07.148230 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" podUID="bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 31 12:52:07.433771 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:07.432159 5693 patch_prober.go:29] interesting pod/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 31 12:52:07.433771 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:07.432244 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" podUID="48005f27-24cb-4a42-8b06-51a2254c3473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 31 12:52:08.148293 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:08.148241 5693 patch_prober.go:29] interesting pod/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 31 12:52:08.148479 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:08.148331 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" podUID="bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 31 12:52:08.400727 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:08.400116 5693 patch_prober.go:29] interesting pod/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 31 12:52:08.400727 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:08.400202 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" podUID="48005f27-24cb-4a42-8b06-51a2254c3473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 31 12:52:09.147604 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:09.147568 5693 patch_prober.go:29] interesting pod/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 31 12:52:09.148057 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:09.147620 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" podUID="bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 31 12:52:09.406784 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:09.406677 5693 patch_prober.go:29] interesting pod/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 31 12:52:09.406784 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:09.406773 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" podUID="48005f27-24cb-4a42-8b06-51a2254c3473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 31 12:52:10.151348 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:10.151317 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" Oct 31 12:52:10.151754 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:10.151602 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" Oct 31 12:52:10.152331 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:10.152315 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" Oct 31 12:52:10.204826 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:10.204797 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4" Oct 31 12:52:10.402173 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:10.402083 5693 patch_prober.go:29] interesting pod/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 31 12:52:10.402440 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:10.402410 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" podUID="48005f27-24cb-4a42-8b06-51a2254c3473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 31 12:52:11.026619 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:11.026591 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb"] Oct 31 12:52:11.027127 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:11.026829 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb" podUID="0ac16366-4faa-45e8-89a2-8b5c82f4bf29" containerName="maas-api" containerID="cri-o://b1d2c6b8c3f2866ccf9723b865100aa045fad33566b8e33b3d3a882e2cc7b8ad" gracePeriod=30 Oct 31 12:52:11.258902 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:11.258867 5693 generic.go:356] "Generic (PLEG): container finished" podID="0ac16366-4faa-45e8-89a2-8b5c82f4bf29" containerID="b1d2c6b8c3f2866ccf9723b865100aa045fad33566b8e33b3d3a882e2cc7b8ad" exitCode=0 Oct 31 12:52:11.261478 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:11.259722 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb" event={"ID":"0ac16366-4faa-45e8-89a2-8b5c82f4bf29","Type":"ContainerDied","Data":"b1d2c6b8c3f2866ccf9723b865100aa045fad33566b8e33b3d3a882e2cc7b8ad"} Oct 31 12:52:11.374864 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:11.374836 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb" Oct 31 12:52:11.412709 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:11.411802 5693 patch_prober.go:29] interesting pod/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 31 12:52:11.412709 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:11.411874 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" podUID="48005f27-24cb-4a42-8b06-51a2254c3473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 31 12:52:11.512999 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:11.512962 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ac16366-4faa-45e8-89a2-8b5c82f4bf29-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ac16366-4faa-45e8-89a2-8b5c82f4bf29-metrics-certs\") pod \"0ac16366-4faa-45e8-89a2-8b5c82f4bf29\" (UID: \"0ac16366-4faa-45e8-89a2-8b5c82f4bf29\") " Oct 31 12:52:11.512999 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:11.513000 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ac16366-4faa-45e8-89a2-8b5c82f4bf29-kube-api-access-tqdt5\" (UniqueName: \"kubernetes.io/projected/0ac16366-4faa-45e8-89a2-8b5c82f4bf29-kube-api-access-tqdt5\") pod \"0ac16366-4faa-45e8-89a2-8b5c82f4bf29\" (UID: \"0ac16366-4faa-45e8-89a2-8b5c82f4bf29\") " Oct 31 12:52:11.513330 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:11.513034 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ac16366-4faa-45e8-89a2-8b5c82f4bf29-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac16366-4faa-45e8-89a2-8b5c82f4bf29-openshift-service-ca\") pod \"0ac16366-4faa-45e8-89a2-8b5c82f4bf29\" (UID: \"0ac16366-4faa-45e8-89a2-8b5c82f4bf29\") " Oct 31 12:52:11.513330 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:11.513058 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ac16366-4faa-45e8-89a2-8b5c82f4bf29-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac16366-4faa-45e8-89a2-8b5c82f4bf29-maas-api-tls\") pod \"0ac16366-4faa-45e8-89a2-8b5c82f4bf29\" (UID: \"0ac16366-4faa-45e8-89a2-8b5c82f4bf29\") " Oct 31 12:52:11.513450 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:11.513427 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ac16366-4faa-45e8-89a2-8b5c82f4bf29-openshift-service-ca" pod "0ac16366-4faa-45e8-89a2-8b5c82f4bf29" (UID: "0ac16366-4faa-45e8-89a2-8b5c82f4bf29"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:52:11.515358 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:11.515328 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ac16366-4faa-45e8-89a2-8b5c82f4bf29-metrics-certs" pod "0ac16366-4faa-45e8-89a2-8b5c82f4bf29" (UID: "0ac16366-4faa-45e8-89a2-8b5c82f4bf29"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:52:11.515539 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:11.515514 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ac16366-4faa-45e8-89a2-8b5c82f4bf29-kube-api-access-tqdt5" pod "0ac16366-4faa-45e8-89a2-8b5c82f4bf29" (UID: "0ac16366-4faa-45e8-89a2-8b5c82f4bf29"). InnerVolumeSpecName "kube-api-access-tqdt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:52:11.515672 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:11.515581 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ac16366-4faa-45e8-89a2-8b5c82f4bf29-maas-api-tls" pod "0ac16366-4faa-45e8-89a2-8b5c82f4bf29" (UID: "0ac16366-4faa-45e8-89a2-8b5c82f4bf29"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:52:11.615139 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:11.615040 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ac16366-4faa-45e8-89a2-8b5c82f4bf29-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:11.615139 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:11.615074 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqdt5\" (UniqueName: \"kubernetes.io/projected/0ac16366-4faa-45e8-89a2-8b5c82f4bf29-kube-api-access-tqdt5\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:11.615139 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:11.615085 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac16366-4faa-45e8-89a2-8b5c82f4bf29-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:11.615139 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:11.615094 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac16366-4faa-45e8-89a2-8b5c82f4bf29-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:12.101333 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:12.101298 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck"] Oct 31 12:52:12.265218 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:12.265188 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb" Oct 31 12:52:12.265655 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:12.265221 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb" event={"ID":"0ac16366-4faa-45e8-89a2-8b5c82f4bf29","Type":"ContainerDied","Data":"7ee756083f9af455decba360143674c0ee196bbc9b1949491330d5a71ef5b329"} Oct 31 12:52:12.265655 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:12.265284 5693 scope.go:122] "RemoveContainer" containerID="b1d2c6b8c3f2866ccf9723b865100aa045fad33566b8e33b3d3a882e2cc7b8ad" Oct 31 12:52:12.265655 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:12.265541 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" podUID="bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c" containerName="istio-proxy" containerID="cri-o://92a28f21572fcf617065aac57f1530c1157cb289a064ec37f1d4591236124691" gracePeriod=30 Oct 31 12:52:12.308475 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:12.308439 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb"] Oct 31 12:52:12.310583 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:12.310556 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cf7e6604-76c974db47-fdkgb"] Oct 31 12:52:12.404611 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:12.404579 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" Oct 31 12:52:12.404984 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:12.404961 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" Oct 31 12:52:12.405833 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:12.405810 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" Oct 31 12:52:13.408816 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:13.408774 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ac16366-4faa-45e8-89a2-8b5c82f4bf29" path="/var/lib/kubelet/pods/0ac16366-4faa-45e8-89a2-8b5c82f4bf29/volumes" Oct 31 12:52:16.265589 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:16.265555 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b03bbf98-77cd8cfdbc-gl7bg"] Oct 31 12:52:17.532288 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:17.532228 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" Oct 31 12:52:17.581888 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:17.581853 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-istio-podinfo\") pod \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\" (UID: \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\") " Oct 31 12:52:17.582066 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:17.581966 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-istiod-ca-cert\") pod \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\" (UID: \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\") " Oct 31 12:52:17.582147 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:17.582076 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-istio-data\") pod \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\" (UID: \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\") " Oct 31 12:52:17.582147 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:17.582106 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-istio-token\" (UniqueName: \"kubernetes.io/projected/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-istio-token\") pod \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\" (UID: \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\") " Oct 31 12:52:17.582147 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:17.582140 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-workload-socket\") pod \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\" (UID: \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\") " Oct 31 12:52:17.582297 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:17.582189 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-istio-envoy\") pod \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\" (UID: \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\") " Oct 31 12:52:17.582297 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:17.582225 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-workload-certs\") pod \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\" (UID: \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\") " Oct 31 12:52:17.582432 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:17.582397 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-istiod-ca-cert" pod "bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c" (UID: "bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:52:17.582498 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:17.582475 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-kube-api-access-shl86\" (UniqueName: \"kubernetes.io/projected/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-kube-api-access-shl86\") pod \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\" (UID: \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\") " Oct 31 12:52:17.582555 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:17.582539 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-credential-socket\") pod \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\" (UID: \"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c\") " Oct 31 12:52:17.582750 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:17.582676 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-workload-certs" pod "bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c" (UID: "bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:52:17.582989 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:17.582874 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-workload-socket" pod "bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c" (UID: "bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:52:17.583106 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:17.583066 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-credential-socket" pod "bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c" (UID: "bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:52:17.583106 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:17.583095 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:17.583279 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:17.583117 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:17.583279 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:17.583131 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:17.583434 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:17.583390 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-istio-data" pod "bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c" (UID: "bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:52:17.585313 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:17.585282 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-kube-api-access-shl86" pod "bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c" (UID: "bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c"). InnerVolumeSpecName "kube-api-access-shl86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:52:17.585313 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:17.585286 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-istio-podinfo" pod "bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c" (UID: "bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:52:17.585589 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:17.585563 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-istio-token" pod "bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c" (UID: "bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:52:17.585685 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:17.585612 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-istio-envoy" pod "bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c" (UID: "bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:52:17.684556 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:17.684512 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:17.684556 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:17.684552 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:17.684784 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:17.684566 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:17.684784 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:17.684581 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shl86\" (UniqueName: \"kubernetes.io/projected/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-kube-api-access-shl86\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:17.684784 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:17.684594 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:17.684784 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:17.684607 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:18.180356 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.180322 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b03bbf98-6844c68f55-k967j"] Oct 31 12:52:18.180606 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.180579 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b03bbf98-6844c68f55-k967j" podUID="c81fe16d-d0a1-4976-8801-b84f46cccaed" containerName="maas-api" containerID="cri-o://e0a24ade385c970ad834fedee142d550c27af2fac9f93a6cf024aaf66c7f06b1" gracePeriod=30 Oct 31 12:52:18.307057 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.307029 5693 generic.go:356] "Generic (PLEG): container finished" podID="c81fe16d-d0a1-4976-8801-b84f46cccaed" containerID="e0a24ade385c970ad834fedee142d550c27af2fac9f93a6cf024aaf66c7f06b1" exitCode=0 Oct 31 12:52:18.307197 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.307098 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b03bbf98-6844c68f55-k967j" event={"ID":"c81fe16d-d0a1-4976-8801-b84f46cccaed","Type":"ContainerDied","Data":"e0a24ade385c970ad834fedee142d550c27af2fac9f93a6cf024aaf66c7f06b1"} Oct 31 12:52:18.308795 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.308773 5693 generic.go:356] "Generic (PLEG): container finished" podID="bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c" containerID="92a28f21572fcf617065aac57f1530c1157cb289a064ec37f1d4591236124691" exitCode=0 Oct 31 12:52:18.308908 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.308830 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" Oct 31 12:52:18.308908 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.308847 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" event={"ID":"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c","Type":"ContainerDied","Data":"92a28f21572fcf617065aac57f1530c1157cb289a064ec37f1d4591236124691"} Oct 31 12:52:18.308908 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.308901 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck" event={"ID":"bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c","Type":"ContainerDied","Data":"4a2eaa4b25c059a7b9270f6c771e9c9671a6acea70cf483dd30df6ee9316c13d"} Oct 31 12:52:18.309064 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.308934 5693 scope.go:122] "RemoveContainer" containerID="92a28f21572fcf617065aac57f1530c1157cb289a064ec37f1d4591236124691" Oct 31 12:52:18.319548 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.319530 5693 scope.go:122] "RemoveContainer" containerID="84ecd081bcf7eacf2460045d2e9335c3865d981dce932ba67c9668bbb9472e82" Oct 31 12:52:18.328448 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.328423 5693 scope.go:122] "RemoveContainer" containerID="92a28f21572fcf617065aac57f1530c1157cb289a064ec37f1d4591236124691" Oct 31 12:52:18.328925 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:52:18.328891 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92a28f21572fcf617065aac57f1530c1157cb289a064ec37f1d4591236124691\": container with ID starting with 92a28f21572fcf617065aac57f1530c1157cb289a064ec37f1d4591236124691 not found: ID does not exist" containerID="92a28f21572fcf617065aac57f1530c1157cb289a064ec37f1d4591236124691" Oct 31 12:52:18.329018 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.328932 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92a28f21572fcf617065aac57f1530c1157cb289a064ec37f1d4591236124691"} err="failed to get container status \"92a28f21572fcf617065aac57f1530c1157cb289a064ec37f1d4591236124691\": rpc error: code = NotFound desc = could not find container \"92a28f21572fcf617065aac57f1530c1157cb289a064ec37f1d4591236124691\": container with ID starting with 92a28f21572fcf617065aac57f1530c1157cb289a064ec37f1d4591236124691 not found: ID does not exist" Oct 31 12:52:18.329018 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.328950 5693 scope.go:122] "RemoveContainer" containerID="84ecd081bcf7eacf2460045d2e9335c3865d981dce932ba67c9668bbb9472e82" Oct 31 12:52:18.329223 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:52:18.329208 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84ecd081bcf7eacf2460045d2e9335c3865d981dce932ba67c9668bbb9472e82\": container with ID starting with 84ecd081bcf7eacf2460045d2e9335c3865d981dce932ba67c9668bbb9472e82 not found: ID does not exist" containerID="84ecd081bcf7eacf2460045d2e9335c3865d981dce932ba67c9668bbb9472e82" Oct 31 12:52:18.329265 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.329227 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84ecd081bcf7eacf2460045d2e9335c3865d981dce932ba67c9668bbb9472e82"} err="failed to get container status \"84ecd081bcf7eacf2460045d2e9335c3865d981dce932ba67c9668bbb9472e82\": rpc error: code = NotFound desc = could not find container \"84ecd081bcf7eacf2460045d2e9335c3865d981dce932ba67c9668bbb9472e82\": container with ID starting with 84ecd081bcf7eacf2460045d2e9335c3865d981dce932ba67c9668bbb9472e82 not found: ID does not exist" Oct 31 12:52:18.345405 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.345379 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck"] Oct 31 12:52:18.351184 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.351156 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-cf7e6604-gw-openshift-default-5f6bbcdc7c-nqzck"] Oct 31 12:52:18.459882 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.459859 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b03bbf98-6844c68f55-k967j" Oct 31 12:52:18.493334 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.493291 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c81fe16d-d0a1-4976-8801-b84f46cccaed-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c81fe16d-d0a1-4976-8801-b84f46cccaed-maas-api-tls\") pod \"c81fe16d-d0a1-4976-8801-b84f46cccaed\" (UID: \"c81fe16d-d0a1-4976-8801-b84f46cccaed\") " Oct 31 12:52:18.493808 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.493774 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c81fe16d-d0a1-4976-8801-b84f46cccaed-kube-api-access-rf9bb\" (UniqueName: \"kubernetes.io/projected/c81fe16d-d0a1-4976-8801-b84f46cccaed-kube-api-access-rf9bb\") pod \"c81fe16d-d0a1-4976-8801-b84f46cccaed\" (UID: \"c81fe16d-d0a1-4976-8801-b84f46cccaed\") " Oct 31 12:52:18.494520 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.494494 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c81fe16d-d0a1-4976-8801-b84f46cccaed-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c81fe16d-d0a1-4976-8801-b84f46cccaed-openshift-service-ca\") pod \"c81fe16d-d0a1-4976-8801-b84f46cccaed\" (UID: \"c81fe16d-d0a1-4976-8801-b84f46cccaed\") " Oct 31 12:52:18.494734 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.494716 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c81fe16d-d0a1-4976-8801-b84f46cccaed-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c81fe16d-d0a1-4976-8801-b84f46cccaed-metrics-certs\") pod \"c81fe16d-d0a1-4976-8801-b84f46cccaed\" (UID: \"c81fe16d-d0a1-4976-8801-b84f46cccaed\") " Oct 31 12:52:18.495226 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.495202 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c81fe16d-d0a1-4976-8801-b84f46cccaed-openshift-service-ca" pod "c81fe16d-d0a1-4976-8801-b84f46cccaed" (UID: "c81fe16d-d0a1-4976-8801-b84f46cccaed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:52:18.497358 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.495615 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c81fe16d-d0a1-4976-8801-b84f46cccaed-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:18.497358 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.496388 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c81fe16d-d0a1-4976-8801-b84f46cccaed-maas-api-tls" pod "c81fe16d-d0a1-4976-8801-b84f46cccaed" (UID: "c81fe16d-d0a1-4976-8801-b84f46cccaed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:52:18.498074 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.498049 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c81fe16d-d0a1-4976-8801-b84f46cccaed-kube-api-access-rf9bb" pod "c81fe16d-d0a1-4976-8801-b84f46cccaed" (UID: "c81fe16d-d0a1-4976-8801-b84f46cccaed"). InnerVolumeSpecName "kube-api-access-rf9bb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:52:18.499297 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.499277 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c81fe16d-d0a1-4976-8801-b84f46cccaed-metrics-certs" pod "c81fe16d-d0a1-4976-8801-b84f46cccaed" (UID: "c81fe16d-d0a1-4976-8801-b84f46cccaed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:52:18.597265 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.597185 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c81fe16d-d0a1-4976-8801-b84f46cccaed-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:18.597265 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.597221 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c81fe16d-d0a1-4976-8801-b84f46cccaed-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:18.597265 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.597238 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf9bb\" (UniqueName: \"kubernetes.io/projected/c81fe16d-d0a1-4976-8801-b84f46cccaed-kube-api-access-rf9bb\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:18.687289 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:52:18.687198 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:52:18.755255 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.755216 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-47bcd9-5c675b8fdc-6gsn9"] Oct 31 12:52:18.755693 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.755678 5693 cpu_manager.go:414] "containerMap: removing container" podUID="bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c" containerName="istio-proxy" Oct 31 12:52:18.755693 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.755693 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c" containerName="istio-proxy" Oct 31 12:52:18.755842 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.755729 5693 cpu_manager.go:414] "containerMap: removing container" podUID="c81fe16d-d0a1-4976-8801-b84f46cccaed" containerName="maas-api" Oct 31 12:52:18.755842 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.755734 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c81fe16d-d0a1-4976-8801-b84f46cccaed" containerName="maas-api" Oct 31 12:52:18.755842 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.755752 5693 cpu_manager.go:414] "containerMap: removing container" podUID="0ac16366-4faa-45e8-89a2-8b5c82f4bf29" containerName="maas-api" Oct 31 12:52:18.755842 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.755761 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ac16366-4faa-45e8-89a2-8b5c82f4bf29" containerName="maas-api" Oct 31 12:52:18.756058 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.755847 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="c81fe16d-d0a1-4976-8801-b84f46cccaed" containerName="maas-api" Oct 31 12:52:18.756058 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.755855 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ac16366-4faa-45e8-89a2-8b5c82f4bf29" containerName="maas-api" Oct 31 12:52:18.756058 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.755862 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c" containerName="istio-proxy" Oct 31 12:52:18.756058 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.755868 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c" containerName="istio-proxy" Oct 31 12:52:18.759248 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.759229 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-47bcd9-5c675b8fdc-6gsn9" Oct 31 12:52:18.774879 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.774849 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-47bcd9-dockercfg-ckjbb\"" Oct 31 12:52:18.782425 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.782400 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-47bcd9-5c675b8fdc-6gsn9"] Oct 31 12:52:18.789692 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.787136 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-47bcd9-66cc57988c-9hldq"] Oct 31 12:52:18.789692 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.787705 5693 cpu_manager.go:414] "containerMap: removing container" podUID="bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c" containerName="istio-proxy" Oct 31 12:52:18.789692 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.787719 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c" containerName="istio-proxy" Oct 31 12:52:18.790666 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.790627 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-47bcd9-66cc57988c-9hldq" Oct 31 12:52:18.799904 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.798471 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-47bcd9-5c675b8fdc-6gsn9\" (UID: \"c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-47bcd9-5c675b8fdc-6gsn9" Oct 31 12:52:18.799904 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.798544 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nht6n\" (UniqueName: \"kubernetes.io/projected/c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5-kube-api-access-nht6n\") pod \"payload-pre-processing-e2e-worker-w3-47bcd9-5c675b8fdc-6gsn9\" (UID: \"c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-47bcd9-5c675b8fdc-6gsn9" Oct 31 12:52:18.804916 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.804892 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-47bcd9-66cc57988c-9hldq"] Oct 31 12:52:18.867867 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.867843 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-47bcd9\"" Oct 31 12:52:18.899573 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.899533 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-47bcd9-66cc57988c-9hldq\" (UID: \"f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-47bcd9-66cc57988c-9hldq" Oct 31 12:52:18.899850 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.899699 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzxs2\" (UniqueName: \"kubernetes.io/projected/f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0-kube-api-access-fzxs2\") pod \"payload-processing-e2e-worker-w3-47bcd9-66cc57988c-9hldq\" (UID: \"f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-47bcd9-66cc57988c-9hldq" Oct 31 12:52:18.899850 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.899815 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-47bcd9-5c675b8fdc-6gsn9\" (UID: \"c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-47bcd9-5c675b8fdc-6gsn9" Oct 31 12:52:18.899982 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.899886 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nht6n\" (UniqueName: \"kubernetes.io/projected/c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5-kube-api-access-nht6n\") pod \"payload-pre-processing-e2e-worker-w3-47bcd9-5c675b8fdc-6gsn9\" (UID: \"c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-47bcd9-5c675b8fdc-6gsn9" Oct 31 12:52:18.901149 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.901117 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-47bcd9-5c675b8fdc-6gsn9\" (UID: \"c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-47bcd9-5c675b8fdc-6gsn9" Oct 31 12:52:18.914218 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:18.914188 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nht6n\" (UniqueName: \"kubernetes.io/projected/c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5-kube-api-access-nht6n\") pod \"payload-pre-processing-e2e-worker-w3-47bcd9-5c675b8fdc-6gsn9\" (UID: \"c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-47bcd9-5c675b8fdc-6gsn9" Oct 31 12:52:19.000607 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:19.000527 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-47bcd9-66cc57988c-9hldq\" (UID: \"f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-47bcd9-66cc57988c-9hldq" Oct 31 12:52:19.000799 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:19.000611 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzxs2\" (UniqueName: \"kubernetes.io/projected/f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0-kube-api-access-fzxs2\") pod \"payload-processing-e2e-worker-w3-47bcd9-66cc57988c-9hldq\" (UID: \"f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-47bcd9-66cc57988c-9hldq" Oct 31 12:52:19.001594 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:19.001576 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-47bcd9-66cc57988c-9hldq\" (UID: \"f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-47bcd9-66cc57988c-9hldq" Oct 31 12:52:19.014597 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:19.014572 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzxs2\" (UniqueName: \"kubernetes.io/projected/f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0-kube-api-access-fzxs2\") pod \"payload-processing-e2e-worker-w3-47bcd9-66cc57988c-9hldq\" (UID: \"f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-47bcd9-66cc57988c-9hldq" Oct 31 12:52:19.074815 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:19.074785 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-47bcd9-5c675b8fdc-6gsn9" Oct 31 12:52:19.107268 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:19.107237 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-47bcd9-66cc57988c-9hldq" Oct 31 12:52:19.315837 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:19.315731 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b03bbf98-6844c68f55-k967j" Oct 31 12:52:19.315837 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:19.315755 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b03bbf98-6844c68f55-k967j" event={"ID":"c81fe16d-d0a1-4976-8801-b84f46cccaed","Type":"ContainerDied","Data":"2e9197703be483a1a3abfdcdb2084ce811ac8502893d35847234678ccfff37a4"} Oct 31 12:52:19.315837 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:19.315830 5693 scope.go:122] "RemoveContainer" containerID="e0a24ade385c970ad834fedee142d550c27af2fac9f93a6cf024aaf66c7f06b1" Oct 31 12:52:19.341364 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:19.341335 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b03bbf98-6844c68f55-k967j"] Oct 31 12:52:19.363897 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:19.363868 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b03bbf98-6844c68f55-k967j"] Oct 31 12:52:19.365839 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:19.365816 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-47bcd9-5c675b8fdc-6gsn9"] Oct 31 12:52:19.368987 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:52:19.368951 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc92a5443_7c18_4eb7_b2b5_cfdcfd1e25a5.slice/crio-e786140e74e77031f8bc4765b98039709dabc64ce737b8f0ab576265b63e0e85 WatchSource:0}: Error finding container e786140e74e77031f8bc4765b98039709dabc64ce737b8f0ab576265b63e0e85: Status 404 returned error can't find the container with id e786140e74e77031f8bc4765b98039709dabc64ce737b8f0ab576265b63e0e85 Oct 31 12:52:19.408351 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:19.408311 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c" path="/var/lib/kubelet/pods/bc14cc0b-8b70-4999-b4b4-f9a0aa1f5d5c/volumes" Oct 31 12:52:19.409127 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:19.409101 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c81fe16d-d0a1-4976-8801-b84f46cccaed" path="/var/lib/kubelet/pods/c81fe16d-d0a1-4976-8801-b84f46cccaed/volumes" Oct 31 12:52:19.409539 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:19.409522 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-47bcd9-66cc57988c-9hldq"] Oct 31 12:52:19.410724 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:52:19.410698 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8e5aafa_21a8_4daa_a44d_90a3cc1a74c0.slice/crio-2c49f3b87fc84423f317da073fbe43ee521a9feadb86dfa34d730efe358378f5 WatchSource:0}: Error finding container 2c49f3b87fc84423f317da073fbe43ee521a9feadb86dfa34d730efe358378f5: Status 404 returned error can't find the container with id 2c49f3b87fc84423f317da073fbe43ee521a9feadb86dfa34d730efe358378f5 Oct 31 12:52:20.324350 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:20.324311 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-47bcd9-5c675b8fdc-6gsn9" event={"ID":"c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5","Type":"ContainerStarted","Data":"2b47cff06188d7e60fa1c9502cf828c068352e0567da15475faff2bedbd1f67b"} Oct 31 12:52:20.324825 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:20.324395 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-47bcd9-5c675b8fdc-6gsn9" event={"ID":"c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5","Type":"ContainerStarted","Data":"e786140e74e77031f8bc4765b98039709dabc64ce737b8f0ab576265b63e0e85"} Oct 31 12:52:20.324825 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:20.324424 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-47bcd9-5c675b8fdc-6gsn9" Oct 31 12:52:20.325836 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:20.325815 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-47bcd9-66cc57988c-9hldq" event={"ID":"f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0","Type":"ContainerStarted","Data":"d712642f3c9c668fe14e4457f173f326aa0d84e509810c1def44ce9230e7c7ea"} Oct 31 12:52:20.325961 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:20.325849 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-47bcd9-66cc57988c-9hldq" event={"ID":"f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0","Type":"ContainerStarted","Data":"2c49f3b87fc84423f317da073fbe43ee521a9feadb86dfa34d730efe358378f5"} Oct 31 12:52:20.325961 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:20.325869 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-47bcd9-66cc57988c-9hldq" Oct 31 12:52:20.348722 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:20.348673 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-47bcd9-5c675b8fdc-6gsn9" podStartSLOduration=2.348656984 podStartE2EDuration="2.348656984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:52:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:52:20.347020243 +0000 UTC m=+1191.054606593" watchObservedRunningTime="2026-10-31 12:52:20.348656984 +0000 UTC m=+1191.056243277" Oct 31 12:52:20.375322 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:20.375273 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-47bcd9-66cc57988c-9hldq" podStartSLOduration=2.37526029 podStartE2EDuration="2.37526029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:52:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:52:20.37302796 +0000 UTC m=+1191.080614257" watchObservedRunningTime="2026-10-31 12:52:20.37526029 +0000 UTC m=+1191.082846584" Oct 31 12:52:21.266694 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:21.266606 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b03bbf98-77cd8cfdbc-gl7bg" podUID="bb36a5c9-b8b9-4b0d-a247-6ba4f35a8b34" containerName="payload-processing" containerID="cri-o://a49a4f13779cd621162937d4bb4503bbb511523357c7639c1afeda9bb3e253c0" gracePeriod=25 Oct 31 12:52:21.523158 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:21.523102 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b03bbf98-77cd8cfdbc-gl7bg" Oct 31 12:52:21.627374 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:21.627346 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb36a5c9-b8b9-4b0d-a247-6ba4f35a8b34-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb36a5c9-b8b9-4b0d-a247-6ba4f35a8b34-plugins-config-volume\") pod \"bb36a5c9-b8b9-4b0d-a247-6ba4f35a8b34\" (UID: \"bb36a5c9-b8b9-4b0d-a247-6ba4f35a8b34\") " Oct 31 12:52:21.627529 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:21.627507 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb36a5c9-b8b9-4b0d-a247-6ba4f35a8b34-kube-api-access-qzpfr\" (UniqueName: \"kubernetes.io/projected/bb36a5c9-b8b9-4b0d-a247-6ba4f35a8b34-kube-api-access-qzpfr\") pod \"bb36a5c9-b8b9-4b0d-a247-6ba4f35a8b34\" (UID: \"bb36a5c9-b8b9-4b0d-a247-6ba4f35a8b34\") " Oct 31 12:52:21.627769 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:21.627747 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb36a5c9-b8b9-4b0d-a247-6ba4f35a8b34-plugins-config-volume" pod "bb36a5c9-b8b9-4b0d-a247-6ba4f35a8b34" (UID: "bb36a5c9-b8b9-4b0d-a247-6ba4f35a8b34"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:52:21.629896 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:21.629864 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb36a5c9-b8b9-4b0d-a247-6ba4f35a8b34-kube-api-access-qzpfr" pod "bb36a5c9-b8b9-4b0d-a247-6ba4f35a8b34" (UID: "bb36a5c9-b8b9-4b0d-a247-6ba4f35a8b34"). InnerVolumeSpecName "kube-api-access-qzpfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:52:21.728936 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:21.728907 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzpfr\" (UniqueName: \"kubernetes.io/projected/bb36a5c9-b8b9-4b0d-a247-6ba4f35a8b34-kube-api-access-qzpfr\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:21.728936 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:21.728930 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb36a5c9-b8b9-4b0d-a247-6ba4f35a8b34-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:21.876873 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:21.876786 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8"] Oct 31 12:52:21.877465 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:21.877445 5693 cpu_manager.go:414] "containerMap: removing container" podUID="bb36a5c9-b8b9-4b0d-a247-6ba4f35a8b34" containerName="payload-processing" Oct 31 12:52:21.877527 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:21.877471 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb36a5c9-b8b9-4b0d-a247-6ba4f35a8b34" containerName="payload-processing" Oct 31 12:52:21.877627 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:21.877612 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb36a5c9-b8b9-4b0d-a247-6ba4f35a8b34" containerName="payload-processing" Oct 31 12:52:21.880484 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:21.880456 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8" Oct 31 12:52:21.889174 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:21.889151 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-47bcd9-serving-cert\"" Oct 31 12:52:21.899857 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:21.899832 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8"] Oct 31 12:52:21.931324 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:21.931289 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6421f4c-ffa6-4da2-91b6-81f2cb8ab421-maas-api-tls\") pod \"maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8\" (UID: \"f6421f4c-ffa6-4da2-91b6-81f2cb8ab421\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8" Oct 31 12:52:21.931487 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:21.931337 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6421f4c-ffa6-4da2-91b6-81f2cb8ab421-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8\" (UID: \"f6421f4c-ffa6-4da2-91b6-81f2cb8ab421\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8" Oct 31 12:52:21.931487 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:21.931448 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6421f4c-ffa6-4da2-91b6-81f2cb8ab421-metrics-certs\") pod \"maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8\" (UID: \"f6421f4c-ffa6-4da2-91b6-81f2cb8ab421\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8" Oct 31 12:52:21.931618 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:21.931497 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvq9j\" (UniqueName: \"kubernetes.io/projected/f6421f4c-ffa6-4da2-91b6-81f2cb8ab421-kube-api-access-fvq9j\") pod \"maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8\" (UID: \"f6421f4c-ffa6-4da2-91b6-81f2cb8ab421\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8" Oct 31 12:52:22.032576 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:22.032543 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6421f4c-ffa6-4da2-91b6-81f2cb8ab421-maas-api-tls\") pod \"maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8\" (UID: \"f6421f4c-ffa6-4da2-91b6-81f2cb8ab421\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8" Oct 31 12:52:22.032758 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:22.032613 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6421f4c-ffa6-4da2-91b6-81f2cb8ab421-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8\" (UID: \"f6421f4c-ffa6-4da2-91b6-81f2cb8ab421\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8" Oct 31 12:52:22.032758 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:22.032709 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6421f4c-ffa6-4da2-91b6-81f2cb8ab421-metrics-certs\") pod \"maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8\" (UID: \"f6421f4c-ffa6-4da2-91b6-81f2cb8ab421\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8" Oct 31 12:52:22.032882 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:22.032776 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvq9j\" (UniqueName: \"kubernetes.io/projected/f6421f4c-ffa6-4da2-91b6-81f2cb8ab421-kube-api-access-fvq9j\") pod \"maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8\" (UID: \"f6421f4c-ffa6-4da2-91b6-81f2cb8ab421\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8" Oct 31 12:52:22.033577 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:22.033547 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6421f4c-ffa6-4da2-91b6-81f2cb8ab421-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8\" (UID: \"f6421f4c-ffa6-4da2-91b6-81f2cb8ab421\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8" Oct 31 12:52:22.035206 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:22.035184 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6421f4c-ffa6-4da2-91b6-81f2cb8ab421-metrics-certs\") pod \"maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8\" (UID: \"f6421f4c-ffa6-4da2-91b6-81f2cb8ab421\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8" Oct 31 12:52:22.035311 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:22.035254 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6421f4c-ffa6-4da2-91b6-81f2cb8ab421-maas-api-tls\") pod \"maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8\" (UID: \"f6421f4c-ffa6-4da2-91b6-81f2cb8ab421\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8" Oct 31 12:52:22.047623 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:22.047596 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvq9j\" (UniqueName: \"kubernetes.io/projected/f6421f4c-ffa6-4da2-91b6-81f2cb8ab421-kube-api-access-fvq9j\") pod \"maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8\" (UID: \"f6421f4c-ffa6-4da2-91b6-81f2cb8ab421\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8" Oct 31 12:52:22.194835 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:22.194798 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8" Oct 31 12:52:22.336432 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:22.336402 5693 generic.go:356] "Generic (PLEG): container finished" podID="bb36a5c9-b8b9-4b0d-a247-6ba4f35a8b34" containerID="a49a4f13779cd621162937d4bb4503bbb511523357c7639c1afeda9bb3e253c0" exitCode=0 Oct 31 12:52:22.336589 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:22.336453 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b03bbf98-77cd8cfdbc-gl7bg" Oct 31 12:52:22.336589 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:22.336489 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b03bbf98-77cd8cfdbc-gl7bg" event={"ID":"bb36a5c9-b8b9-4b0d-a247-6ba4f35a8b34","Type":"ContainerDied","Data":"a49a4f13779cd621162937d4bb4503bbb511523357c7639c1afeda9bb3e253c0"} Oct 31 12:52:22.336589 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:22.336545 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b03bbf98-77cd8cfdbc-gl7bg" event={"ID":"bb36a5c9-b8b9-4b0d-a247-6ba4f35a8b34","Type":"ContainerDied","Data":"024b60898384fb10e6b85a5508d9a282a5be5a92e7030d703578d33e1f429a62"} Oct 31 12:52:22.336589 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:22.336573 5693 scope.go:122] "RemoveContainer" containerID="a49a4f13779cd621162937d4bb4503bbb511523357c7639c1afeda9bb3e253c0" Oct 31 12:52:22.345834 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:22.345818 5693 scope.go:122] "RemoveContainer" containerID="a49a4f13779cd621162937d4bb4503bbb511523357c7639c1afeda9bb3e253c0" Oct 31 12:52:22.346136 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:52:22.346115 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a49a4f13779cd621162937d4bb4503bbb511523357c7639c1afeda9bb3e253c0\": container with ID starting with a49a4f13779cd621162937d4bb4503bbb511523357c7639c1afeda9bb3e253c0 not found: ID does not exist" containerID="a49a4f13779cd621162937d4bb4503bbb511523357c7639c1afeda9bb3e253c0" Oct 31 12:52:22.346205 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:22.346147 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a49a4f13779cd621162937d4bb4503bbb511523357c7639c1afeda9bb3e253c0"} err="failed to get container status \"a49a4f13779cd621162937d4bb4503bbb511523357c7639c1afeda9bb3e253c0\": rpc error: code = NotFound desc = could not find container \"a49a4f13779cd621162937d4bb4503bbb511523357c7639c1afeda9bb3e253c0\": container with ID starting with a49a4f13779cd621162937d4bb4503bbb511523357c7639c1afeda9bb3e253c0 not found: ID does not exist" Oct 31 12:52:22.361596 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:22.361572 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b03bbf98-77cd8cfdbc-gl7bg"] Oct 31 12:52:22.368541 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:22.368519 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b03bbf98-77cd8cfdbc-gl7bg"] Oct 31 12:52:22.475460 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:22.475439 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8"] Oct 31 12:52:22.477236 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:52:22.477208 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6421f4c_ffa6_4da2_91b6_81f2cb8ab421.slice/crio-b8ff809d970073e6def9eb6ded883d30b255edbd2a2aad5cda9209d6b931f555 WatchSource:0}: Error finding container b8ff809d970073e6def9eb6ded883d30b255edbd2a2aad5cda9209d6b931f555: Status 404 returned error can't find the container with id b8ff809d970073e6def9eb6ded883d30b255edbd2a2aad5cda9209d6b931f555 Oct 31 12:52:23.344211 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:23.344177 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8" event={"ID":"f6421f4c-ffa6-4da2-91b6-81f2cb8ab421","Type":"ContainerStarted","Data":"93bb269d0359be794bae4b5171616da4ef00e70baab9a40ec8abaf5c4047e283"} Oct 31 12:52:23.344613 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:23.344226 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8" event={"ID":"f6421f4c-ffa6-4da2-91b6-81f2cb8ab421","Type":"ContainerStarted","Data":"b8ff809d970073e6def9eb6ded883d30b255edbd2a2aad5cda9209d6b931f555"} Oct 31 12:52:23.344613 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:23.344291 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8" Oct 31 12:52:23.366029 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:23.365986 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8" podStartSLOduration=1.973787702 podStartE2EDuration="2.365970056s" totalImagesPullingTime="392.182354ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:52:23.365265167 +0000 UTC m=+1194.072851466" watchObservedRunningTime="2026-10-31 12:52:23.365970056 +0000 UTC m=+1194.073556353" Oct 31 12:52:23.406758 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:23.406722 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb36a5c9-b8b9-4b0d-a247-6ba4f35a8b34" path="/var/lib/kubelet/pods/bb36a5c9-b8b9-4b0d-a247-6ba4f35a8b34/volumes" Oct 31 12:52:27.291320 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:27.291281 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d48f454a-67bddb4b87-75psm"] Oct 31 12:52:27.327841 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:27.327799 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m"] Oct 31 12:52:27.328208 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:27.328080 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m" podUID="72445e01-d558-4fd1-80fd-05a86379726a" containerName="maas-api" containerID="cri-o://ac642ad678b3b95dbb240f072c674327ac13c6300c77fa0ce6202e0936745f02" gracePeriod=30 Oct 31 12:52:27.593598 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:27.593574 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m" Oct 31 12:52:27.688753 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:27.688721 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72445e01-d558-4fd1-80fd-05a86379726a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72445e01-d558-4fd1-80fd-05a86379726a-openshift-service-ca\") pod \"72445e01-d558-4fd1-80fd-05a86379726a\" (UID: \"72445e01-d558-4fd1-80fd-05a86379726a\") " Oct 31 12:52:27.688950 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:27.688824 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72445e01-d558-4fd1-80fd-05a86379726a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72445e01-d558-4fd1-80fd-05a86379726a-maas-api-tls\") pod \"72445e01-d558-4fd1-80fd-05a86379726a\" (UID: \"72445e01-d558-4fd1-80fd-05a86379726a\") " Oct 31 12:52:27.688950 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:27.688902 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72445e01-d558-4fd1-80fd-05a86379726a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/72445e01-d558-4fd1-80fd-05a86379726a-metrics-certs\") pod \"72445e01-d558-4fd1-80fd-05a86379726a\" (UID: \"72445e01-d558-4fd1-80fd-05a86379726a\") " Oct 31 12:52:27.689076 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:27.688961 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72445e01-d558-4fd1-80fd-05a86379726a-kube-api-access-mwwvk\" (UniqueName: \"kubernetes.io/projected/72445e01-d558-4fd1-80fd-05a86379726a-kube-api-access-mwwvk\") pod \"72445e01-d558-4fd1-80fd-05a86379726a\" (UID: \"72445e01-d558-4fd1-80fd-05a86379726a\") " Oct 31 12:52:27.689177 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:27.689152 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72445e01-d558-4fd1-80fd-05a86379726a-openshift-service-ca" pod "72445e01-d558-4fd1-80fd-05a86379726a" (UID: "72445e01-d558-4fd1-80fd-05a86379726a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:52:27.689388 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:27.689369 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72445e01-d558-4fd1-80fd-05a86379726a-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:27.691043 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:27.691020 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72445e01-d558-4fd1-80fd-05a86379726a-metrics-certs" pod "72445e01-d558-4fd1-80fd-05a86379726a" (UID: "72445e01-d558-4fd1-80fd-05a86379726a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:52:27.691129 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:27.691046 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72445e01-d558-4fd1-80fd-05a86379726a-maas-api-tls" pod "72445e01-d558-4fd1-80fd-05a86379726a" (UID: "72445e01-d558-4fd1-80fd-05a86379726a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:52:27.691213 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:27.691189 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72445e01-d558-4fd1-80fd-05a86379726a-kube-api-access-mwwvk" pod "72445e01-d558-4fd1-80fd-05a86379726a" (UID: "72445e01-d558-4fd1-80fd-05a86379726a"). InnerVolumeSpecName "kube-api-access-mwwvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:52:27.790190 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:27.790151 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72445e01-d558-4fd1-80fd-05a86379726a-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:27.790190 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:27.790178 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwwvk\" (UniqueName: \"kubernetes.io/projected/72445e01-d558-4fd1-80fd-05a86379726a-kube-api-access-mwwvk\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:27.790190 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:27.790189 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72445e01-d558-4fd1-80fd-05a86379726a-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:28.222652 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:28.222606 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn"] Oct 31 12:52:28.222944 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:28.222897 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" podUID="48005f27-24cb-4a42-8b06-51a2254c3473" containerName="istio-proxy" containerID="cri-o://2314a90358a3e21a8d2295847000c3db99c131b54dccd7490d292a28a0f3ece2" gracePeriod=30 Oct 31 12:52:28.372887 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:28.372854 5693 generic.go:356] "Generic (PLEG): container finished" podID="72445e01-d558-4fd1-80fd-05a86379726a" containerID="ac642ad678b3b95dbb240f072c674327ac13c6300c77fa0ce6202e0936745f02" exitCode=0 Oct 31 12:52:28.373357 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:28.372910 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m" Oct 31 12:52:28.373357 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:28.372939 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m" event={"ID":"72445e01-d558-4fd1-80fd-05a86379726a","Type":"ContainerDied","Data":"ac642ad678b3b95dbb240f072c674327ac13c6300c77fa0ce6202e0936745f02"} Oct 31 12:52:28.373357 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:28.372992 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m" event={"ID":"72445e01-d558-4fd1-80fd-05a86379726a","Type":"ContainerDied","Data":"78e7785325f6d3338594623ef10711630e76a3dab16968fb11f67b73e1946a14"} Oct 31 12:52:28.373357 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:28.373014 5693 scope.go:122] "RemoveContainer" containerID="ac642ad678b3b95dbb240f072c674327ac13c6300c77fa0ce6202e0936745f02" Oct 31 12:52:28.383450 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:28.383427 5693 scope.go:122] "RemoveContainer" containerID="ac642ad678b3b95dbb240f072c674327ac13c6300c77fa0ce6202e0936745f02" Oct 31 12:52:28.383781 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:52:28.383759 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac642ad678b3b95dbb240f072c674327ac13c6300c77fa0ce6202e0936745f02\": container with ID starting with ac642ad678b3b95dbb240f072c674327ac13c6300c77fa0ce6202e0936745f02 not found: ID does not exist" containerID="ac642ad678b3b95dbb240f072c674327ac13c6300c77fa0ce6202e0936745f02" Oct 31 12:52:28.383890 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:28.383787 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac642ad678b3b95dbb240f072c674327ac13c6300c77fa0ce6202e0936745f02"} err="failed to get container status \"ac642ad678b3b95dbb240f072c674327ac13c6300c77fa0ce6202e0936745f02\": rpc error: code = NotFound desc = could not find container \"ac642ad678b3b95dbb240f072c674327ac13c6300c77fa0ce6202e0936745f02\": container with ID starting with ac642ad678b3b95dbb240f072c674327ac13c6300c77fa0ce6202e0936745f02 not found: ID does not exist" Oct 31 12:52:28.413998 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:28.413970 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m"] Oct 31 12:52:28.415560 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:28.415537 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1abc54b4-7cc9fbb8bc-q9q7m"] Oct 31 12:52:28.464404 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:28.464368 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj"] Oct 31 12:52:28.464869 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:28.464852 5693 cpu_manager.go:414] "containerMap: removing container" podUID="72445e01-d558-4fd1-80fd-05a86379726a" containerName="maas-api" Oct 31 12:52:28.464869 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:28.464867 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72445e01-d558-4fd1-80fd-05a86379726a" containerName="maas-api" Oct 31 12:52:28.465052 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:28.464963 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="72445e01-d558-4fd1-80fd-05a86379726a" containerName="maas-api" Oct 31 12:52:28.469513 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:28.469483 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj" Oct 31 12:52:28.482739 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:28.478162 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-7ae9e7-serving-cert\"" Oct 31 12:52:28.482739 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:28.482255 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj"] Oct 31 12:52:28.599193 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:28.599160 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71e7cbcd-adef-4e0e-8944-b3c62e6407bc-maas-api-tls\") pod \"maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj\" (UID: \"71e7cbcd-adef-4e0e-8944-b3c62e6407bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj" Oct 31 12:52:28.599361 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:28.599294 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71e7cbcd-adef-4e0e-8944-b3c62e6407bc-metrics-certs\") pod \"maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj\" (UID: \"71e7cbcd-adef-4e0e-8944-b3c62e6407bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj" Oct 31 12:52:28.599361 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:28.599347 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lpmb\" (UniqueName: \"kubernetes.io/projected/71e7cbcd-adef-4e0e-8944-b3c62e6407bc-kube-api-access-2lpmb\") pod \"maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj\" (UID: \"71e7cbcd-adef-4e0e-8944-b3c62e6407bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj" Oct 31 12:52:28.599439 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:28.599389 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e7cbcd-adef-4e0e-8944-b3c62e6407bc-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj\" (UID: \"71e7cbcd-adef-4e0e-8944-b3c62e6407bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj" Oct 31 12:52:28.701190 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:28.701142 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e7cbcd-adef-4e0e-8944-b3c62e6407bc-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj\" (UID: \"71e7cbcd-adef-4e0e-8944-b3c62e6407bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj" Oct 31 12:52:28.701374 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:28.701221 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71e7cbcd-adef-4e0e-8944-b3c62e6407bc-maas-api-tls\") pod \"maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj\" (UID: \"71e7cbcd-adef-4e0e-8944-b3c62e6407bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj" Oct 31 12:52:28.701374 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:28.701360 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71e7cbcd-adef-4e0e-8944-b3c62e6407bc-metrics-certs\") pod \"maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj\" (UID: \"71e7cbcd-adef-4e0e-8944-b3c62e6407bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj" Oct 31 12:52:28.701476 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:28.701397 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lpmb\" (UniqueName: \"kubernetes.io/projected/71e7cbcd-adef-4e0e-8944-b3c62e6407bc-kube-api-access-2lpmb\") pod \"maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj\" (UID: \"71e7cbcd-adef-4e0e-8944-b3c62e6407bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj" Oct 31 12:52:28.702130 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:28.702048 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e7cbcd-adef-4e0e-8944-b3c62e6407bc-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj\" (UID: \"71e7cbcd-adef-4e0e-8944-b3c62e6407bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj" Oct 31 12:52:28.704191 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:28.704166 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71e7cbcd-adef-4e0e-8944-b3c62e6407bc-maas-api-tls\") pod \"maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj\" (UID: \"71e7cbcd-adef-4e0e-8944-b3c62e6407bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj" Oct 31 12:52:28.704191 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:28.704189 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71e7cbcd-adef-4e0e-8944-b3c62e6407bc-metrics-certs\") pod \"maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj\" (UID: \"71e7cbcd-adef-4e0e-8944-b3c62e6407bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj" Oct 31 12:52:28.714069 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:28.714044 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lpmb\" (UniqueName: \"kubernetes.io/projected/71e7cbcd-adef-4e0e-8944-b3c62e6407bc-kube-api-access-2lpmb\") pod \"maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj\" (UID: \"71e7cbcd-adef-4e0e-8944-b3c62e6407bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj" Oct 31 12:52:28.788915 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:28.788824 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj" Oct 31 12:52:29.074546 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:29.074525 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj"] Oct 31 12:52:29.077130 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:52:29.077100 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71e7cbcd_adef_4e0e_8944_b3c62e6407bc.slice/crio-9013b8d8e9e17cbd03fe2dca702661f2a59154e55278f383d187eddbe1e298fc WatchSource:0}: Error finding container 9013b8d8e9e17cbd03fe2dca702661f2a59154e55278f383d187eddbe1e298fc: Status 404 returned error can't find the container with id 9013b8d8e9e17cbd03fe2dca702661f2a59154e55278f383d187eddbe1e298fc Oct 31 12:52:29.358444 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:29.358371 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8" Oct 31 12:52:29.381956 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:29.381917 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj" event={"ID":"71e7cbcd-adef-4e0e-8944-b3c62e6407bc","Type":"ContainerStarted","Data":"9013b8d8e9e17cbd03fe2dca702661f2a59154e55278f383d187eddbe1e298fc"} Oct 31 12:52:29.411073 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:29.411024 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72445e01-d558-4fd1-80fd-05a86379726a" path="/var/lib/kubelet/pods/72445e01-d558-4fd1-80fd-05a86379726a/volumes" Oct 31 12:52:30.388882 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:30.388846 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj" event={"ID":"71e7cbcd-adef-4e0e-8944-b3c62e6407bc","Type":"ContainerStarted","Data":"e763633f260c89b77a7c7f1c6375ca2f7d4bfc336c59c1058c64c9e79c60af73"} Oct 31 12:52:30.389340 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:30.388903 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj" Oct 31 12:52:30.412746 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:30.412702 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj" podStartSLOduration=1.874093785 podStartE2EDuration="2.412688646s" totalImagesPullingTime="538.594861ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:52:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:52:30.411524764 +0000 UTC m=+1201.119111050" watchObservedRunningTime="2026-10-31 12:52:30.412688646 +0000 UTC m=+1201.120274935" Oct 31 12:52:31.333549 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:31.333520 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-47bcd9-5c675b8fdc-6gsn9" Oct 31 12:52:31.333901 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:31.333881 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-47bcd9-66cc57988c-9hldq" Oct 31 12:52:31.889510 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:31.889474 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4"] Oct 31 12:52:31.889965 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:31.889792 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4" podUID="bee8161c-23f9-4563-a8c7-506fcf777846" containerName="maas-api" containerID="cri-o://773c318899981743e1e028098b32f0545c9dea00ec0b010b07a748a8f08cc1c6" gracePeriod=30 Oct 31 12:52:32.161319 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.161295 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4" Oct 31 12:52:32.237907 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.237870 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bee8161c-23f9-4563-a8c7-506fcf777846-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bee8161c-23f9-4563-a8c7-506fcf777846-maas-api-tls\") pod \"bee8161c-23f9-4563-a8c7-506fcf777846\" (UID: \"bee8161c-23f9-4563-a8c7-506fcf777846\") " Oct 31 12:52:32.238093 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.237941 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bee8161c-23f9-4563-a8c7-506fcf777846-kube-api-access-rrcgx\" (UniqueName: \"kubernetes.io/projected/bee8161c-23f9-4563-a8c7-506fcf777846-kube-api-access-rrcgx\") pod \"bee8161c-23f9-4563-a8c7-506fcf777846\" (UID: \"bee8161c-23f9-4563-a8c7-506fcf777846\") " Oct 31 12:52:32.238093 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.237990 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bee8161c-23f9-4563-a8c7-506fcf777846-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bee8161c-23f9-4563-a8c7-506fcf777846-metrics-certs\") pod \"bee8161c-23f9-4563-a8c7-506fcf777846\" (UID: \"bee8161c-23f9-4563-a8c7-506fcf777846\") " Oct 31 12:52:32.238093 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.238048 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bee8161c-23f9-4563-a8c7-506fcf777846-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bee8161c-23f9-4563-a8c7-506fcf777846-openshift-service-ca\") pod \"bee8161c-23f9-4563-a8c7-506fcf777846\" (UID: \"bee8161c-23f9-4563-a8c7-506fcf777846\") " Oct 31 12:52:32.238486 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.238453 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bee8161c-23f9-4563-a8c7-506fcf777846-openshift-service-ca" pod "bee8161c-23f9-4563-a8c7-506fcf777846" (UID: "bee8161c-23f9-4563-a8c7-506fcf777846"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:52:32.240301 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.240275 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bee8161c-23f9-4563-a8c7-506fcf777846-maas-api-tls" pod "bee8161c-23f9-4563-a8c7-506fcf777846" (UID: "bee8161c-23f9-4563-a8c7-506fcf777846"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:52:32.240301 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.240295 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bee8161c-23f9-4563-a8c7-506fcf777846-metrics-certs" pod "bee8161c-23f9-4563-a8c7-506fcf777846" (UID: "bee8161c-23f9-4563-a8c7-506fcf777846"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:52:32.240446 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.240385 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bee8161c-23f9-4563-a8c7-506fcf777846-kube-api-access-rrcgx" pod "bee8161c-23f9-4563-a8c7-506fcf777846" (UID: "bee8161c-23f9-4563-a8c7-506fcf777846"). InnerVolumeSpecName "kube-api-access-rrcgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:52:32.292419 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.292361 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d48f454a-67bddb4b87-75psm" podUID="657aca5d-eafd-4554-80ae-a2b6d1b01987" containerName="payload-pre-processing" containerID="cri-o://d5736d32537b2d9a7298c0ddf7d50e1c1b21aac49e73ce28da7840fd040508aa" gracePeriod=25 Oct 31 12:52:32.339166 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.339135 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bee8161c-23f9-4563-a8c7-506fcf777846-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:32.339166 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.339164 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bee8161c-23f9-4563-a8c7-506fcf777846-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:32.339340 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.339174 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrcgx\" (UniqueName: \"kubernetes.io/projected/bee8161c-23f9-4563-a8c7-506fcf777846-kube-api-access-rrcgx\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:32.339340 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.339183 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bee8161c-23f9-4563-a8c7-506fcf777846-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:32.400099 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.400021 5693 generic.go:356] "Generic (PLEG): container finished" podID="bee8161c-23f9-4563-a8c7-506fcf777846" containerID="773c318899981743e1e028098b32f0545c9dea00ec0b010b07a748a8f08cc1c6" exitCode=0 Oct 31 12:52:32.400099 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.400064 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4" Oct 31 12:52:32.400274 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.400094 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4" event={"ID":"bee8161c-23f9-4563-a8c7-506fcf777846","Type":"ContainerDied","Data":"773c318899981743e1e028098b32f0545c9dea00ec0b010b07a748a8f08cc1c6"} Oct 31 12:52:32.400274 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.400183 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4" event={"ID":"bee8161c-23f9-4563-a8c7-506fcf777846","Type":"ContainerDied","Data":"b7fc87183e507c01b4f8ecb8cbe3c8b8100524ea93113ae1404f77df50cf17af"} Oct 31 12:52:32.400274 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.400210 5693 scope.go:122] "RemoveContainer" containerID="773c318899981743e1e028098b32f0545c9dea00ec0b010b07a748a8f08cc1c6" Oct 31 12:52:32.413442 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.413420 5693 scope.go:122] "RemoveContainer" containerID="773c318899981743e1e028098b32f0545c9dea00ec0b010b07a748a8f08cc1c6" Oct 31 12:52:32.414162 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:52:32.414078 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"773c318899981743e1e028098b32f0545c9dea00ec0b010b07a748a8f08cc1c6\": container with ID starting with 773c318899981743e1e028098b32f0545c9dea00ec0b010b07a748a8f08cc1c6 not found: ID does not exist" containerID="773c318899981743e1e028098b32f0545c9dea00ec0b010b07a748a8f08cc1c6" Oct 31 12:52:32.414162 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.414117 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"773c318899981743e1e028098b32f0545c9dea00ec0b010b07a748a8f08cc1c6"} err="failed to get container status \"773c318899981743e1e028098b32f0545c9dea00ec0b010b07a748a8f08cc1c6\": rpc error: code = NotFound desc = could not find container \"773c318899981743e1e028098b32f0545c9dea00ec0b010b07a748a8f08cc1c6\": container with ID starting with 773c318899981743e1e028098b32f0545c9dea00ec0b010b07a748a8f08cc1c6 not found: ID does not exist" Oct 31 12:52:32.427583 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.427527 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4"] Oct 31 12:52:32.429416 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.429392 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-785c04b9-db64b9c54-gsrz4"] Oct 31 12:52:32.553715 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.553691 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d48f454a-67bddb4b87-75psm" Oct 31 12:52:32.594146 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.594111 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz"] Oct 31 12:52:32.594882 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.594859 5693 cpu_manager.go:414] "containerMap: removing container" podUID="bee8161c-23f9-4563-a8c7-506fcf777846" containerName="maas-api" Oct 31 12:52:32.594935 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.594887 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bee8161c-23f9-4563-a8c7-506fcf777846" containerName="maas-api" Oct 31 12:52:32.594935 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.594921 5693 cpu_manager.go:414] "containerMap: removing container" podUID="657aca5d-eafd-4554-80ae-a2b6d1b01987" containerName="payload-pre-processing" Oct 31 12:52:32.594935 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.594929 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="657aca5d-eafd-4554-80ae-a2b6d1b01987" containerName="payload-pre-processing" Oct 31 12:52:32.595045 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.595033 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="bee8161c-23f9-4563-a8c7-506fcf777846" containerName="maas-api" Oct 31 12:52:32.595076 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.595054 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="657aca5d-eafd-4554-80ae-a2b6d1b01987" containerName="payload-pre-processing" Oct 31 12:52:32.598709 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.598682 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" Oct 31 12:52:32.618392 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.618363 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-3dd1e5-openshift-default-dockercfg-ktcdl\"" Oct 31 12:52:32.630085 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.630051 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz"] Oct 31 12:52:32.643169 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.643141 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/657aca5d-eafd-4554-80ae-a2b6d1b01987-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/657aca5d-eafd-4554-80ae-a2b6d1b01987-plugins-config-volume\") pod \"657aca5d-eafd-4554-80ae-a2b6d1b01987\" (UID: \"657aca5d-eafd-4554-80ae-a2b6d1b01987\") " Oct 31 12:52:32.643312 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.643225 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/657aca5d-eafd-4554-80ae-a2b6d1b01987-kube-api-access-gjg9n\" (UniqueName: \"kubernetes.io/projected/657aca5d-eafd-4554-80ae-a2b6d1b01987-kube-api-access-gjg9n\") pod \"657aca5d-eafd-4554-80ae-a2b6d1b01987\" (UID: \"657aca5d-eafd-4554-80ae-a2b6d1b01987\") " Oct 31 12:52:32.643558 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.643535 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/657aca5d-eafd-4554-80ae-a2b6d1b01987-plugins-config-volume" pod "657aca5d-eafd-4554-80ae-a2b6d1b01987" (UID: "657aca5d-eafd-4554-80ae-a2b6d1b01987"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:52:32.643735 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.643716 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/657aca5d-eafd-4554-80ae-a2b6d1b01987-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:32.646407 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.646384 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/657aca5d-eafd-4554-80ae-a2b6d1b01987-kube-api-access-gjg9n" pod "657aca5d-eafd-4554-80ae-a2b6d1b01987" (UID: "657aca5d-eafd-4554-80ae-a2b6d1b01987"). InnerVolumeSpecName "kube-api-access-gjg9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:52:32.744503 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.744410 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49a12158-0b73-4dbe-86ee-42b72e883cee-istio-token\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz\" (UID: \"49a12158-0b73-4dbe-86ee-42b72e883cee\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" Oct 31 12:52:32.744503 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.744468 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a12158-0b73-4dbe-86ee-42b72e883cee-credential-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz\" (UID: \"49a12158-0b73-4dbe-86ee-42b72e883cee\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" Oct 31 12:52:32.744735 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.744553 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49a12158-0b73-4dbe-86ee-42b72e883cee-istio-podinfo\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz\" (UID: \"49a12158-0b73-4dbe-86ee-42b72e883cee\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" Oct 31 12:52:32.744735 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.744582 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49a12158-0b73-4dbe-86ee-42b72e883cee-istio-envoy\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz\" (UID: \"49a12158-0b73-4dbe-86ee-42b72e883cee\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" Oct 31 12:52:32.744735 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.744607 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49a12158-0b73-4dbe-86ee-42b72e883cee-istiod-ca-cert\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz\" (UID: \"49a12158-0b73-4dbe-86ee-42b72e883cee\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" Oct 31 12:52:32.744735 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.744623 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a12158-0b73-4dbe-86ee-42b72e883cee-workload-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz\" (UID: \"49a12158-0b73-4dbe-86ee-42b72e883cee\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" Oct 31 12:52:32.744928 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.744814 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49a12158-0b73-4dbe-86ee-42b72e883cee-istio-data\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz\" (UID: \"49a12158-0b73-4dbe-86ee-42b72e883cee\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" Oct 31 12:52:32.744928 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.744872 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49a12158-0b73-4dbe-86ee-42b72e883cee-workload-certs\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz\" (UID: \"49a12158-0b73-4dbe-86ee-42b72e883cee\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" Oct 31 12:52:32.745026 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.745002 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwszl\" (UniqueName: \"kubernetes.io/projected/49a12158-0b73-4dbe-86ee-42b72e883cee-kube-api-access-vwszl\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz\" (UID: \"49a12158-0b73-4dbe-86ee-42b72e883cee\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" Oct 31 12:52:32.745159 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.745141 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjg9n\" (UniqueName: \"kubernetes.io/projected/657aca5d-eafd-4554-80ae-a2b6d1b01987-kube-api-access-gjg9n\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:32.845732 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.845690 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwszl\" (UniqueName: \"kubernetes.io/projected/49a12158-0b73-4dbe-86ee-42b72e883cee-kube-api-access-vwszl\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz\" (UID: \"49a12158-0b73-4dbe-86ee-42b72e883cee\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" Oct 31 12:52:32.845938 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.845756 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49a12158-0b73-4dbe-86ee-42b72e883cee-istio-token\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz\" (UID: \"49a12158-0b73-4dbe-86ee-42b72e883cee\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" Oct 31 12:52:32.845938 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.845803 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a12158-0b73-4dbe-86ee-42b72e883cee-credential-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz\" (UID: \"49a12158-0b73-4dbe-86ee-42b72e883cee\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" Oct 31 12:52:32.845938 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.845825 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49a12158-0b73-4dbe-86ee-42b72e883cee-istio-podinfo\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz\" (UID: \"49a12158-0b73-4dbe-86ee-42b72e883cee\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" Oct 31 12:52:32.845938 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.845868 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49a12158-0b73-4dbe-86ee-42b72e883cee-istio-envoy\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz\" (UID: \"49a12158-0b73-4dbe-86ee-42b72e883cee\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" Oct 31 12:52:32.845938 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.845897 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49a12158-0b73-4dbe-86ee-42b72e883cee-istiod-ca-cert\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz\" (UID: \"49a12158-0b73-4dbe-86ee-42b72e883cee\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" Oct 31 12:52:32.845938 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.845923 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a12158-0b73-4dbe-86ee-42b72e883cee-workload-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz\" (UID: \"49a12158-0b73-4dbe-86ee-42b72e883cee\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" Oct 31 12:52:32.846251 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.845975 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49a12158-0b73-4dbe-86ee-42b72e883cee-istio-data\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz\" (UID: \"49a12158-0b73-4dbe-86ee-42b72e883cee\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" Oct 31 12:52:32.846251 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.846040 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49a12158-0b73-4dbe-86ee-42b72e883cee-workload-certs\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz\" (UID: \"49a12158-0b73-4dbe-86ee-42b72e883cee\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" Oct 31 12:52:32.846251 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.846226 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a12158-0b73-4dbe-86ee-42b72e883cee-credential-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz\" (UID: \"49a12158-0b73-4dbe-86ee-42b72e883cee\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" Oct 31 12:52:32.846406 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.846376 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a12158-0b73-4dbe-86ee-42b72e883cee-workload-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz\" (UID: \"49a12158-0b73-4dbe-86ee-42b72e883cee\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" Oct 31 12:52:32.846464 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.846437 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49a12158-0b73-4dbe-86ee-42b72e883cee-workload-certs\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz\" (UID: \"49a12158-0b73-4dbe-86ee-42b72e883cee\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" Oct 31 12:52:32.846522 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.846510 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49a12158-0b73-4dbe-86ee-42b72e883cee-istio-data\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz\" (UID: \"49a12158-0b73-4dbe-86ee-42b72e883cee\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" Oct 31 12:52:32.846724 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.846703 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49a12158-0b73-4dbe-86ee-42b72e883cee-istiod-ca-cert\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz\" (UID: \"49a12158-0b73-4dbe-86ee-42b72e883cee\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" Oct 31 12:52:32.848385 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.848368 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49a12158-0b73-4dbe-86ee-42b72e883cee-istio-envoy\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz\" (UID: \"49a12158-0b73-4dbe-86ee-42b72e883cee\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" Oct 31 12:52:32.848536 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.848521 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49a12158-0b73-4dbe-86ee-42b72e883cee-istio-podinfo\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz\" (UID: \"49a12158-0b73-4dbe-86ee-42b72e883cee\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" Oct 31 12:52:32.860122 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.860099 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49a12158-0b73-4dbe-86ee-42b72e883cee-istio-token\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz\" (UID: \"49a12158-0b73-4dbe-86ee-42b72e883cee\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" Oct 31 12:52:32.860423 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.860404 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwszl\" (UniqueName: \"kubernetes.io/projected/49a12158-0b73-4dbe-86ee-42b72e883cee-kube-api-access-vwszl\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz\" (UID: \"49a12158-0b73-4dbe-86ee-42b72e883cee\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" Oct 31 12:52:32.914328 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:32.914086 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" Oct 31 12:52:33.207708 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.207680 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz"] Oct 31 12:52:33.212245 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.212207 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:52:33.212343 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.212304 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:52:33.212387 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.212355 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:52:33.386324 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:52:33.386287 5693 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee8161c_23f9_4563_a8c7_506fcf777846.slice/crio-b7fc87183e507c01b4f8ecb8cbe3c8b8100524ea93113ae1404f77df50cf17af\": RecentStats: unable to find data in memory cache]" Oct 31 12:52:33.412183 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.412136 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bee8161c-23f9-4563-a8c7-506fcf777846" path="/var/lib/kubelet/pods/bee8161c-23f9-4563-a8c7-506fcf777846/volumes" Oct 31 12:52:33.437669 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.434820 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" event={"ID":"49a12158-0b73-4dbe-86ee-42b72e883cee","Type":"ContainerStarted","Data":"e4724401ffda94aaa10ed5c3cb501889a22d50f9b1ba1e9ba5fe6d132089d8b0"} Oct 31 12:52:33.437669 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.434870 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" event={"ID":"49a12158-0b73-4dbe-86ee-42b72e883cee","Type":"ContainerStarted","Data":"22a4fbc0d8850ff8cb3b3d1c8d45a8dfb532d29aacb8c67c0d62180e91ce604f"} Oct 31 12:52:33.438670 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.438630 5693 generic.go:356] "Generic (PLEG): container finished" podID="48005f27-24cb-4a42-8b06-51a2254c3473" containerID="2314a90358a3e21a8d2295847000c3db99c131b54dccd7490d292a28a0f3ece2" exitCode=0 Oct 31 12:52:33.443674 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.438811 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" event={"ID":"48005f27-24cb-4a42-8b06-51a2254c3473","Type":"ContainerDied","Data":"2314a90358a3e21a8d2295847000c3db99c131b54dccd7490d292a28a0f3ece2"} Oct 31 12:52:33.443674 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.438869 5693 scope.go:122] "RemoveContainer" containerID="14c0ec5e0c98aaf83cfbadb497ce2a77443d5fdf97d11398cb6ae7de709e3699" Oct 31 12:52:33.443674 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.441967 5693 generic.go:356] "Generic (PLEG): container finished" podID="657aca5d-eafd-4554-80ae-a2b6d1b01987" containerID="d5736d32537b2d9a7298c0ddf7d50e1c1b21aac49e73ce28da7840fd040508aa" exitCode=0 Oct 31 12:52:33.443674 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.442028 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d48f454a-67bddb4b87-75psm" event={"ID":"657aca5d-eafd-4554-80ae-a2b6d1b01987","Type":"ContainerDied","Data":"d5736d32537b2d9a7298c0ddf7d50e1c1b21aac49e73ce28da7840fd040508aa"} Oct 31 12:52:33.443674 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.442057 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d48f454a-67bddb4b87-75psm" event={"ID":"657aca5d-eafd-4554-80ae-a2b6d1b01987","Type":"ContainerDied","Data":"b2c4f3b10c1b298b4e60ac776e3adafd83a756f79a7d58358080a1bda0c14537"} Oct 31 12:52:33.443674 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.442109 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d48f454a-67bddb4b87-75psm" Oct 31 12:52:33.456390 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.456365 5693 scope.go:122] "RemoveContainer" containerID="d5736d32537b2d9a7298c0ddf7d50e1c1b21aac49e73ce28da7840fd040508aa" Oct 31 12:52:33.472111 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.463619 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" podStartSLOduration=1.463601568 podStartE2EDuration="1.463601568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:52:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:52:33.462172079 +0000 UTC m=+1204.169758375" watchObservedRunningTime="2026-10-31 12:52:33.463601568 +0000 UTC m=+1204.171187863" Oct 31 12:52:33.472111 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.471357 5693 scope.go:122] "RemoveContainer" containerID="d5736d32537b2d9a7298c0ddf7d50e1c1b21aac49e73ce28da7840fd040508aa" Oct 31 12:52:33.472111 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:52:33.471739 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5736d32537b2d9a7298c0ddf7d50e1c1b21aac49e73ce28da7840fd040508aa\": container with ID starting with d5736d32537b2d9a7298c0ddf7d50e1c1b21aac49e73ce28da7840fd040508aa not found: ID does not exist" containerID="d5736d32537b2d9a7298c0ddf7d50e1c1b21aac49e73ce28da7840fd040508aa" Oct 31 12:52:33.472111 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.471779 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5736d32537b2d9a7298c0ddf7d50e1c1b21aac49e73ce28da7840fd040508aa"} err="failed to get container status \"d5736d32537b2d9a7298c0ddf7d50e1c1b21aac49e73ce28da7840fd040508aa\": rpc error: code = NotFound desc = could not find container \"d5736d32537b2d9a7298c0ddf7d50e1c1b21aac49e73ce28da7840fd040508aa\": container with ID starting with d5736d32537b2d9a7298c0ddf7d50e1c1b21aac49e73ce28da7840fd040508aa not found: ID does not exist" Oct 31 12:52:33.479033 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.478883 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d48f454a-67bddb4b87-75psm"] Oct 31 12:52:33.490562 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.490537 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" Oct 31 12:52:33.492129 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.492108 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d48f454a-67bddb4b87-75psm"] Oct 31 12:52:33.556765 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.556733 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48005f27-24cb-4a42-8b06-51a2254c3473-kube-api-access-qwqx6\" (UniqueName: \"kubernetes.io/projected/48005f27-24cb-4a42-8b06-51a2254c3473-kube-api-access-qwqx6\") pod \"48005f27-24cb-4a42-8b06-51a2254c3473\" (UID: \"48005f27-24cb-4a42-8b06-51a2254c3473\") " Oct 31 12:52:33.556967 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.556793 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48005f27-24cb-4a42-8b06-51a2254c3473-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48005f27-24cb-4a42-8b06-51a2254c3473-istio-envoy\") pod \"48005f27-24cb-4a42-8b06-51a2254c3473\" (UID: \"48005f27-24cb-4a42-8b06-51a2254c3473\") " Oct 31 12:52:33.556967 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.556869 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48005f27-24cb-4a42-8b06-51a2254c3473-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48005f27-24cb-4a42-8b06-51a2254c3473-credential-socket\") pod \"48005f27-24cb-4a42-8b06-51a2254c3473\" (UID: \"48005f27-24cb-4a42-8b06-51a2254c3473\") " Oct 31 12:52:33.556967 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.556908 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48005f27-24cb-4a42-8b06-51a2254c3473-istio-token\" (UniqueName: \"kubernetes.io/projected/48005f27-24cb-4a42-8b06-51a2254c3473-istio-token\") pod \"48005f27-24cb-4a42-8b06-51a2254c3473\" (UID: \"48005f27-24cb-4a42-8b06-51a2254c3473\") " Oct 31 12:52:33.557127 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.556994 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48005f27-24cb-4a42-8b06-51a2254c3473-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48005f27-24cb-4a42-8b06-51a2254c3473-istiod-ca-cert\") pod \"48005f27-24cb-4a42-8b06-51a2254c3473\" (UID: \"48005f27-24cb-4a42-8b06-51a2254c3473\") " Oct 31 12:52:33.557127 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.557036 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48005f27-24cb-4a42-8b06-51a2254c3473-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48005f27-24cb-4a42-8b06-51a2254c3473-workload-certs\") pod \"48005f27-24cb-4a42-8b06-51a2254c3473\" (UID: \"48005f27-24cb-4a42-8b06-51a2254c3473\") " Oct 31 12:52:33.557127 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.557083 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48005f27-24cb-4a42-8b06-51a2254c3473-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48005f27-24cb-4a42-8b06-51a2254c3473-istio-podinfo\") pod \"48005f27-24cb-4a42-8b06-51a2254c3473\" (UID: \"48005f27-24cb-4a42-8b06-51a2254c3473\") " Oct 31 12:52:33.557127 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.557116 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48005f27-24cb-4a42-8b06-51a2254c3473-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48005f27-24cb-4a42-8b06-51a2254c3473-workload-socket\") pod \"48005f27-24cb-4a42-8b06-51a2254c3473\" (UID: \"48005f27-24cb-4a42-8b06-51a2254c3473\") " Oct 31 12:52:33.557314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.557150 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48005f27-24cb-4a42-8b06-51a2254c3473-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48005f27-24cb-4a42-8b06-51a2254c3473-istio-data\") pod \"48005f27-24cb-4a42-8b06-51a2254c3473\" (UID: \"48005f27-24cb-4a42-8b06-51a2254c3473\") " Oct 31 12:52:33.557314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.557160 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48005f27-24cb-4a42-8b06-51a2254c3473-credential-socket" pod "48005f27-24cb-4a42-8b06-51a2254c3473" (UID: "48005f27-24cb-4a42-8b06-51a2254c3473"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:52:33.557412 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.557378 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48005f27-24cb-4a42-8b06-51a2254c3473-workload-certs" pod "48005f27-24cb-4a42-8b06-51a2254c3473" (UID: "48005f27-24cb-4a42-8b06-51a2254c3473"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:52:33.557523 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.557470 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48005f27-24cb-4a42-8b06-51a2254c3473-istiod-ca-cert" pod "48005f27-24cb-4a42-8b06-51a2254c3473" (UID: "48005f27-24cb-4a42-8b06-51a2254c3473"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:52:33.557767 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.557734 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48005f27-24cb-4a42-8b06-51a2254c3473-workload-socket" pod "48005f27-24cb-4a42-8b06-51a2254c3473" (UID: "48005f27-24cb-4a42-8b06-51a2254c3473"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:52:33.558067 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.557998 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48005f27-24cb-4a42-8b06-51a2254c3473-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:33.558067 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.558024 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48005f27-24cb-4a42-8b06-51a2254c3473-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:33.558067 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.558039 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48005f27-24cb-4a42-8b06-51a2254c3473-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:33.558067 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.558054 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48005f27-24cb-4a42-8b06-51a2254c3473-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:33.558478 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.558321 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48005f27-24cb-4a42-8b06-51a2254c3473-istio-data" pod "48005f27-24cb-4a42-8b06-51a2254c3473" (UID: "48005f27-24cb-4a42-8b06-51a2254c3473"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:52:33.559703 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.559657 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48005f27-24cb-4a42-8b06-51a2254c3473-kube-api-access-qwqx6" pod "48005f27-24cb-4a42-8b06-51a2254c3473" (UID: "48005f27-24cb-4a42-8b06-51a2254c3473"). InnerVolumeSpecName "kube-api-access-qwqx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:52:33.559703 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.559667 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48005f27-24cb-4a42-8b06-51a2254c3473-istio-podinfo" pod "48005f27-24cb-4a42-8b06-51a2254c3473" (UID: "48005f27-24cb-4a42-8b06-51a2254c3473"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:52:33.560014 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.559994 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48005f27-24cb-4a42-8b06-51a2254c3473-istio-envoy" pod "48005f27-24cb-4a42-8b06-51a2254c3473" (UID: "48005f27-24cb-4a42-8b06-51a2254c3473"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:52:33.560852 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.560836 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48005f27-24cb-4a42-8b06-51a2254c3473-istio-token" pod "48005f27-24cb-4a42-8b06-51a2254c3473" (UID: "48005f27-24cb-4a42-8b06-51a2254c3473"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:52:33.659605 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.659567 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48005f27-24cb-4a42-8b06-51a2254c3473-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:33.659605 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.659593 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48005f27-24cb-4a42-8b06-51a2254c3473-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:33.659605 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.659606 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48005f27-24cb-4a42-8b06-51a2254c3473-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:33.659605 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.659616 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwqx6\" (UniqueName: \"kubernetes.io/projected/48005f27-24cb-4a42-8b06-51a2254c3473-kube-api-access-qwqx6\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:33.660006 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.659626 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48005f27-24cb-4a42-8b06-51a2254c3473-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:33.844903 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.844821 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t"] Oct 31 12:52:33.845380 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.845360 5693 cpu_manager.go:414] "containerMap: removing container" podUID="48005f27-24cb-4a42-8b06-51a2254c3473" containerName="istio-proxy" Oct 31 12:52:33.845480 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.845380 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48005f27-24cb-4a42-8b06-51a2254c3473" containerName="istio-proxy" Oct 31 12:52:33.845480 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.845401 5693 cpu_manager.go:414] "containerMap: removing container" podUID="48005f27-24cb-4a42-8b06-51a2254c3473" containerName="istio-proxy" Oct 31 12:52:33.845480 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.845408 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48005f27-24cb-4a42-8b06-51a2254c3473" containerName="istio-proxy" Oct 31 12:52:33.845659 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.845521 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="48005f27-24cb-4a42-8b06-51a2254c3473" containerName="istio-proxy" Oct 31 12:52:33.845659 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.845536 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="48005f27-24cb-4a42-8b06-51a2254c3473" containerName="istio-proxy" Oct 31 12:52:33.849824 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.849804 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" Oct 31 12:52:33.855472 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.855446 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-622e03-openshift-default-dockercfg-br2cc\"" Oct 31 12:52:33.914805 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.914774 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" Oct 31 12:52:33.930397 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.930363 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t"] Oct 31 12:52:33.963195 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.963158 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/595d267e-2643-4570-b24c-8b03e9e34705-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x898t\" (UID: \"595d267e-2643-4570-b24c-8b03e9e34705\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" Oct 31 12:52:33.963333 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.963201 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/595d267e-2643-4570-b24c-8b03e9e34705-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x898t\" (UID: \"595d267e-2643-4570-b24c-8b03e9e34705\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" Oct 31 12:52:33.963390 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.963328 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/595d267e-2643-4570-b24c-8b03e9e34705-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x898t\" (UID: \"595d267e-2643-4570-b24c-8b03e9e34705\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" Oct 31 12:52:33.963390 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.963384 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/595d267e-2643-4570-b24c-8b03e9e34705-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x898t\" (UID: \"595d267e-2643-4570-b24c-8b03e9e34705\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" Oct 31 12:52:33.963490 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.963476 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/595d267e-2643-4570-b24c-8b03e9e34705-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x898t\" (UID: \"595d267e-2643-4570-b24c-8b03e9e34705\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" Oct 31 12:52:33.963570 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.963554 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/595d267e-2643-4570-b24c-8b03e9e34705-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x898t\" (UID: \"595d267e-2643-4570-b24c-8b03e9e34705\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" Oct 31 12:52:33.963678 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.963635 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/595d267e-2643-4570-b24c-8b03e9e34705-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x898t\" (UID: \"595d267e-2643-4570-b24c-8b03e9e34705\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" Oct 31 12:52:33.963742 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.963724 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/595d267e-2643-4570-b24c-8b03e9e34705-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x898t\" (UID: \"595d267e-2643-4570-b24c-8b03e9e34705\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" Oct 31 12:52:33.963793 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:33.963762 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjzc4\" (UniqueName: \"kubernetes.io/projected/595d267e-2643-4570-b24c-8b03e9e34705-kube-api-access-vjzc4\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x898t\" (UID: \"595d267e-2643-4570-b24c-8b03e9e34705\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" Oct 31 12:52:34.064954 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:34.064922 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/595d267e-2643-4570-b24c-8b03e9e34705-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x898t\" (UID: \"595d267e-2643-4570-b24c-8b03e9e34705\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" Oct 31 12:52:34.064954 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:34.064958 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/595d267e-2643-4570-b24c-8b03e9e34705-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x898t\" (UID: \"595d267e-2643-4570-b24c-8b03e9e34705\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" Oct 31 12:52:34.065175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:34.065009 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/595d267e-2643-4570-b24c-8b03e9e34705-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x898t\" (UID: \"595d267e-2643-4570-b24c-8b03e9e34705\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" Oct 31 12:52:34.065175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:34.065033 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/595d267e-2643-4570-b24c-8b03e9e34705-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x898t\" (UID: \"595d267e-2643-4570-b24c-8b03e9e34705\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" Oct 31 12:52:34.065175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:34.065053 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/595d267e-2643-4570-b24c-8b03e9e34705-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x898t\" (UID: \"595d267e-2643-4570-b24c-8b03e9e34705\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" Oct 31 12:52:34.065175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:34.065087 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjzc4\" (UniqueName: \"kubernetes.io/projected/595d267e-2643-4570-b24c-8b03e9e34705-kube-api-access-vjzc4\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x898t\" (UID: \"595d267e-2643-4570-b24c-8b03e9e34705\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" Oct 31 12:52:34.065175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:34.065137 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/595d267e-2643-4570-b24c-8b03e9e34705-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x898t\" (UID: \"595d267e-2643-4570-b24c-8b03e9e34705\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" Oct 31 12:52:34.065175 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:34.065168 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/595d267e-2643-4570-b24c-8b03e9e34705-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x898t\" (UID: \"595d267e-2643-4570-b24c-8b03e9e34705\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" Oct 31 12:52:34.065484 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:34.065225 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/595d267e-2643-4570-b24c-8b03e9e34705-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x898t\" (UID: \"595d267e-2643-4570-b24c-8b03e9e34705\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" Oct 31 12:52:34.065484 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:34.065415 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/595d267e-2643-4570-b24c-8b03e9e34705-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x898t\" (UID: \"595d267e-2643-4570-b24c-8b03e9e34705\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" Oct 31 12:52:34.065484 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:34.065474 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/595d267e-2643-4570-b24c-8b03e9e34705-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x898t\" (UID: \"595d267e-2643-4570-b24c-8b03e9e34705\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" Oct 31 12:52:34.065603 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:34.065587 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/595d267e-2643-4570-b24c-8b03e9e34705-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x898t\" (UID: \"595d267e-2643-4570-b24c-8b03e9e34705\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" Oct 31 12:52:34.065869 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:34.065845 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/595d267e-2643-4570-b24c-8b03e9e34705-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x898t\" (UID: \"595d267e-2643-4570-b24c-8b03e9e34705\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" Oct 31 12:52:34.066137 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:34.065847 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/595d267e-2643-4570-b24c-8b03e9e34705-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x898t\" (UID: \"595d267e-2643-4570-b24c-8b03e9e34705\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" Oct 31 12:52:34.067550 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:34.067516 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/595d267e-2643-4570-b24c-8b03e9e34705-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x898t\" (UID: \"595d267e-2643-4570-b24c-8b03e9e34705\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" Oct 31 12:52:34.067837 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:34.067821 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/595d267e-2643-4570-b24c-8b03e9e34705-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x898t\" (UID: \"595d267e-2643-4570-b24c-8b03e9e34705\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" Oct 31 12:52:34.078973 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:34.078952 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/595d267e-2643-4570-b24c-8b03e9e34705-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x898t\" (UID: \"595d267e-2643-4570-b24c-8b03e9e34705\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" Oct 31 12:52:34.082753 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:34.082732 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjzc4\" (UniqueName: \"kubernetes.io/projected/595d267e-2643-4570-b24c-8b03e9e34705-kube-api-access-vjzc4\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x898t\" (UID: \"595d267e-2643-4570-b24c-8b03e9e34705\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" Oct 31 12:52:34.164874 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:34.164843 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" Oct 31 12:52:34.449023 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:34.448943 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" Oct 31 12:52:34.449023 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:34.448955 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn" event={"ID":"48005f27-24cb-4a42-8b06-51a2254c3473","Type":"ContainerDied","Data":"86b24648ee7948ce4f24637c82b41677182e281db6fc2e0849032eb6303ec426"} Oct 31 12:52:34.449023 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:34.449020 5693 scope.go:122] "RemoveContainer" containerID="2314a90358a3e21a8d2295847000c3db99c131b54dccd7490d292a28a0f3ece2" Oct 31 12:52:34.454906 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:34.454853 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t"] Oct 31 12:52:34.460274 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:52:34.459627 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod595d267e_2643_4570_b24c_8b03e9e34705.slice/crio-52257c23a4a7ca6090cf083ad1df629ce02e895d1214e3a0e76f904cc1f4d65a WatchSource:0}: Error finding container 52257c23a4a7ca6090cf083ad1df629ce02e895d1214e3a0e76f904cc1f4d65a: Status 404 returned error can't find the container with id 52257c23a4a7ca6090cf083ad1df629ce02e895d1214e3a0e76f904cc1f4d65a Oct 31 12:52:34.462267 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:34.462215 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:52:34.462384 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:34.462276 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:52:34.462384 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:34.462305 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:52:34.482586 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:34.482557 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn"] Oct 31 12:52:34.489392 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:34.489365 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-1abc54b4-openshift-default-7dd777459c-hcbxn"] Oct 31 12:52:34.919864 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:34.919836 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" Oct 31 12:52:35.409092 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:35.409056 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48005f27-24cb-4a42-8b06-51a2254c3473" path="/var/lib/kubelet/pods/48005f27-24cb-4a42-8b06-51a2254c3473/volumes" Oct 31 12:52:35.409669 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:35.409652 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="657aca5d-eafd-4554-80ae-a2b6d1b01987" path="/var/lib/kubelet/pods/657aca5d-eafd-4554-80ae-a2b6d1b01987/volumes" Oct 31 12:52:35.457821 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:35.457788 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" event={"ID":"595d267e-2643-4570-b24c-8b03e9e34705","Type":"ContainerStarted","Data":"2cf22199be26c300893f693a59db956954a1edc575787dbbd7775f19c8765834"} Oct 31 12:52:35.458021 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:35.457838 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" event={"ID":"595d267e-2643-4570-b24c-8b03e9e34705","Type":"ContainerStarted","Data":"52257c23a4a7ca6090cf083ad1df629ce02e895d1214e3a0e76f904cc1f4d65a"} Oct 31 12:52:35.458140 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:35.458124 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" Oct 31 12:52:35.458964 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:35.458945 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" Oct 31 12:52:35.481624 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:35.481575 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" podStartSLOduration=2.481562412 podStartE2EDuration="2.481562412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:52:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:52:35.480570022 +0000 UTC m=+1206.188156318" watchObservedRunningTime="2026-10-31 12:52:35.481562412 +0000 UTC m=+1206.189148711" Oct 31 12:52:36.166059 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:36.166017 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" Oct 31 12:52:36.171379 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:36.171355 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" Oct 31 12:52:36.399788 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:36.399749 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj" Oct 31 12:52:36.464784 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:36.464705 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" Oct 31 12:52:36.465808 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:36.465787 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" Oct 31 12:52:36.728827 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:52:36.728744 5693 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee8161c_23f9_4563_a8c7_506fcf777846.slice/crio-b7fc87183e507c01b4f8ecb8cbe3c8b8100524ea93113ae1404f77df50cf17af\": RecentStats: unable to find data in memory cache]" Oct 31 12:52:43.443243 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:52:43.443207 5693 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee8161c_23f9_4563_a8c7_506fcf777846.slice/crio-b7fc87183e507c01b4f8ecb8cbe3c8b8100524ea93113ae1404f77df50cf17af\": RecentStats: unable to find data in memory cache]" Oct 31 12:52:45.398667 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:45.382862 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j"] Oct 31 12:52:45.410731 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:45.410703 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j"] Oct 31 12:52:45.411682 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:45.411023 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" Oct 31 12:52:45.423321 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:45.423304 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-7ae9e7\"/\"kube-root-ca.crt\"" Oct 31 12:52:45.423570 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:45.423546 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-7ae9e7\"/\"openshift-service-ca.crt\"" Oct 31 12:52:45.423743 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:45.423579 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7ae9e7\"/\"default-dockercfg-5qpt6\"" Oct 31 12:52:45.423743 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:45.423628 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7ae9e7\"/\"simulator-w4-7ae9e7-kserve-self-signed-certs\"" Oct 31 12:52:45.607666 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:45.607316 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5a40b79-9879-4981-ab9f-446e1750a013-model-cache\") pod \"simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j\" (UID: \"e5a40b79-9879-4981-ab9f-446e1750a013\") " pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" Oct 31 12:52:45.607666 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:45.607453 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5a40b79-9879-4981-ab9f-446e1750a013-tmp-dir\") pod \"simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j\" (UID: \"e5a40b79-9879-4981-ab9f-446e1750a013\") " pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" Oct 31 12:52:45.607666 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:45.607491 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5a40b79-9879-4981-ab9f-446e1750a013-tls-certs\") pod \"simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j\" (UID: \"e5a40b79-9879-4981-ab9f-446e1750a013\") " pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" Oct 31 12:52:45.607666 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:45.607556 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5a40b79-9879-4981-ab9f-446e1750a013-dshm\") pod \"simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j\" (UID: \"e5a40b79-9879-4981-ab9f-446e1750a013\") " pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" Oct 31 12:52:45.607666 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:45.607596 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5a40b79-9879-4981-ab9f-446e1750a013-home\") pod \"simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j\" (UID: \"e5a40b79-9879-4981-ab9f-446e1750a013\") " pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" Oct 31 12:52:45.608086 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:45.607631 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z65xd\" (UniqueName: \"kubernetes.io/projected/e5a40b79-9879-4981-ab9f-446e1750a013-kube-api-access-z65xd\") pod \"simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j\" (UID: \"e5a40b79-9879-4981-ab9f-446e1750a013\") " pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" Oct 31 12:52:45.709284 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:45.709169 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5a40b79-9879-4981-ab9f-446e1750a013-dshm\") pod \"simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j\" (UID: \"e5a40b79-9879-4981-ab9f-446e1750a013\") " pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" Oct 31 12:52:45.709284 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:45.709206 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5a40b79-9879-4981-ab9f-446e1750a013-home\") pod \"simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j\" (UID: \"e5a40b79-9879-4981-ab9f-446e1750a013\") " pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" Oct 31 12:52:45.709284 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:45.709227 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z65xd\" (UniqueName: \"kubernetes.io/projected/e5a40b79-9879-4981-ab9f-446e1750a013-kube-api-access-z65xd\") pod \"simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j\" (UID: \"e5a40b79-9879-4981-ab9f-446e1750a013\") " pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" Oct 31 12:52:45.709515 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:45.709294 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5a40b79-9879-4981-ab9f-446e1750a013-model-cache\") pod \"simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j\" (UID: \"e5a40b79-9879-4981-ab9f-446e1750a013\") " pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" Oct 31 12:52:45.709515 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:45.709347 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5a40b79-9879-4981-ab9f-446e1750a013-tmp-dir\") pod \"simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j\" (UID: \"e5a40b79-9879-4981-ab9f-446e1750a013\") " pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" Oct 31 12:52:45.709515 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:45.709366 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5a40b79-9879-4981-ab9f-446e1750a013-tls-certs\") pod \"simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j\" (UID: \"e5a40b79-9879-4981-ab9f-446e1750a013\") " pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" Oct 31 12:52:45.709800 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:45.709772 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5a40b79-9879-4981-ab9f-446e1750a013-model-cache\") pod \"simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j\" (UID: \"e5a40b79-9879-4981-ab9f-446e1750a013\") " pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" Oct 31 12:52:45.709931 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:45.709777 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5a40b79-9879-4981-ab9f-446e1750a013-home\") pod \"simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j\" (UID: \"e5a40b79-9879-4981-ab9f-446e1750a013\") " pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" Oct 31 12:52:45.709931 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:45.709860 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5a40b79-9879-4981-ab9f-446e1750a013-tmp-dir\") pod \"simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j\" (UID: \"e5a40b79-9879-4981-ab9f-446e1750a013\") " pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" Oct 31 12:52:45.711805 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:45.711780 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5a40b79-9879-4981-ab9f-446e1750a013-dshm\") pod \"simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j\" (UID: \"e5a40b79-9879-4981-ab9f-446e1750a013\") " pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" Oct 31 12:52:45.712026 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:45.712007 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5a40b79-9879-4981-ab9f-446e1750a013-tls-certs\") pod \"simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j\" (UID: \"e5a40b79-9879-4981-ab9f-446e1750a013\") " pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" Oct 31 12:52:45.724673 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:45.724612 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z65xd\" (UniqueName: \"kubernetes.io/projected/e5a40b79-9879-4981-ab9f-446e1750a013-kube-api-access-z65xd\") pod \"simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j\" (UID: \"e5a40b79-9879-4981-ab9f-446e1750a013\") " pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" Oct 31 12:52:45.726881 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:45.726861 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" Oct 31 12:52:46.029447 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:46.029422 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j"] Oct 31 12:52:46.031445 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:52:46.031422 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5a40b79_9879_4981_ab9f_446e1750a013.slice/crio-05699a910ce6a66a9709b3f09ff2f34d5c28aba16a03f7787f841bd190fee7f1 WatchSource:0}: Error finding container 05699a910ce6a66a9709b3f09ff2f34d5c28aba16a03f7787f841bd190fee7f1: Status 404 returned error can't find the container with id 05699a910ce6a66a9709b3f09ff2f34d5c28aba16a03f7787f841bd190fee7f1 Oct 31 12:52:46.356810 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:52:46.356725 5693 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee8161c_23f9_4563_a8c7_506fcf777846.slice/crio-b7fc87183e507c01b4f8ecb8cbe3c8b8100524ea93113ae1404f77df50cf17af\": RecentStats: unable to find data in memory cache]" Oct 31 12:52:46.357232 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:52:46.357207 5693 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee8161c_23f9_4563_a8c7_506fcf777846.slice/crio-b7fc87183e507c01b4f8ecb8cbe3c8b8100524ea93113ae1404f77df50cf17af\": RecentStats: unable to find data in memory cache]" Oct 31 12:52:46.525204 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:46.525166 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" event={"ID":"e5a40b79-9879-4981-ab9f-446e1750a013","Type":"ContainerStarted","Data":"a74d6f1cfb955ab0647c84c8822d7d82daee55c15cbdb3e092a9a5387c31b086"} Oct 31 12:52:46.525204 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:46.525208 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" event={"ID":"e5a40b79-9879-4981-ab9f-446e1750a013","Type":"ContainerStarted","Data":"05699a910ce6a66a9709b3f09ff2f34d5c28aba16a03f7787f841bd190fee7f1"} Oct 31 12:52:46.545115 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:46.545069 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" podStartSLOduration=1.5450555110000002 podStartE2EDuration="1.545055511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:52:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:52:46.543588976 +0000 UTC m=+1217.251175271" watchObservedRunningTime="2026-10-31 12:52:46.545055511 +0000 UTC m=+1217.252641803" Oct 31 12:52:46.727506 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:46.727477 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" Oct 31 12:52:48.707237 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:52:48.707201 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:52:51.765941 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:52:51.765909 5693 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee8161c_23f9_4563_a8c7_506fcf777846.slice/crio-b7fc87183e507c01b4f8ecb8cbe3c8b8100524ea93113ae1404f77df50cf17af\": RecentStats: unable to find data in memory cache]" Oct 31 12:52:53.454614 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:52:53.454582 5693 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee8161c_23f9_4563_a8c7_506fcf777846.slice/crio-b7fc87183e507c01b4f8ecb8cbe3c8b8100524ea93113ae1404f77df50cf17af\": RecentStats: unable to find data in memory cache]" Oct 31 12:52:55.572079 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:55.572042 5693 generic.go:356] "Generic (PLEG): container finished" podID="49a12158-0b73-4dbe-86ee-42b72e883cee" containerID="e4724401ffda94aaa10ed5c3cb501889a22d50f9b1ba1e9ba5fe6d132089d8b0" exitCode=0 Oct 31 12:52:55.572504 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:55.572119 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" event={"ID":"49a12158-0b73-4dbe-86ee-42b72e883cee","Type":"ContainerDied","Data":"e4724401ffda94aaa10ed5c3cb501889a22d50f9b1ba1e9ba5fe6d132089d8b0"} Oct 31 12:52:55.572550 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:55.572523 5693 scope.go:122] "RemoveContainer" containerID="e4724401ffda94aaa10ed5c3cb501889a22d50f9b1ba1e9ba5fe6d132089d8b0" Oct 31 12:52:55.573854 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:55.573826 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:52:55.573909 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:55.573892 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:52:55.573943 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:55.573924 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:52:55.729322 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:55.728294 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" Oct 31 12:52:55.742061 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:55.742034 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" Oct 31 12:52:56.607667 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:56.592790 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" event={"ID":"49a12158-0b73-4dbe-86ee-42b72e883cee","Type":"ContainerStarted","Data":"a7f0da2e9965bcc2978a2cd7beea63fa718a1a2cad77c5cca8961754e21f978a"} Oct 31 12:52:56.622705 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:56.622676 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" Oct 31 12:52:56.915234 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:56.915203 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" Oct 31 12:52:56.915606 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:56.915583 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 31 12:52:56.915727 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:56.915671 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" podUID="49a12158-0b73-4dbe-86ee-42b72e883cee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 31 12:52:57.915214 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:57.915183 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 31 12:52:57.915670 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:57.915248 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" podUID="49a12158-0b73-4dbe-86ee-42b72e883cee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 31 12:52:58.919715 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:58.919087 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 31 12:52:58.919715 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:58.919146 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" podUID="49a12158-0b73-4dbe-86ee-42b72e883cee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 31 12:52:59.915450 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:59.915311 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 31 12:52:59.915450 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:52:59.915409 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" podUID="49a12158-0b73-4dbe-86ee-42b72e883cee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 31 12:53:00.636984 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:00.636897 5693 generic.go:356] "Generic (PLEG): container finished" podID="595d267e-2643-4570-b24c-8b03e9e34705" containerID="2cf22199be26c300893f693a59db956954a1edc575787dbbd7775f19c8765834" exitCode=0 Oct 31 12:53:00.636984 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:00.636971 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" event={"ID":"595d267e-2643-4570-b24c-8b03e9e34705","Type":"ContainerDied","Data":"2cf22199be26c300893f693a59db956954a1edc575787dbbd7775f19c8765834"} Oct 31 12:53:00.637452 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:00.637337 5693 scope.go:122] "RemoveContainer" containerID="2cf22199be26c300893f693a59db956954a1edc575787dbbd7775f19c8765834" Oct 31 12:53:00.638497 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:00.638469 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:53:00.638627 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:00.638535 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:53:00.638627 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:00.638569 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:53:00.915190 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:00.915096 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 31 12:53:00.915190 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:00.915159 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" podUID="49a12158-0b73-4dbe-86ee-42b72e883cee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 31 12:53:01.644817 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:01.644775 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" event={"ID":"595d267e-2643-4570-b24c-8b03e9e34705","Type":"ContainerStarted","Data":"4544866727db81303852e515e9c8b14d5f745c1d44756019a86065e3890ee087"} Oct 31 12:53:01.919707 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:01.919602 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" Oct 31 12:53:01.920116 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:01.920099 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" Oct 31 12:53:01.921004 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:01.920984 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" Oct 31 12:53:02.165698 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:02.165663 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" Oct 31 12:53:02.167028 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:02.167001 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 31 12:53:02.167199 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:02.167060 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" podUID="595d267e-2643-4570-b24c-8b03e9e34705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 31 12:53:03.165426 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:03.165393 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 31 12:53:03.165830 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:03.165444 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" podUID="595d267e-2643-4570-b24c-8b03e9e34705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 31 12:53:03.530888 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:53:03.530800 5693 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee8161c_23f9_4563_a8c7_506fcf777846.slice/crio-b7fc87183e507c01b4f8ecb8cbe3c8b8100524ea93113ae1404f77df50cf17af\": RecentStats: unable to find data in memory cache]" Oct 31 12:53:04.165752 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:04.165720 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 31 12:53:04.166134 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:04.165779 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" podUID="595d267e-2643-4570-b24c-8b03e9e34705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 31 12:53:05.165552 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:05.165518 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 31 12:53:05.165762 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:05.165577 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" podUID="595d267e-2643-4570-b24c-8b03e9e34705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 31 12:53:05.614944 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:05.614858 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v"] Oct 31 12:53:05.619336 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:05.619311 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" Oct 31 12:53:05.623593 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:05.623567 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-622e03\"/\"openshift-service-ca.crt\"" Oct 31 12:53:05.624347 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:05.624322 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-622e03\"/\"kube-root-ca.crt\"" Oct 31 12:53:05.624347 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:05.624345 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-622e03\"/\"simulator-w1-622e03-kserve-self-signed-certs\"" Oct 31 12:53:05.624508 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:05.624416 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-622e03\"/\"default-dockercfg-57z5d\"" Oct 31 12:53:05.630403 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:05.630380 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v"] Oct 31 12:53:05.711938 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:05.711902 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17d56bfd-8cb1-42bf-bb11-05810c46d469-home\") pod \"simulator-w1-622e03-kserve-55b6694ffb-84l5v\" (UID: \"17d56bfd-8cb1-42bf-bb11-05810c46d469\") " pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" Oct 31 12:53:05.712130 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:05.711960 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17d56bfd-8cb1-42bf-bb11-05810c46d469-tls-certs\") pod \"simulator-w1-622e03-kserve-55b6694ffb-84l5v\" (UID: \"17d56bfd-8cb1-42bf-bb11-05810c46d469\") " pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" Oct 31 12:53:05.712130 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:05.712071 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf8xh\" (UniqueName: \"kubernetes.io/projected/17d56bfd-8cb1-42bf-bb11-05810c46d469-kube-api-access-tf8xh\") pod \"simulator-w1-622e03-kserve-55b6694ffb-84l5v\" (UID: \"17d56bfd-8cb1-42bf-bb11-05810c46d469\") " pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" Oct 31 12:53:05.712130 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:05.712114 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17d56bfd-8cb1-42bf-bb11-05810c46d469-dshm\") pod \"simulator-w1-622e03-kserve-55b6694ffb-84l5v\" (UID: \"17d56bfd-8cb1-42bf-bb11-05810c46d469\") " pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" Oct 31 12:53:05.712608 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:05.712182 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17d56bfd-8cb1-42bf-bb11-05810c46d469-tmp-dir\") pod \"simulator-w1-622e03-kserve-55b6694ffb-84l5v\" (UID: \"17d56bfd-8cb1-42bf-bb11-05810c46d469\") " pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" Oct 31 12:53:05.712608 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:05.712251 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17d56bfd-8cb1-42bf-bb11-05810c46d469-model-cache\") pod \"simulator-w1-622e03-kserve-55b6694ffb-84l5v\" (UID: \"17d56bfd-8cb1-42bf-bb11-05810c46d469\") " pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" Oct 31 12:53:05.813304 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:05.813261 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17d56bfd-8cb1-42bf-bb11-05810c46d469-home\") pod \"simulator-w1-622e03-kserve-55b6694ffb-84l5v\" (UID: \"17d56bfd-8cb1-42bf-bb11-05810c46d469\") " pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" Oct 31 12:53:05.814655 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:05.814608 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17d56bfd-8cb1-42bf-bb11-05810c46d469-tls-certs\") pod \"simulator-w1-622e03-kserve-55b6694ffb-84l5v\" (UID: \"17d56bfd-8cb1-42bf-bb11-05810c46d469\") " pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" Oct 31 12:53:05.815285 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:05.815262 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf8xh\" (UniqueName: \"kubernetes.io/projected/17d56bfd-8cb1-42bf-bb11-05810c46d469-kube-api-access-tf8xh\") pod \"simulator-w1-622e03-kserve-55b6694ffb-84l5v\" (UID: \"17d56bfd-8cb1-42bf-bb11-05810c46d469\") " pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" Oct 31 12:53:05.815483 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:05.815466 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17d56bfd-8cb1-42bf-bb11-05810c46d469-dshm\") pod \"simulator-w1-622e03-kserve-55b6694ffb-84l5v\" (UID: \"17d56bfd-8cb1-42bf-bb11-05810c46d469\") " pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" Oct 31 12:53:05.815655 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:05.815618 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17d56bfd-8cb1-42bf-bb11-05810c46d469-tmp-dir\") pod \"simulator-w1-622e03-kserve-55b6694ffb-84l5v\" (UID: \"17d56bfd-8cb1-42bf-bb11-05810c46d469\") " pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" Oct 31 12:53:05.815836 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:05.815806 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17d56bfd-8cb1-42bf-bb11-05810c46d469-model-cache\") pod \"simulator-w1-622e03-kserve-55b6694ffb-84l5v\" (UID: \"17d56bfd-8cb1-42bf-bb11-05810c46d469\") " pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" Oct 31 12:53:05.816337 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:05.816317 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17d56bfd-8cb1-42bf-bb11-05810c46d469-model-cache\") pod \"simulator-w1-622e03-kserve-55b6694ffb-84l5v\" (UID: \"17d56bfd-8cb1-42bf-bb11-05810c46d469\") " pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" Oct 31 12:53:05.816468 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:05.814115 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17d56bfd-8cb1-42bf-bb11-05810c46d469-home\") pod \"simulator-w1-622e03-kserve-55b6694ffb-84l5v\" (UID: \"17d56bfd-8cb1-42bf-bb11-05810c46d469\") " pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" Oct 31 12:53:05.821657 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:05.817982 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17d56bfd-8cb1-42bf-bb11-05810c46d469-tmp-dir\") pod \"simulator-w1-622e03-kserve-55b6694ffb-84l5v\" (UID: \"17d56bfd-8cb1-42bf-bb11-05810c46d469\") " pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" Oct 31 12:53:05.822146 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:05.822129 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17d56bfd-8cb1-42bf-bb11-05810c46d469-tls-certs\") pod \"simulator-w1-622e03-kserve-55b6694ffb-84l5v\" (UID: \"17d56bfd-8cb1-42bf-bb11-05810c46d469\") " pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" Oct 31 12:53:05.824570 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:05.824551 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17d56bfd-8cb1-42bf-bb11-05810c46d469-dshm\") pod \"simulator-w1-622e03-kserve-55b6694ffb-84l5v\" (UID: \"17d56bfd-8cb1-42bf-bb11-05810c46d469\") " pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" Oct 31 12:53:05.830593 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:05.830567 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf8xh\" (UniqueName: \"kubernetes.io/projected/17d56bfd-8cb1-42bf-bb11-05810c46d469-kube-api-access-tf8xh\") pod \"simulator-w1-622e03-kserve-55b6694ffb-84l5v\" (UID: \"17d56bfd-8cb1-42bf-bb11-05810c46d469\") " pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" Oct 31 12:53:05.933318 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:05.933282 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" Oct 31 12:53:06.165202 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:06.165168 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 31 12:53:06.165375 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:06.165218 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" podUID="595d267e-2643-4570-b24c-8b03e9e34705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 31 12:53:06.235566 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:06.235545 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v"] Oct 31 12:53:06.237589 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:53:06.237565 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17d56bfd_8cb1_42bf_bb11_05810c46d469.slice/crio-ee19eecabf6fabe7c5cf1ca0c320eb2a462208460a4464cdb3fb489cc8adfa01 WatchSource:0}: Error finding container ee19eecabf6fabe7c5cf1ca0c320eb2a462208460a4464cdb3fb489cc8adfa01: Status 404 returned error can't find the container with id ee19eecabf6fabe7c5cf1ca0c320eb2a462208460a4464cdb3fb489cc8adfa01 Oct 31 12:53:06.465574 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:06.465527 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" Oct 31 12:53:06.671782 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:06.671749 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" event={"ID":"17d56bfd-8cb1-42bf-bb11-05810c46d469","Type":"ContainerStarted","Data":"c92189754374164d4d3e5651ed1a6eef96c07caf0f02c06c73ec6b035beeb4fa"} Oct 31 12:53:06.672198 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:06.671791 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" event={"ID":"17d56bfd-8cb1-42bf-bb11-05810c46d469","Type":"ContainerStarted","Data":"ee19eecabf6fabe7c5cf1ca0c320eb2a462208460a4464cdb3fb489cc8adfa01"} Oct 31 12:53:06.702776 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:06.702725 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" podStartSLOduration=1.702711117 podStartE2EDuration="1.702711117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:06.700349174 +0000 UTC m=+1237.407935495" watchObservedRunningTime="2026-10-31 12:53:06.702711117 +0000 UTC m=+1237.410297411" Oct 31 12:53:06.719095 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:53:06.719069 5693 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee8161c_23f9_4563_a8c7_506fcf777846.slice/crio-b7fc87183e507c01b4f8ecb8cbe3c8b8100524ea93113ae1404f77df50cf17af\": RecentStats: unable to find data in memory cache]" Oct 31 12:53:06.933798 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:06.933706 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" Oct 31 12:53:07.173346 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:07.173312 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" Oct 31 12:53:07.174571 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:07.174547 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" Oct 31 12:53:12.514546 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:12.514512 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44"] Oct 31 12:53:12.518387 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:12.518366 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" Oct 31 12:53:12.522797 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:12.522773 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7ae9e7\"/\"premium-w4-7ae9e7-kserve-self-signed-certs\"" Oct 31 12:53:12.527969 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:12.527949 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44"] Oct 31 12:53:12.693941 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:12.693897 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fb66a94-c399-43fb-945c-610ca63f0d76-tmp-dir\") pod \"premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44\" (UID: \"8fb66a94-c399-43fb-945c-610ca63f0d76\") " pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" Oct 31 12:53:12.694131 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:12.694012 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx2dm\" (UniqueName: \"kubernetes.io/projected/8fb66a94-c399-43fb-945c-610ca63f0d76-kube-api-access-wx2dm\") pod \"premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44\" (UID: \"8fb66a94-c399-43fb-945c-610ca63f0d76\") " pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" Oct 31 12:53:12.694131 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:12.694090 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fb66a94-c399-43fb-945c-610ca63f0d76-dshm\") pod \"premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44\" (UID: \"8fb66a94-c399-43fb-945c-610ca63f0d76\") " pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" Oct 31 12:53:12.694216 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:12.694152 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fb66a94-c399-43fb-945c-610ca63f0d76-home\") pod \"premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44\" (UID: \"8fb66a94-c399-43fb-945c-610ca63f0d76\") " pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" Oct 31 12:53:12.694373 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:12.694354 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fb66a94-c399-43fb-945c-610ca63f0d76-tls-certs\") pod \"premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44\" (UID: \"8fb66a94-c399-43fb-945c-610ca63f0d76\") " pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" Oct 31 12:53:12.694432 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:12.694389 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fb66a94-c399-43fb-945c-610ca63f0d76-model-cache\") pod \"premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44\" (UID: \"8fb66a94-c399-43fb-945c-610ca63f0d76\") " pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" Oct 31 12:53:12.795365 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:12.795215 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fb66a94-c399-43fb-945c-610ca63f0d76-tls-certs\") pod \"premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44\" (UID: \"8fb66a94-c399-43fb-945c-610ca63f0d76\") " pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" Oct 31 12:53:12.795365 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:12.795262 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fb66a94-c399-43fb-945c-610ca63f0d76-model-cache\") pod \"premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44\" (UID: \"8fb66a94-c399-43fb-945c-610ca63f0d76\") " pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" Oct 31 12:53:12.795365 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:12.795343 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fb66a94-c399-43fb-945c-610ca63f0d76-tmp-dir\") pod \"premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44\" (UID: \"8fb66a94-c399-43fb-945c-610ca63f0d76\") " pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" Oct 31 12:53:12.795708 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:12.795371 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx2dm\" (UniqueName: \"kubernetes.io/projected/8fb66a94-c399-43fb-945c-610ca63f0d76-kube-api-access-wx2dm\") pod \"premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44\" (UID: \"8fb66a94-c399-43fb-945c-610ca63f0d76\") " pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" Oct 31 12:53:12.795708 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:12.795429 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fb66a94-c399-43fb-945c-610ca63f0d76-dshm\") pod \"premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44\" (UID: \"8fb66a94-c399-43fb-945c-610ca63f0d76\") " pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" Oct 31 12:53:12.795708 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:12.795462 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fb66a94-c399-43fb-945c-610ca63f0d76-home\") pod \"premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44\" (UID: \"8fb66a94-c399-43fb-945c-610ca63f0d76\") " pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" Oct 31 12:53:12.795883 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:12.795761 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fb66a94-c399-43fb-945c-610ca63f0d76-model-cache\") pod \"premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44\" (UID: \"8fb66a94-c399-43fb-945c-610ca63f0d76\") " pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" Oct 31 12:53:12.795883 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:12.795853 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fb66a94-c399-43fb-945c-610ca63f0d76-home\") pod \"premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44\" (UID: \"8fb66a94-c399-43fb-945c-610ca63f0d76\") " pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" Oct 31 12:53:12.795953 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:12.795932 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fb66a94-c399-43fb-945c-610ca63f0d76-tmp-dir\") pod \"premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44\" (UID: \"8fb66a94-c399-43fb-945c-610ca63f0d76\") " pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" Oct 31 12:53:12.797790 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:12.797766 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fb66a94-c399-43fb-945c-610ca63f0d76-dshm\") pod \"premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44\" (UID: \"8fb66a94-c399-43fb-945c-610ca63f0d76\") " pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" Oct 31 12:53:12.798079 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:12.798046 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fb66a94-c399-43fb-945c-610ca63f0d76-tls-certs\") pod \"premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44\" (UID: \"8fb66a94-c399-43fb-945c-610ca63f0d76\") " pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" Oct 31 12:53:12.808623 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:12.808591 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx2dm\" (UniqueName: \"kubernetes.io/projected/8fb66a94-c399-43fb-945c-610ca63f0d76-kube-api-access-wx2dm\") pod \"premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44\" (UID: \"8fb66a94-c399-43fb-945c-610ca63f0d76\") " pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" Oct 31 12:53:12.831632 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:12.831609 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" Oct 31 12:53:13.107888 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:13.107862 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44"] Oct 31 12:53:13.109605 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:53:13.109579 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8fb66a94_c399_43fb_945c_610ca63f0d76.slice/crio-aa94e953b4e213dde0da4a3d2142ad42e95be062a966ccee6cdb6863d97511e4 WatchSource:0}: Error finding container aa94e953b4e213dde0da4a3d2142ad42e95be062a966ccee6cdb6863d97511e4: Status 404 returned error can't find the container with id aa94e953b4e213dde0da4a3d2142ad42e95be062a966ccee6cdb6863d97511e4 Oct 31 12:53:13.597728 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:53:13.597682 5693 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee8161c_23f9_4563_a8c7_506fcf777846.slice/crio-b7fc87183e507c01b4f8ecb8cbe3c8b8100524ea93113ae1404f77df50cf17af\": RecentStats: unable to find data in memory cache]" Oct 31 12:53:13.711737 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:13.711691 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" event={"ID":"8fb66a94-c399-43fb-945c-610ca63f0d76","Type":"ContainerStarted","Data":"83b955b87f7980f30ebb11e76b8cee966e5f12bb55771e2c2198ccad8f80b64d"} Oct 31 12:53:13.711918 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:13.711746 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" event={"ID":"8fb66a94-c399-43fb-945c-610ca63f0d76","Type":"ContainerStarted","Data":"aa94e953b4e213dde0da4a3d2142ad42e95be062a966ccee6cdb6863d97511e4"} Oct 31 12:53:13.732986 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:13.732935 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" podStartSLOduration=1.732921677 podStartE2EDuration="1.732921677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:53:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:13.730856066 +0000 UTC m=+1244.438442375" watchObservedRunningTime="2026-10-31 12:53:13.732921677 +0000 UTC m=+1244.440507988" Oct 31 12:53:13.832109 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:13.832080 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" Oct 31 12:53:15.934143 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:15.934112 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" Oct 31 12:53:15.946705 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:15.946678 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" Oct 31 12:53:16.740804 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:16.740772 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" Oct 31 12:53:18.687813 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:53:18.687774 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:53:18.928730 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:18.928697 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl"] Oct 31 12:53:18.932842 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:18.932818 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" Oct 31 12:53:18.941550 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:18.941523 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-107955\"/\"default-dockercfg-gxcbm\"" Oct 31 12:53:18.941711 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:18.941588 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-107955\"/\"openshift-service-ca.crt\"" Oct 31 12:53:18.941711 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:18.941680 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-107955\"/\"kube-root-ca.crt\"" Oct 31 12:53:18.941905 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:18.941764 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-107955\"/\"premium-w0-107955-kserve-self-signed-certs\"" Oct 31 12:53:18.943249 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:18.943197 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl"] Oct 31 12:53:19.063676 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:19.063623 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a2abd3f-fb4f-4990-9a32-5f945a588068-home\") pod \"premium-w0-107955-kserve-69b87d495-dckzl\" (UID: \"1a2abd3f-fb4f-4990-9a32-5f945a588068\") " pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" Oct 31 12:53:19.065226 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:19.063786 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a2abd3f-fb4f-4990-9a32-5f945a588068-dshm\") pod \"premium-w0-107955-kserve-69b87d495-dckzl\" (UID: \"1a2abd3f-fb4f-4990-9a32-5f945a588068\") " pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" Oct 31 12:53:19.065226 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:19.063827 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a2abd3f-fb4f-4990-9a32-5f945a588068-tmp-dir\") pod \"premium-w0-107955-kserve-69b87d495-dckzl\" (UID: \"1a2abd3f-fb4f-4990-9a32-5f945a588068\") " pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" Oct 31 12:53:19.065226 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:19.063879 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a2abd3f-fb4f-4990-9a32-5f945a588068-model-cache\") pod \"premium-w0-107955-kserve-69b87d495-dckzl\" (UID: \"1a2abd3f-fb4f-4990-9a32-5f945a588068\") " pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" Oct 31 12:53:19.065226 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:19.063903 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlpkl\" (UniqueName: \"kubernetes.io/projected/1a2abd3f-fb4f-4990-9a32-5f945a588068-kube-api-access-tlpkl\") pod \"premium-w0-107955-kserve-69b87d495-dckzl\" (UID: \"1a2abd3f-fb4f-4990-9a32-5f945a588068\") " pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" Oct 31 12:53:19.065226 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:19.064037 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a2abd3f-fb4f-4990-9a32-5f945a588068-tls-certs\") pod \"premium-w0-107955-kserve-69b87d495-dckzl\" (UID: \"1a2abd3f-fb4f-4990-9a32-5f945a588068\") " pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" Oct 31 12:53:19.166304 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:19.166272 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a2abd3f-fb4f-4990-9a32-5f945a588068-dshm\") pod \"premium-w0-107955-kserve-69b87d495-dckzl\" (UID: \"1a2abd3f-fb4f-4990-9a32-5f945a588068\") " pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" Oct 31 12:53:19.166499 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:19.166328 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a2abd3f-fb4f-4990-9a32-5f945a588068-tmp-dir\") pod \"premium-w0-107955-kserve-69b87d495-dckzl\" (UID: \"1a2abd3f-fb4f-4990-9a32-5f945a588068\") " pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" Oct 31 12:53:19.166499 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:19.166379 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a2abd3f-fb4f-4990-9a32-5f945a588068-model-cache\") pod \"premium-w0-107955-kserve-69b87d495-dckzl\" (UID: \"1a2abd3f-fb4f-4990-9a32-5f945a588068\") " pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" Oct 31 12:53:19.166499 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:19.166406 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlpkl\" (UniqueName: \"kubernetes.io/projected/1a2abd3f-fb4f-4990-9a32-5f945a588068-kube-api-access-tlpkl\") pod \"premium-w0-107955-kserve-69b87d495-dckzl\" (UID: \"1a2abd3f-fb4f-4990-9a32-5f945a588068\") " pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" Oct 31 12:53:19.166499 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:19.166472 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a2abd3f-fb4f-4990-9a32-5f945a588068-tls-certs\") pod \"premium-w0-107955-kserve-69b87d495-dckzl\" (UID: \"1a2abd3f-fb4f-4990-9a32-5f945a588068\") " pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" Oct 31 12:53:19.166737 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:19.166516 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a2abd3f-fb4f-4990-9a32-5f945a588068-home\") pod \"premium-w0-107955-kserve-69b87d495-dckzl\" (UID: \"1a2abd3f-fb4f-4990-9a32-5f945a588068\") " pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" Oct 31 12:53:19.166801 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:19.166751 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a2abd3f-fb4f-4990-9a32-5f945a588068-tmp-dir\") pod \"premium-w0-107955-kserve-69b87d495-dckzl\" (UID: \"1a2abd3f-fb4f-4990-9a32-5f945a588068\") " pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" Oct 31 12:53:19.166865 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:19.166844 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a2abd3f-fb4f-4990-9a32-5f945a588068-model-cache\") pod \"premium-w0-107955-kserve-69b87d495-dckzl\" (UID: \"1a2abd3f-fb4f-4990-9a32-5f945a588068\") " pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" Oct 31 12:53:19.166924 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:19.166850 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a2abd3f-fb4f-4990-9a32-5f945a588068-home\") pod \"premium-w0-107955-kserve-69b87d495-dckzl\" (UID: \"1a2abd3f-fb4f-4990-9a32-5f945a588068\") " pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" Oct 31 12:53:19.170160 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:19.170113 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a2abd3f-fb4f-4990-9a32-5f945a588068-dshm\") pod \"premium-w0-107955-kserve-69b87d495-dckzl\" (UID: \"1a2abd3f-fb4f-4990-9a32-5f945a588068\") " pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" Oct 31 12:53:19.172209 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:19.172177 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a2abd3f-fb4f-4990-9a32-5f945a588068-tls-certs\") pod \"premium-w0-107955-kserve-69b87d495-dckzl\" (UID: \"1a2abd3f-fb4f-4990-9a32-5f945a588068\") " pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" Oct 31 12:53:19.181516 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:19.181485 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlpkl\" (UniqueName: \"kubernetes.io/projected/1a2abd3f-fb4f-4990-9a32-5f945a588068-kube-api-access-tlpkl\") pod \"premium-w0-107955-kserve-69b87d495-dckzl\" (UID: \"1a2abd3f-fb4f-4990-9a32-5f945a588068\") " pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" Oct 31 12:53:19.249306 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:19.249216 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" Oct 31 12:53:19.530662 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:19.530619 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl"] Oct 31 12:53:19.532566 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:53:19.532539 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a2abd3f_fb4f_4990_9a32_5f945a588068.slice/crio-73a8f41938d4846d3b1aad1cd1f57425280b4c04fffbd073539bb4ca986dfa27 WatchSource:0}: Error finding container 73a8f41938d4846d3b1aad1cd1f57425280b4c04fffbd073539bb4ca986dfa27: Status 404 returned error can't find the container with id 73a8f41938d4846d3b1aad1cd1f57425280b4c04fffbd073539bb4ca986dfa27 Oct 31 12:53:19.754659 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:19.754613 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" event={"ID":"1a2abd3f-fb4f-4990-9a32-5f945a588068","Type":"ContainerStarted","Data":"4db86fb4b1d8cd02ab6169d455af04822523b9903ff6ac18dfe7896623d5a0db"} Oct 31 12:53:19.755162 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:19.754690 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" event={"ID":"1a2abd3f-fb4f-4990-9a32-5f945a588068","Type":"ContainerStarted","Data":"73a8f41938d4846d3b1aad1cd1f57425280b4c04fffbd073539bb4ca986dfa27"} Oct 31 12:53:19.782594 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:19.782487 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" podStartSLOduration=1.782472279 podStartE2EDuration="1.782472279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:53:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:19.781180184 +0000 UTC m=+1250.488766481" watchObservedRunningTime="2026-10-31 12:53:19.782472279 +0000 UTC m=+1250.490058573" Oct 31 12:53:20.250434 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:20.250382 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" Oct 31 12:53:21.774041 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:53:21.774006 5693 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee8161c_23f9_4563_a8c7_506fcf777846.slice/crio-b7fc87183e507c01b4f8ecb8cbe3c8b8100524ea93113ae1404f77df50cf17af\": RecentStats: unable to find data in memory cache]" Oct 31 12:53:22.832574 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:22.832537 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" Oct 31 12:53:22.844944 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:22.844920 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" Oct 31 12:53:23.608884 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:53:23.608852 5693 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee8161c_23f9_4563_a8c7_506fcf777846.slice/crio-b7fc87183e507c01b4f8ecb8cbe3c8b8100524ea93113ae1404f77df50cf17af\": RecentStats: unable to find data in memory cache]" Oct 31 12:53:23.792106 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:23.792080 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" Oct 31 12:53:29.250124 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:29.250087 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" Oct 31 12:53:29.263026 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:29.263003 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" Oct 31 12:53:29.830613 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:53:29.830587 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" Oct 31 12:53:48.690026 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:53:48.689938 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:54:12.902244 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:12.902209 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7f85c5f9-6tx9r"] Oct 31 12:54:12.907742 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:12.907407 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7f85c5f9-6tx9r" Oct 31 12:54:12.920237 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:12.920212 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 31 12:54:12.921328 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:12.921307 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 31 12:54:12.921491 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:12.921332 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-e8c983619803\"" Oct 31 12:54:12.921491 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:12.921333 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 31 12:54:12.921491 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:12.921332 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 31 12:54:12.921720 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:12.921382 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-vms8t\"" Oct 31 12:54:12.921720 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:12.921405 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-c59bbce24cdf\"" Oct 31 12:54:12.923894 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:12.923873 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7f85c5f9-6tx9r"] Oct 31 12:54:13.035631 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:13.035595 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96086778-e6af-42e9-89eb-f2537e9025b7-plugins-config-volume\") pod \"payload-processing-external-model-7f85c5f9-6tx9r\" (UID: \"96086778-e6af-42e9-89eb-f2537e9025b7\") " pod="models-as-a-service/payload-processing-external-model-7f85c5f9-6tx9r" Oct 31 12:54:13.035833 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:13.035734 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85pgw\" (UniqueName: \"kubernetes.io/projected/96086778-e6af-42e9-89eb-f2537e9025b7-kube-api-access-85pgw\") pod \"payload-processing-external-model-7f85c5f9-6tx9r\" (UID: \"96086778-e6af-42e9-89eb-f2537e9025b7\") " pod="models-as-a-service/payload-processing-external-model-7f85c5f9-6tx9r" Oct 31 12:54:13.035833 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:13.035778 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/96086778-e6af-42e9-89eb-f2537e9025b7-provider-credentials\") pod \"payload-processing-external-model-7f85c5f9-6tx9r\" (UID: \"96086778-e6af-42e9-89eb-f2537e9025b7\") " pod="models-as-a-service/payload-processing-external-model-7f85c5f9-6tx9r" Oct 31 12:54:13.035919 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:13.035870 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/96086778-e6af-42e9-89eb-f2537e9025b7-routing-overlay-volume\") pod \"payload-processing-external-model-7f85c5f9-6tx9r\" (UID: \"96086778-e6af-42e9-89eb-f2537e9025b7\") " pod="models-as-a-service/payload-processing-external-model-7f85c5f9-6tx9r" Oct 31 12:54:13.137482 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:13.137442 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/96086778-e6af-42e9-89eb-f2537e9025b7-routing-overlay-volume\") pod \"payload-processing-external-model-7f85c5f9-6tx9r\" (UID: \"96086778-e6af-42e9-89eb-f2537e9025b7\") " pod="models-as-a-service/payload-processing-external-model-7f85c5f9-6tx9r" Oct 31 12:54:13.137713 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:13.137543 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96086778-e6af-42e9-89eb-f2537e9025b7-plugins-config-volume\") pod \"payload-processing-external-model-7f85c5f9-6tx9r\" (UID: \"96086778-e6af-42e9-89eb-f2537e9025b7\") " pod="models-as-a-service/payload-processing-external-model-7f85c5f9-6tx9r" Oct 31 12:54:13.137713 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:13.137588 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85pgw\" (UniqueName: \"kubernetes.io/projected/96086778-e6af-42e9-89eb-f2537e9025b7-kube-api-access-85pgw\") pod \"payload-processing-external-model-7f85c5f9-6tx9r\" (UID: \"96086778-e6af-42e9-89eb-f2537e9025b7\") " pod="models-as-a-service/payload-processing-external-model-7f85c5f9-6tx9r" Oct 31 12:54:13.137713 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:13.137620 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/96086778-e6af-42e9-89eb-f2537e9025b7-provider-credentials\") pod \"payload-processing-external-model-7f85c5f9-6tx9r\" (UID: \"96086778-e6af-42e9-89eb-f2537e9025b7\") " pod="models-as-a-service/payload-processing-external-model-7f85c5f9-6tx9r" Oct 31 12:54:13.138329 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:13.138300 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/96086778-e6af-42e9-89eb-f2537e9025b7-routing-overlay-volume\") pod \"payload-processing-external-model-7f85c5f9-6tx9r\" (UID: \"96086778-e6af-42e9-89eb-f2537e9025b7\") " pod="models-as-a-service/payload-processing-external-model-7f85c5f9-6tx9r" Oct 31 12:54:13.138480 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:13.138455 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96086778-e6af-42e9-89eb-f2537e9025b7-plugins-config-volume\") pod \"payload-processing-external-model-7f85c5f9-6tx9r\" (UID: \"96086778-e6af-42e9-89eb-f2537e9025b7\") " pod="models-as-a-service/payload-processing-external-model-7f85c5f9-6tx9r" Oct 31 12:54:13.140478 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:13.140454 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/96086778-e6af-42e9-89eb-f2537e9025b7-provider-credentials\") pod \"payload-processing-external-model-7f85c5f9-6tx9r\" (UID: \"96086778-e6af-42e9-89eb-f2537e9025b7\") " pod="models-as-a-service/payload-processing-external-model-7f85c5f9-6tx9r" Oct 31 12:54:13.152176 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:13.152131 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85pgw\" (UniqueName: \"kubernetes.io/projected/96086778-e6af-42e9-89eb-f2537e9025b7-kube-api-access-85pgw\") pod \"payload-processing-external-model-7f85c5f9-6tx9r\" (UID: \"96086778-e6af-42e9-89eb-f2537e9025b7\") " pod="models-as-a-service/payload-processing-external-model-7f85c5f9-6tx9r" Oct 31 12:54:13.220094 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:13.219987 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7f85c5f9-6tx9r" Oct 31 12:54:13.520736 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:13.518898 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7f85c5f9-6tx9r"] Oct 31 12:54:13.525124 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:54:13.524256 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96086778_e6af_42e9_89eb_f2537e9025b7.slice/crio-bfa86ad05fd2b02c7dbbdfd7ba6932c75e22579872ffa03918fde0cf421c6e73 WatchSource:0}: Error finding container bfa86ad05fd2b02c7dbbdfd7ba6932c75e22579872ffa03918fde0cf421c6e73: Status 404 returned error can't find the container with id bfa86ad05fd2b02c7dbbdfd7ba6932c75e22579872ffa03918fde0cf421c6e73 Oct 31 12:54:14.166471 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:14.163912 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7f85c5f9-6tx9r" event={"ID":"96086778-e6af-42e9-89eb-f2537e9025b7","Type":"ContainerStarted","Data":"c4845160ef3b17cb4e26e2cb5d4ac39b8e135633a1ec7dcb7ea8a8664e1f72b5"} Oct 31 12:54:14.166471 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:14.164019 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7f85c5f9-6tx9r" event={"ID":"96086778-e6af-42e9-89eb-f2537e9025b7","Type":"ContainerStarted","Data":"bfa86ad05fd2b02c7dbbdfd7ba6932c75e22579872ffa03918fde0cf421c6e73"} Oct 31 12:54:14.166471 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:14.164078 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7f85c5f9-6tx9r" Oct 31 12:54:14.226278 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:14.226211 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7f85c5f9-6tx9r" podStartSLOduration=2.226191458 podStartE2EDuration="2.226191458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:54:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:54:14.223080648 +0000 UTC m=+1304.930666944" watchObservedRunningTime="2026-10-31 12:54:14.226191458 +0000 UTC m=+1304.933777754" Oct 31 12:54:18.687124 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:54:18.687082 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:54:25.173872 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:25.173841 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7f85c5f9-6tx9r" Oct 31 12:54:28.577693 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:28.577628 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm"] Oct 31 12:54:28.581494 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:28.581478 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" Oct 31 12:54:28.586674 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:28.586615 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-47bcd9-openshift-default-dockercfg-dwvfp\"" Oct 31 12:54:28.601687 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:28.594743 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm"] Oct 31 12:54:28.637116 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:28.637085 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3797b53-75e3-4584-8211-a06b15e03526-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm\" (UID: \"a3797b53-75e3-4584-8211-a06b15e03526\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" Oct 31 12:54:28.637116 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:28.637118 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3797b53-75e3-4584-8211-a06b15e03526-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm\" (UID: \"a3797b53-75e3-4584-8211-a06b15e03526\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" Oct 31 12:54:28.637376 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:28.637162 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3797b53-75e3-4584-8211-a06b15e03526-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm\" (UID: \"a3797b53-75e3-4584-8211-a06b15e03526\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" Oct 31 12:54:28.637376 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:28.637181 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3797b53-75e3-4584-8211-a06b15e03526-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm\" (UID: \"a3797b53-75e3-4584-8211-a06b15e03526\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" Oct 31 12:54:28.637376 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:28.637205 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3797b53-75e3-4584-8211-a06b15e03526-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm\" (UID: \"a3797b53-75e3-4584-8211-a06b15e03526\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" Oct 31 12:54:28.637376 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:28.637235 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3797b53-75e3-4584-8211-a06b15e03526-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm\" (UID: \"a3797b53-75e3-4584-8211-a06b15e03526\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" Oct 31 12:54:28.637376 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:28.637253 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfkrh\" (UniqueName: \"kubernetes.io/projected/a3797b53-75e3-4584-8211-a06b15e03526-kube-api-access-cfkrh\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm\" (UID: \"a3797b53-75e3-4584-8211-a06b15e03526\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" Oct 31 12:54:28.637376 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:28.637274 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3797b53-75e3-4584-8211-a06b15e03526-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm\" (UID: \"a3797b53-75e3-4584-8211-a06b15e03526\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" Oct 31 12:54:28.637376 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:28.637309 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3797b53-75e3-4584-8211-a06b15e03526-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm\" (UID: \"a3797b53-75e3-4584-8211-a06b15e03526\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" Oct 31 12:54:28.738173 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:28.738135 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3797b53-75e3-4584-8211-a06b15e03526-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm\" (UID: \"a3797b53-75e3-4584-8211-a06b15e03526\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" Oct 31 12:54:28.738383 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:28.738179 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3797b53-75e3-4584-8211-a06b15e03526-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm\" (UID: \"a3797b53-75e3-4584-8211-a06b15e03526\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" Oct 31 12:54:28.738383 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:28.738231 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3797b53-75e3-4584-8211-a06b15e03526-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm\" (UID: \"a3797b53-75e3-4584-8211-a06b15e03526\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" Oct 31 12:54:28.738383 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:28.738260 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3797b53-75e3-4584-8211-a06b15e03526-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm\" (UID: \"a3797b53-75e3-4584-8211-a06b15e03526\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" Oct 31 12:54:28.738695 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:28.738654 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3797b53-75e3-4584-8211-a06b15e03526-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm\" (UID: \"a3797b53-75e3-4584-8211-a06b15e03526\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" Oct 31 12:54:28.738893 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:28.738871 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3797b53-75e3-4584-8211-a06b15e03526-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm\" (UID: \"a3797b53-75e3-4584-8211-a06b15e03526\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" Oct 31 12:54:28.738981 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:28.738923 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3797b53-75e3-4584-8211-a06b15e03526-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm\" (UID: \"a3797b53-75e3-4584-8211-a06b15e03526\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" Oct 31 12:54:28.738981 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:28.738950 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfkrh\" (UniqueName: \"kubernetes.io/projected/a3797b53-75e3-4584-8211-a06b15e03526-kube-api-access-cfkrh\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm\" (UID: \"a3797b53-75e3-4584-8211-a06b15e03526\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" Oct 31 12:54:28.739153 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:28.738990 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3797b53-75e3-4584-8211-a06b15e03526-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm\" (UID: \"a3797b53-75e3-4584-8211-a06b15e03526\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" Oct 31 12:54:28.739153 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:28.739039 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3797b53-75e3-4584-8211-a06b15e03526-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm\" (UID: \"a3797b53-75e3-4584-8211-a06b15e03526\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" Oct 31 12:54:28.739153 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:28.739049 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3797b53-75e3-4584-8211-a06b15e03526-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm\" (UID: \"a3797b53-75e3-4584-8211-a06b15e03526\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" Oct 31 12:54:28.739153 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:28.739097 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3797b53-75e3-4584-8211-a06b15e03526-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm\" (UID: \"a3797b53-75e3-4584-8211-a06b15e03526\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" Oct 31 12:54:28.739356 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:28.739325 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3797b53-75e3-4584-8211-a06b15e03526-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm\" (UID: \"a3797b53-75e3-4584-8211-a06b15e03526\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" Oct 31 12:54:28.739576 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:28.739538 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3797b53-75e3-4584-8211-a06b15e03526-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm\" (UID: \"a3797b53-75e3-4584-8211-a06b15e03526\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" Oct 31 12:54:28.744297 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:28.742942 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3797b53-75e3-4584-8211-a06b15e03526-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm\" (UID: \"a3797b53-75e3-4584-8211-a06b15e03526\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" Oct 31 12:54:28.745163 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:28.745141 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3797b53-75e3-4584-8211-a06b15e03526-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm\" (UID: \"a3797b53-75e3-4584-8211-a06b15e03526\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" Oct 31 12:54:28.751859 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:28.751837 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfkrh\" (UniqueName: \"kubernetes.io/projected/a3797b53-75e3-4584-8211-a06b15e03526-kube-api-access-cfkrh\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm\" (UID: \"a3797b53-75e3-4584-8211-a06b15e03526\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" Oct 31 12:54:28.751974 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:28.751894 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3797b53-75e3-4584-8211-a06b15e03526-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm\" (UID: \"a3797b53-75e3-4584-8211-a06b15e03526\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" Oct 31 12:54:28.932208 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:28.932167 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" Oct 31 12:54:29.277434 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:29.277242 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm"] Oct 31 12:54:29.297279 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:29.297220 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:54:29.297512 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:29.297489 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:54:29.297697 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:29.297665 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:54:29.300237 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:29.300186 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" event={"ID":"a3797b53-75e3-4584-8211-a06b15e03526","Type":"ContainerStarted","Data":"5ea64016e4ecd34c026284dd7fcc6ffa4a85f6bd55a6de801af3e7e8b8feb0b6"} Oct 31 12:54:30.369666 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:30.368862 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" event={"ID":"a3797b53-75e3-4584-8211-a06b15e03526","Type":"ContainerStarted","Data":"5a5c583549e48c36d054da710bdec34969d42f08049fde50dfe54c6363a55ea4"} Oct 31 12:54:30.408668 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:30.407752 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" podStartSLOduration=2.407718206 podStartE2EDuration="2.407718206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:54:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:54:30.403153753 +0000 UTC m=+1321.110740063" watchObservedRunningTime="2026-10-31 12:54:30.407718206 +0000 UTC m=+1321.115304514" Oct 31 12:54:30.932832 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:30.932789 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" Oct 31 12:54:30.934999 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:30.934971 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 12:54:30.935248 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:30.935203 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" podUID="a3797b53-75e3-4584-8211-a06b15e03526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 12:54:31.936186 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:31.936153 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 12:54:31.936856 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:31.936809 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" podUID="a3797b53-75e3-4584-8211-a06b15e03526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 12:54:32.933551 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:32.933513 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 12:54:32.933800 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:32.933575 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" podUID="a3797b53-75e3-4584-8211-a06b15e03526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 12:54:33.445357 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:33.445324 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5ccdc5599d-mz2k5"] Oct 31 12:54:33.450767 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:33.450735 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5ccdc5599d-mz2k5" Oct 31 12:54:33.461189 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:33.459100 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/34abc9d3-a4c7-4f9b-9efd-64e156b45fc1-provider-credentials\") pod \"payload-processing-external-model-5ccdc5599d-mz2k5\" (UID: \"34abc9d3-a4c7-4f9b-9efd-64e156b45fc1\") " pod="models-as-a-service/payload-processing-external-model-5ccdc5599d-mz2k5" Oct 31 12:54:33.461189 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:33.459181 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34abc9d3-a4c7-4f9b-9efd-64e156b45fc1-routing-overlay-volume\") pod \"payload-processing-external-model-5ccdc5599d-mz2k5\" (UID: \"34abc9d3-a4c7-4f9b-9efd-64e156b45fc1\") " pod="models-as-a-service/payload-processing-external-model-5ccdc5599d-mz2k5" Oct 31 12:54:33.461189 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:33.459205 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj97v\" (UniqueName: \"kubernetes.io/projected/34abc9d3-a4c7-4f9b-9efd-64e156b45fc1-kube-api-access-nj97v\") pod \"payload-processing-external-model-5ccdc5599d-mz2k5\" (UID: \"34abc9d3-a4c7-4f9b-9efd-64e156b45fc1\") " pod="models-as-a-service/payload-processing-external-model-5ccdc5599d-mz2k5" Oct 31 12:54:33.461189 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:33.459344 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34abc9d3-a4c7-4f9b-9efd-64e156b45fc1-plugins-config-volume\") pod \"payload-processing-external-model-5ccdc5599d-mz2k5\" (UID: \"34abc9d3-a4c7-4f9b-9efd-64e156b45fc1\") " pod="models-as-a-service/payload-processing-external-model-5ccdc5599d-mz2k5" Oct 31 12:54:33.467367 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:33.467341 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5ccdc5599d-mz2k5"] Oct 31 12:54:33.560442 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:33.560396 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34abc9d3-a4c7-4f9b-9efd-64e156b45fc1-plugins-config-volume\") pod \"payload-processing-external-model-5ccdc5599d-mz2k5\" (UID: \"34abc9d3-a4c7-4f9b-9efd-64e156b45fc1\") " pod="models-as-a-service/payload-processing-external-model-5ccdc5599d-mz2k5" Oct 31 12:54:33.561247 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:33.561221 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34abc9d3-a4c7-4f9b-9efd-64e156b45fc1-plugins-config-volume\") pod \"payload-processing-external-model-5ccdc5599d-mz2k5\" (UID: \"34abc9d3-a4c7-4f9b-9efd-64e156b45fc1\") " pod="models-as-a-service/payload-processing-external-model-5ccdc5599d-mz2k5" Oct 31 12:54:33.561396 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:33.561378 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/34abc9d3-a4c7-4f9b-9efd-64e156b45fc1-provider-credentials\") pod \"payload-processing-external-model-5ccdc5599d-mz2k5\" (UID: \"34abc9d3-a4c7-4f9b-9efd-64e156b45fc1\") " pod="models-as-a-service/payload-processing-external-model-5ccdc5599d-mz2k5" Oct 31 12:54:33.561469 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:33.561441 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34abc9d3-a4c7-4f9b-9efd-64e156b45fc1-routing-overlay-volume\") pod \"payload-processing-external-model-5ccdc5599d-mz2k5\" (UID: \"34abc9d3-a4c7-4f9b-9efd-64e156b45fc1\") " pod="models-as-a-service/payload-processing-external-model-5ccdc5599d-mz2k5" Oct 31 12:54:33.561703 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:33.561679 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj97v\" (UniqueName: \"kubernetes.io/projected/34abc9d3-a4c7-4f9b-9efd-64e156b45fc1-kube-api-access-nj97v\") pod \"payload-processing-external-model-5ccdc5599d-mz2k5\" (UID: \"34abc9d3-a4c7-4f9b-9efd-64e156b45fc1\") " pod="models-as-a-service/payload-processing-external-model-5ccdc5599d-mz2k5" Oct 31 12:54:33.562305 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:33.562286 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34abc9d3-a4c7-4f9b-9efd-64e156b45fc1-routing-overlay-volume\") pod \"payload-processing-external-model-5ccdc5599d-mz2k5\" (UID: \"34abc9d3-a4c7-4f9b-9efd-64e156b45fc1\") " pod="models-as-a-service/payload-processing-external-model-5ccdc5599d-mz2k5" Oct 31 12:54:33.564081 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:33.564062 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/34abc9d3-a4c7-4f9b-9efd-64e156b45fc1-provider-credentials\") pod \"payload-processing-external-model-5ccdc5599d-mz2k5\" (UID: \"34abc9d3-a4c7-4f9b-9efd-64e156b45fc1\") " pod="models-as-a-service/payload-processing-external-model-5ccdc5599d-mz2k5" Oct 31 12:54:33.607673 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:33.607124 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj97v\" (UniqueName: \"kubernetes.io/projected/34abc9d3-a4c7-4f9b-9efd-64e156b45fc1-kube-api-access-nj97v\") pod \"payload-processing-external-model-5ccdc5599d-mz2k5\" (UID: \"34abc9d3-a4c7-4f9b-9efd-64e156b45fc1\") " pod="models-as-a-service/payload-processing-external-model-5ccdc5599d-mz2k5" Oct 31 12:54:33.806087 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:33.805985 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5ccdc5599d-mz2k5" Oct 31 12:54:33.932583 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:33.932406 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 12:54:33.932583 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:33.932449 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" podUID="a3797b53-75e3-4584-8211-a06b15e03526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 12:54:34.098744 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:34.092434 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5ccdc5599d-mz2k5"] Oct 31 12:54:34.106429 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:54:34.106405 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34abc9d3_a4c7_4f9b_9efd_64e156b45fc1.slice/crio-083a27bc8318bf03603afd98cca31cb883ef4732fdfbfd92298f412a873933c3 WatchSource:0}: Error finding container 083a27bc8318bf03603afd98cca31cb883ef4732fdfbfd92298f412a873933c3: Status 404 returned error can't find the container with id 083a27bc8318bf03603afd98cca31cb883ef4732fdfbfd92298f412a873933c3 Oct 31 12:54:34.251832 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:34.251793 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp"] Oct 31 12:54:34.256547 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:34.256528 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp" Oct 31 12:54:34.265946 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:34.265923 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp"] Oct 31 12:54:34.270411 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:34.270390 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/800da7e5-09e1-4844-a967-13258ecb47cc-dshm\") pod \"e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp\" (UID: \"800da7e5-09e1-4844-a967-13258ecb47cc\") " pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp" Oct 31 12:54:34.270530 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:34.270425 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/800da7e5-09e1-4844-a967-13258ecb47cc-tmp-dir\") pod \"e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp\" (UID: \"800da7e5-09e1-4844-a967-13258ecb47cc\") " pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp" Oct 31 12:54:34.270530 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:34.270454 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/800da7e5-09e1-4844-a967-13258ecb47cc-home\") pod \"e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp\" (UID: \"800da7e5-09e1-4844-a967-13258ecb47cc\") " pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp" Oct 31 12:54:34.270601 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:34.270585 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/800da7e5-09e1-4844-a967-13258ecb47cc-model-cache\") pod \"e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp\" (UID: \"800da7e5-09e1-4844-a967-13258ecb47cc\") " pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp" Oct 31 12:54:34.270700 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:34.270683 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/800da7e5-09e1-4844-a967-13258ecb47cc-tls-certs\") pod \"e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp\" (UID: \"800da7e5-09e1-4844-a967-13258ecb47cc\") " pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp" Oct 31 12:54:34.270746 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:34.270721 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhjwx\" (UniqueName: \"kubernetes.io/projected/800da7e5-09e1-4844-a967-13258ecb47cc-kube-api-access-xhjwx\") pod \"e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp\" (UID: \"800da7e5-09e1-4844-a967-13258ecb47cc\") " pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp" Oct 31 12:54:34.305252 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:34.305213 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-47bcd9\"/\"openshift-service-ca.crt\"" Oct 31 12:54:34.305790 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:34.305578 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-47bcd9\"/\"e2e-conflict-a-2f2da815-kserve-self-signed-certs\"" Oct 31 12:54:34.305790 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:34.305695 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-47bcd9\"/\"default-dockercfg-pcs5t\"" Oct 31 12:54:34.305790 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:34.305739 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-47bcd9\"/\"kube-root-ca.crt\"" Oct 31 12:54:34.371422 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:34.371331 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/800da7e5-09e1-4844-a967-13258ecb47cc-tmp-dir\") pod \"e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp\" (UID: \"800da7e5-09e1-4844-a967-13258ecb47cc\") " pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp" Oct 31 12:54:34.371422 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:34.371384 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/800da7e5-09e1-4844-a967-13258ecb47cc-home\") pod \"e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp\" (UID: \"800da7e5-09e1-4844-a967-13258ecb47cc\") " pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp" Oct 31 12:54:34.371684 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:34.371425 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/800da7e5-09e1-4844-a967-13258ecb47cc-model-cache\") pod \"e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp\" (UID: \"800da7e5-09e1-4844-a967-13258ecb47cc\") " pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp" Oct 31 12:54:34.371684 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:34.371483 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/800da7e5-09e1-4844-a967-13258ecb47cc-tls-certs\") pod \"e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp\" (UID: \"800da7e5-09e1-4844-a967-13258ecb47cc\") " pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp" Oct 31 12:54:34.371684 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:34.371519 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhjwx\" (UniqueName: \"kubernetes.io/projected/800da7e5-09e1-4844-a967-13258ecb47cc-kube-api-access-xhjwx\") pod \"e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp\" (UID: \"800da7e5-09e1-4844-a967-13258ecb47cc\") " pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp" Oct 31 12:54:34.371684 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:34.371601 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/800da7e5-09e1-4844-a967-13258ecb47cc-dshm\") pod \"e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp\" (UID: \"800da7e5-09e1-4844-a967-13258ecb47cc\") " pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp" Oct 31 12:54:34.371936 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:34.371789 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/800da7e5-09e1-4844-a967-13258ecb47cc-tmp-dir\") pod \"e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp\" (UID: \"800da7e5-09e1-4844-a967-13258ecb47cc\") " pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp" Oct 31 12:54:34.371936 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:34.371842 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/800da7e5-09e1-4844-a967-13258ecb47cc-home\") pod \"e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp\" (UID: \"800da7e5-09e1-4844-a967-13258ecb47cc\") " pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp" Oct 31 12:54:34.371936 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:34.371878 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/800da7e5-09e1-4844-a967-13258ecb47cc-model-cache\") pod \"e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp\" (UID: \"800da7e5-09e1-4844-a967-13258ecb47cc\") " pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp" Oct 31 12:54:34.373994 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:34.373969 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/800da7e5-09e1-4844-a967-13258ecb47cc-dshm\") pod \"e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp\" (UID: \"800da7e5-09e1-4844-a967-13258ecb47cc\") " pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp" Oct 31 12:54:34.374289 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:34.374273 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/800da7e5-09e1-4844-a967-13258ecb47cc-tls-certs\") pod \"e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp\" (UID: \"800da7e5-09e1-4844-a967-13258ecb47cc\") " pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp" Oct 31 12:54:34.410820 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:34.410391 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhjwx\" (UniqueName: \"kubernetes.io/projected/800da7e5-09e1-4844-a967-13258ecb47cc-kube-api-access-xhjwx\") pod \"e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp\" (UID: \"800da7e5-09e1-4844-a967-13258ecb47cc\") " pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp" Oct 31 12:54:34.451841 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:34.451794 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5ccdc5599d-mz2k5" event={"ID":"34abc9d3-a4c7-4f9b-9efd-64e156b45fc1","Type":"ContainerStarted","Data":"1a3f7e84bf8a6c4ae069528325f3a479b8352322a67c9e656bb09870a3e397c4"} Oct 31 12:54:34.452277 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:34.451858 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5ccdc5599d-mz2k5" Oct 31 12:54:34.452277 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:34.451888 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5ccdc5599d-mz2k5" event={"ID":"34abc9d3-a4c7-4f9b-9efd-64e156b45fc1","Type":"ContainerStarted","Data":"083a27bc8318bf03603afd98cca31cb883ef4732fdfbfd92298f412a873933c3"} Oct 31 12:54:34.473525 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:34.473478 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5ccdc5599d-mz2k5" podStartSLOduration=1.4734651539999999 podStartE2EDuration="1.473465154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:54:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:54:34.471119843 +0000 UTC m=+1325.178706138" watchObservedRunningTime="2026-10-31 12:54:34.473465154 +0000 UTC m=+1325.181051449" Oct 31 12:54:34.570203 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:34.570174 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp" Oct 31 12:54:34.906517 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:34.905509 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp"] Oct 31 12:54:34.907892 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:54:34.907859 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod800da7e5_09e1_4844_a967_13258ecb47cc.slice/crio-13dfbf21894342358b6e40ea1d12fc871d791c4ebeda572ff2cb918f032d1d67 WatchSource:0}: Error finding container 13dfbf21894342358b6e40ea1d12fc871d791c4ebeda572ff2cb918f032d1d67: Status 404 returned error can't find the container with id 13dfbf21894342358b6e40ea1d12fc871d791c4ebeda572ff2cb918f032d1d67 Oct 31 12:54:34.933451 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:34.933421 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 12:54:34.933576 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:34.933478 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" podUID="a3797b53-75e3-4584-8211-a06b15e03526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 12:54:35.459566 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:35.459531 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp" event={"ID":"800da7e5-09e1-4844-a967-13258ecb47cc","Type":"ContainerStarted","Data":"d703330527ce9e48594bff3449a2d2d62695aee086fb88e35de0cadd8e49f424"} Oct 31 12:54:35.459566 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:35.459572 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp" event={"ID":"800da7e5-09e1-4844-a967-13258ecb47cc","Type":"ContainerStarted","Data":"13dfbf21894342358b6e40ea1d12fc871d791c4ebeda572ff2cb918f032d1d67"} Oct 31 12:54:35.498663 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:35.489739 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp" podStartSLOduration=1.489705778 podStartE2EDuration="1.489705778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:54:35.489041147 +0000 UTC m=+1326.196627443" watchObservedRunningTime="2026-10-31 12:54:35.489705778 +0000 UTC m=+1326.197292074" Oct 31 12:54:35.571253 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:35.571215 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp" Oct 31 12:54:35.933083 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:35.933052 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 12:54:35.933296 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:35.933103 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" podUID="a3797b53-75e3-4584-8211-a06b15e03526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 12:54:36.938176 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:36.938140 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" Oct 31 12:54:36.940575 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:36.940549 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" Oct 31 12:54:36.942037 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:36.942018 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" Oct 31 12:54:44.571050 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:44.571015 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp" Oct 31 12:54:44.661437 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:44.661401 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp" Oct 31 12:54:45.462538 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:45.462514 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5ccdc5599d-mz2k5" Oct 31 12:54:45.557572 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:45.557520 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7f85c5f9-6tx9r"] Oct 31 12:54:45.598190 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:45.598145 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp" Oct 31 12:54:48.686873 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:54:48.686838 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:54:50.558470 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:50.558420 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7f85c5f9-6tx9r" podUID="96086778-e6af-42e9-89eb-f2537e9025b7" containerName="payload-processing" containerID="cri-o://c4845160ef3b17cb4e26e2cb5d4ac39b8e135633a1ec7dcb7ea8a8664e1f72b5" gracePeriod=25 Oct 31 12:54:50.818221 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:50.818195 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7f85c5f9-6tx9r" Oct 31 12:54:50.915475 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:50.915438 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96086778-e6af-42e9-89eb-f2537e9025b7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/96086778-e6af-42e9-89eb-f2537e9025b7-routing-overlay-volume\") pod \"96086778-e6af-42e9-89eb-f2537e9025b7\" (UID: \"96086778-e6af-42e9-89eb-f2537e9025b7\") " Oct 31 12:54:50.915703 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:50.915491 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96086778-e6af-42e9-89eb-f2537e9025b7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/96086778-e6af-42e9-89eb-f2537e9025b7-provider-credentials\") pod \"96086778-e6af-42e9-89eb-f2537e9025b7\" (UID: \"96086778-e6af-42e9-89eb-f2537e9025b7\") " Oct 31 12:54:50.915703 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:50.915569 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96086778-e6af-42e9-89eb-f2537e9025b7-kube-api-access-85pgw\" (UniqueName: \"kubernetes.io/projected/96086778-e6af-42e9-89eb-f2537e9025b7-kube-api-access-85pgw\") pod \"96086778-e6af-42e9-89eb-f2537e9025b7\" (UID: \"96086778-e6af-42e9-89eb-f2537e9025b7\") " Oct 31 12:54:50.915703 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:50.915602 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96086778-e6af-42e9-89eb-f2537e9025b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96086778-e6af-42e9-89eb-f2537e9025b7-plugins-config-volume\") pod \"96086778-e6af-42e9-89eb-f2537e9025b7\" (UID: \"96086778-e6af-42e9-89eb-f2537e9025b7\") " Oct 31 12:54:50.915905 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:50.915873 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96086778-e6af-42e9-89eb-f2537e9025b7-routing-overlay-volume" pod "96086778-e6af-42e9-89eb-f2537e9025b7" (UID: "96086778-e6af-42e9-89eb-f2537e9025b7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:54:50.916011 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:50.915993 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/96086778-e6af-42e9-89eb-f2537e9025b7-routing-overlay-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:50.916054 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:50.916028 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96086778-e6af-42e9-89eb-f2537e9025b7-plugins-config-volume" pod "96086778-e6af-42e9-89eb-f2537e9025b7" (UID: "96086778-e6af-42e9-89eb-f2537e9025b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:54:50.918078 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:50.918047 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96086778-e6af-42e9-89eb-f2537e9025b7-provider-credentials" pod "96086778-e6af-42e9-89eb-f2537e9025b7" (UID: "96086778-e6af-42e9-89eb-f2537e9025b7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:54:50.918202 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:50.918181 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96086778-e6af-42e9-89eb-f2537e9025b7-kube-api-access-85pgw" pod "96086778-e6af-42e9-89eb-f2537e9025b7" (UID: "96086778-e6af-42e9-89eb-f2537e9025b7"). InnerVolumeSpecName "kube-api-access-85pgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:54:51.017287 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:51.017258 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/96086778-e6af-42e9-89eb-f2537e9025b7-provider-credentials\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:51.017287 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:51.017285 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85pgw\" (UniqueName: \"kubernetes.io/projected/96086778-e6af-42e9-89eb-f2537e9025b7-kube-api-access-85pgw\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:51.017473 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:51.017295 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96086778-e6af-42e9-89eb-f2537e9025b7-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:51.600373 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:51.600334 5693 generic.go:356] "Generic (PLEG): container finished" podID="96086778-e6af-42e9-89eb-f2537e9025b7" containerID="c4845160ef3b17cb4e26e2cb5d4ac39b8e135633a1ec7dcb7ea8a8664e1f72b5" exitCode=0 Oct 31 12:54:51.601202 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:51.600406 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7f85c5f9-6tx9r" Oct 31 12:54:51.601202 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:51.600430 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7f85c5f9-6tx9r" event={"ID":"96086778-e6af-42e9-89eb-f2537e9025b7","Type":"ContainerDied","Data":"c4845160ef3b17cb4e26e2cb5d4ac39b8e135633a1ec7dcb7ea8a8664e1f72b5"} Oct 31 12:54:51.601202 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:51.600479 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7f85c5f9-6tx9r" event={"ID":"96086778-e6af-42e9-89eb-f2537e9025b7","Type":"ContainerDied","Data":"bfa86ad05fd2b02c7dbbdfd7ba6932c75e22579872ffa03918fde0cf421c6e73"} Oct 31 12:54:51.601202 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:51.600510 5693 scope.go:122] "RemoveContainer" containerID="c4845160ef3b17cb4e26e2cb5d4ac39b8e135633a1ec7dcb7ea8a8664e1f72b5" Oct 31 12:54:51.614150 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:51.614127 5693 scope.go:122] "RemoveContainer" containerID="c4845160ef3b17cb4e26e2cb5d4ac39b8e135633a1ec7dcb7ea8a8664e1f72b5" Oct 31 12:54:51.614519 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:54:51.614490 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4845160ef3b17cb4e26e2cb5d4ac39b8e135633a1ec7dcb7ea8a8664e1f72b5\": container with ID starting with c4845160ef3b17cb4e26e2cb5d4ac39b8e135633a1ec7dcb7ea8a8664e1f72b5 not found: ID does not exist" containerID="c4845160ef3b17cb4e26e2cb5d4ac39b8e135633a1ec7dcb7ea8a8664e1f72b5" Oct 31 12:54:51.614633 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:51.614528 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4845160ef3b17cb4e26e2cb5d4ac39b8e135633a1ec7dcb7ea8a8664e1f72b5"} err="failed to get container status \"c4845160ef3b17cb4e26e2cb5d4ac39b8e135633a1ec7dcb7ea8a8664e1f72b5\": rpc error: code = NotFound desc = could not find container \"c4845160ef3b17cb4e26e2cb5d4ac39b8e135633a1ec7dcb7ea8a8664e1f72b5\": container with ID starting with c4845160ef3b17cb4e26e2cb5d4ac39b8e135633a1ec7dcb7ea8a8664e1f72b5 not found: ID does not exist" Oct 31 12:54:51.627792 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:51.627758 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7f85c5f9-6tx9r"] Oct 31 12:54:51.627952 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:51.627808 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7f85c5f9-6tx9r"] Oct 31 12:54:53.408021 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:53.407988 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96086778-e6af-42e9-89eb-f2537e9025b7" path="/var/lib/kubelet/pods/96086778-e6af-42e9-89eb-f2537e9025b7/volumes" Oct 31 12:54:57.488166 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:57.488131 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9"] Oct 31 12:54:57.488848 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:57.488828 5693 cpu_manager.go:414] "containerMap: removing container" podUID="96086778-e6af-42e9-89eb-f2537e9025b7" containerName="payload-processing" Oct 31 12:54:57.488934 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:57.488849 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96086778-e6af-42e9-89eb-f2537e9025b7" containerName="payload-processing" Oct 31 12:54:57.488999 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:57.488977 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="96086778-e6af-42e9-89eb-f2537e9025b7" containerName="payload-processing" Oct 31 12:54:57.493598 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:57.493577 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" Oct 31 12:54:57.498432 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:57.498342 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-3dd1e5\"/\"openshift-service-ca.crt\"" Oct 31 12:54:57.498631 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:57.498454 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-3dd1e5\"/\"distinct-w2-3dd1e5-kserve-self-signed-certs\"" Oct 31 12:54:57.499413 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:57.499292 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-3dd1e5\"/\"kube-root-ca.crt\"" Oct 31 12:54:57.504674 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:57.501367 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-3dd1e5\"/\"default-dockercfg-hzxqn\"" Oct 31 12:54:57.504674 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:57.503006 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9"] Oct 31 12:54:57.596561 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:57.596522 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/710a52fa-2bc4-4bb9-96be-4b08dc60de67-home\") pod \"distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9\" (UID: \"710a52fa-2bc4-4bb9-96be-4b08dc60de67\") " pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" Oct 31 12:54:57.596561 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:57.596556 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/710a52fa-2bc4-4bb9-96be-4b08dc60de67-dshm\") pod \"distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9\" (UID: \"710a52fa-2bc4-4bb9-96be-4b08dc60de67\") " pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" Oct 31 12:54:57.596796 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:57.596573 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/710a52fa-2bc4-4bb9-96be-4b08dc60de67-model-cache\") pod \"distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9\" (UID: \"710a52fa-2bc4-4bb9-96be-4b08dc60de67\") " pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" Oct 31 12:54:57.596796 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:57.596589 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/710a52fa-2bc4-4bb9-96be-4b08dc60de67-tmp-dir\") pod \"distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9\" (UID: \"710a52fa-2bc4-4bb9-96be-4b08dc60de67\") " pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" Oct 31 12:54:57.596796 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:57.596701 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nmdx\" (UniqueName: \"kubernetes.io/projected/710a52fa-2bc4-4bb9-96be-4b08dc60de67-kube-api-access-4nmdx\") pod \"distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9\" (UID: \"710a52fa-2bc4-4bb9-96be-4b08dc60de67\") " pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" Oct 31 12:54:57.596796 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:57.596749 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/710a52fa-2bc4-4bb9-96be-4b08dc60de67-tls-certs\") pod \"distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9\" (UID: \"710a52fa-2bc4-4bb9-96be-4b08dc60de67\") " pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" Oct 31 12:54:57.698084 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:57.698044 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/710a52fa-2bc4-4bb9-96be-4b08dc60de67-home\") pod \"distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9\" (UID: \"710a52fa-2bc4-4bb9-96be-4b08dc60de67\") " pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" Oct 31 12:54:57.698084 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:57.698096 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/710a52fa-2bc4-4bb9-96be-4b08dc60de67-dshm\") pod \"distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9\" (UID: \"710a52fa-2bc4-4bb9-96be-4b08dc60de67\") " pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" Oct 31 12:54:57.698354 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:57.698211 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/710a52fa-2bc4-4bb9-96be-4b08dc60de67-model-cache\") pod \"distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9\" (UID: \"710a52fa-2bc4-4bb9-96be-4b08dc60de67\") " pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" Oct 31 12:54:57.698354 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:57.698257 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/710a52fa-2bc4-4bb9-96be-4b08dc60de67-tmp-dir\") pod \"distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9\" (UID: \"710a52fa-2bc4-4bb9-96be-4b08dc60de67\") " pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" Oct 31 12:54:57.698354 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:57.698315 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nmdx\" (UniqueName: \"kubernetes.io/projected/710a52fa-2bc4-4bb9-96be-4b08dc60de67-kube-api-access-4nmdx\") pod \"distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9\" (UID: \"710a52fa-2bc4-4bb9-96be-4b08dc60de67\") " pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" Oct 31 12:54:57.698519 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:57.698362 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/710a52fa-2bc4-4bb9-96be-4b08dc60de67-tls-certs\") pod \"distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9\" (UID: \"710a52fa-2bc4-4bb9-96be-4b08dc60de67\") " pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" Oct 31 12:54:57.698599 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:57.698546 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/710a52fa-2bc4-4bb9-96be-4b08dc60de67-home\") pod \"distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9\" (UID: \"710a52fa-2bc4-4bb9-96be-4b08dc60de67\") " pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" Oct 31 12:54:57.698829 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:57.698804 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/710a52fa-2bc4-4bb9-96be-4b08dc60de67-tmp-dir\") pod \"distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9\" (UID: \"710a52fa-2bc4-4bb9-96be-4b08dc60de67\") " pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" Oct 31 12:54:57.699081 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:57.699059 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/710a52fa-2bc4-4bb9-96be-4b08dc60de67-model-cache\") pod \"distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9\" (UID: \"710a52fa-2bc4-4bb9-96be-4b08dc60de67\") " pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" Oct 31 12:54:57.701366 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:57.701341 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/710a52fa-2bc4-4bb9-96be-4b08dc60de67-tls-certs\") pod \"distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9\" (UID: \"710a52fa-2bc4-4bb9-96be-4b08dc60de67\") " pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" Oct 31 12:54:57.701545 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:57.701505 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/710a52fa-2bc4-4bb9-96be-4b08dc60de67-dshm\") pod \"distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9\" (UID: \"710a52fa-2bc4-4bb9-96be-4b08dc60de67\") " pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" Oct 31 12:54:57.710805 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:57.710782 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nmdx\" (UniqueName: \"kubernetes.io/projected/710a52fa-2bc4-4bb9-96be-4b08dc60de67-kube-api-access-4nmdx\") pod \"distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9\" (UID: \"710a52fa-2bc4-4bb9-96be-4b08dc60de67\") " pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" Oct 31 12:54:57.809547 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:57.809457 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" Oct 31 12:54:58.091732 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:54:58.091698 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod710a52fa_2bc4_4bb9_96be_4b08dc60de67.slice/crio-ade6940a3d9bc4b56f7d64e574bef8ae721f91497e277bea78d5fb0274572f32 WatchSource:0}: Error finding container ade6940a3d9bc4b56f7d64e574bef8ae721f91497e277bea78d5fb0274572f32: Status 404 returned error can't find the container with id ade6940a3d9bc4b56f7d64e574bef8ae721f91497e277bea78d5fb0274572f32 Oct 31 12:54:58.092439 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:58.092406 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9"] Oct 31 12:54:58.646442 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:58.646405 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" event={"ID":"710a52fa-2bc4-4bb9-96be-4b08dc60de67","Type":"ContainerStarted","Data":"08a64c99723396b4e181f3e1e86de7013dbd2fe18bc6f6bc7193511246fd5463"} Oct 31 12:54:58.646881 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:58.646454 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" event={"ID":"710a52fa-2bc4-4bb9-96be-4b08dc60de67","Type":"ContainerStarted","Data":"ade6940a3d9bc4b56f7d64e574bef8ae721f91497e277bea78d5fb0274572f32"} Oct 31 12:54:58.809690 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:54:58.809631 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" Oct 31 12:55:07.224877 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:07.224827 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" podStartSLOduration=10.224812112 podStartE2EDuration="10.224812112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:54:58.671492734 +0000 UTC m=+1349.379079046" watchObservedRunningTime="2026-10-31 12:55:07.224812112 +0000 UTC m=+1357.932398407" Oct 31 12:55:07.225784 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:07.225764 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv"] Oct 31 12:55:07.230866 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:07.230844 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" Oct 31 12:55:07.238187 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:07.238142 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7ae9e7\"/\"distinct-2-w4-7ae9e7-kserve-self-signed-certs\"" Oct 31 12:55:07.242210 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:07.241741 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv"] Oct 31 12:55:07.308123 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:07.308089 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kchh\" (UniqueName: \"kubernetes.io/projected/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-kube-api-access-8kchh\") pod \"distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv\" (UID: \"f2b3766f-058c-40e7-9a2d-a7f5028e2b1d\") " pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" Oct 31 12:55:07.308123 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:07.308128 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-home\") pod \"distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv\" (UID: \"f2b3766f-058c-40e7-9a2d-a7f5028e2b1d\") " pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" Oct 31 12:55:07.308349 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:07.308220 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-dshm\") pod \"distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv\" (UID: \"f2b3766f-058c-40e7-9a2d-a7f5028e2b1d\") " pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" Oct 31 12:55:07.308349 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:07.308281 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-model-cache\") pod \"distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv\" (UID: \"f2b3766f-058c-40e7-9a2d-a7f5028e2b1d\") " pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" Oct 31 12:55:07.308349 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:07.308320 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-tmp-dir\") pod \"distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv\" (UID: \"f2b3766f-058c-40e7-9a2d-a7f5028e2b1d\") " pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" Oct 31 12:55:07.308479 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:07.308384 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-tls-certs\") pod \"distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv\" (UID: \"f2b3766f-058c-40e7-9a2d-a7f5028e2b1d\") " pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" Oct 31 12:55:07.409062 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:07.409029 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-tls-certs\") pod \"distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv\" (UID: \"f2b3766f-058c-40e7-9a2d-a7f5028e2b1d\") " pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" Oct 31 12:55:07.409274 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:07.409126 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kchh\" (UniqueName: \"kubernetes.io/projected/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-kube-api-access-8kchh\") pod \"distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv\" (UID: \"f2b3766f-058c-40e7-9a2d-a7f5028e2b1d\") " pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" Oct 31 12:55:07.409274 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:07.409158 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-home\") pod \"distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv\" (UID: \"f2b3766f-058c-40e7-9a2d-a7f5028e2b1d\") " pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" Oct 31 12:55:07.409274 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:07.409218 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-dshm\") pod \"distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv\" (UID: \"f2b3766f-058c-40e7-9a2d-a7f5028e2b1d\") " pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" Oct 31 12:55:07.409274 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:07.409259 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-model-cache\") pod \"distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv\" (UID: \"f2b3766f-058c-40e7-9a2d-a7f5028e2b1d\") " pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" Oct 31 12:55:07.409490 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:07.409278 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-tmp-dir\") pod \"distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv\" (UID: \"f2b3766f-058c-40e7-9a2d-a7f5028e2b1d\") " pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" Oct 31 12:55:07.409858 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:07.409821 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-tmp-dir\") pod \"distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv\" (UID: \"f2b3766f-058c-40e7-9a2d-a7f5028e2b1d\") " pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" Oct 31 12:55:07.409989 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:07.409824 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-home\") pod \"distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv\" (UID: \"f2b3766f-058c-40e7-9a2d-a7f5028e2b1d\") " pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" Oct 31 12:55:07.410141 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:07.410116 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-model-cache\") pod \"distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv\" (UID: \"f2b3766f-058c-40e7-9a2d-a7f5028e2b1d\") " pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" Oct 31 12:55:07.412335 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:07.412310 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-dshm\") pod \"distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv\" (UID: \"f2b3766f-058c-40e7-9a2d-a7f5028e2b1d\") " pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" Oct 31 12:55:07.413678 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:07.413631 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-tls-certs\") pod \"distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv\" (UID: \"f2b3766f-058c-40e7-9a2d-a7f5028e2b1d\") " pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" Oct 31 12:55:07.425511 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:07.425485 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kchh\" (UniqueName: \"kubernetes.io/projected/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-kube-api-access-8kchh\") pod \"distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv\" (UID: \"f2b3766f-058c-40e7-9a2d-a7f5028e2b1d\") " pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" Oct 31 12:55:07.549419 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:07.549333 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" Oct 31 12:55:07.810485 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:07.810405 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" Oct 31 12:55:07.823453 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:07.823426 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" Oct 31 12:55:07.831605 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:07.831582 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv"] Oct 31 12:55:07.833855 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:55:07.833831 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2b3766f_058c_40e7_9a2d_a7f5028e2b1d.slice/crio-19551b0daedad477182f85862e846b25e8e75f423c1800f401e27c197fb87d56 WatchSource:0}: Error finding container 19551b0daedad477182f85862e846b25e8e75f423c1800f401e27c197fb87d56: Status 404 returned error can't find the container with id 19551b0daedad477182f85862e846b25e8e75f423c1800f401e27c197fb87d56 Oct 31 12:55:08.704669 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:08.704548 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" event={"ID":"f2b3766f-058c-40e7-9a2d-a7f5028e2b1d","Type":"ContainerStarted","Data":"f3dde38c04b1e25c3a22ff0bd13f65bf7679718b2dec1e1fb665093273c6b508"} Oct 31 12:55:08.704669 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:08.704599 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" event={"ID":"f2b3766f-058c-40e7-9a2d-a7f5028e2b1d","Type":"ContainerStarted","Data":"19551b0daedad477182f85862e846b25e8e75f423c1800f401e27c197fb87d56"} Oct 31 12:55:08.715964 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:08.715941 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" Oct 31 12:55:08.725404 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:08.725360 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" podStartSLOduration=1.7253488529999998 podStartE2EDuration="1.725348853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:55:08.722702363 +0000 UTC m=+1359.430288651" watchObservedRunningTime="2026-10-31 12:55:08.725348853 +0000 UTC m=+1359.432935147" Oct 31 12:55:09.550006 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:09.549980 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" Oct 31 12:55:13.669660 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:13.668297 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf"] Oct 31 12:55:13.680659 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:13.676343 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" Oct 31 12:55:13.700686 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:13.683324 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf"] Oct 31 12:55:13.784143 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:13.784106 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52b4bc93-af99-4c3e-bddb-c087f6c0247c-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf\" (UID: \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" Oct 31 12:55:13.797632 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:13.797596 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b4bc93-af99-4c3e-bddb-c087f6c0247c-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf\" (UID: \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" Oct 31 12:55:13.797917 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:13.797894 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5lxt\" (UniqueName: \"kubernetes.io/projected/52b4bc93-af99-4c3e-bddb-c087f6c0247c-kube-api-access-p5lxt\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf\" (UID: \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" Oct 31 12:55:13.798275 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:13.798241 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52b4bc93-af99-4c3e-bddb-c087f6c0247c-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf\" (UID: \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" Oct 31 12:55:13.798532 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:13.798512 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52b4bc93-af99-4c3e-bddb-c087f6c0247c-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf\" (UID: \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" Oct 31 12:55:13.798712 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:13.798693 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52b4bc93-af99-4c3e-bddb-c087f6c0247c-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf\" (UID: \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" Oct 31 12:55:13.798854 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:13.798837 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52b4bc93-af99-4c3e-bddb-c087f6c0247c-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf\" (UID: \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" Oct 31 12:55:13.798972 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:13.798956 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b4bc93-af99-4c3e-bddb-c087f6c0247c-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf\" (UID: \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" Oct 31 12:55:13.799095 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:13.799079 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52b4bc93-af99-4c3e-bddb-c087f6c0247c-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf\" (UID: \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" Oct 31 12:55:13.902833 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:13.902792 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b4bc93-af99-4c3e-bddb-c087f6c0247c-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf\" (UID: \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" Oct 31 12:55:13.903673 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:13.903631 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52b4bc93-af99-4c3e-bddb-c087f6c0247c-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf\" (UID: \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" Oct 31 12:55:13.904214 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:13.903553 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b4bc93-af99-4c3e-bddb-c087f6c0247c-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf\" (UID: \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" Oct 31 12:55:13.906623 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:13.906600 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52b4bc93-af99-4c3e-bddb-c087f6c0247c-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf\" (UID: \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" Oct 31 12:55:13.907204 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:13.907183 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b4bc93-af99-4c3e-bddb-c087f6c0247c-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf\" (UID: \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" Oct 31 12:55:13.907665 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:13.907627 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5lxt\" (UniqueName: \"kubernetes.io/projected/52b4bc93-af99-4c3e-bddb-c087f6c0247c-kube-api-access-p5lxt\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf\" (UID: \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" Oct 31 12:55:13.908132 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:13.907058 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52b4bc93-af99-4c3e-bddb-c087f6c0247c-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf\" (UID: \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" Oct 31 12:55:13.908264 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:13.907570 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b4bc93-af99-4c3e-bddb-c087f6c0247c-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf\" (UID: \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" Oct 31 12:55:13.908382 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:13.908096 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52b4bc93-af99-4c3e-bddb-c087f6c0247c-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf\" (UID: \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" Oct 31 12:55:13.908938 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:13.908917 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52b4bc93-af99-4c3e-bddb-c087f6c0247c-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf\" (UID: \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" Oct 31 12:55:13.909433 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:13.909410 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52b4bc93-af99-4c3e-bddb-c087f6c0247c-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf\" (UID: \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" Oct 31 12:55:13.910350 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:13.910331 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52b4bc93-af99-4c3e-bddb-c087f6c0247c-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf\" (UID: \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" Oct 31 12:55:13.913016 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:13.909304 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52b4bc93-af99-4c3e-bddb-c087f6c0247c-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf\" (UID: \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" Oct 31 12:55:13.913191 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:13.910263 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52b4bc93-af99-4c3e-bddb-c087f6c0247c-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf\" (UID: \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" Oct 31 12:55:13.919514 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:13.919488 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52b4bc93-af99-4c3e-bddb-c087f6c0247c-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf\" (UID: \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" Oct 31 12:55:13.929716 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:13.926137 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52b4bc93-af99-4c3e-bddb-c087f6c0247c-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf\" (UID: \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" Oct 31 12:55:13.930412 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:13.930389 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52b4bc93-af99-4c3e-bddb-c087f6c0247c-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf\" (UID: \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" Oct 31 12:55:13.942145 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:13.931469 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5lxt\" (UniqueName: \"kubernetes.io/projected/52b4bc93-af99-4c3e-bddb-c087f6c0247c-kube-api-access-p5lxt\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf\" (UID: \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" Oct 31 12:55:14.087664 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:14.086880 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" Oct 31 12:55:14.452811 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:14.452779 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5ccdc5599d-mz2k5"] Oct 31 12:55:14.507044 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:14.505133 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf"] Oct 31 12:55:14.522189 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:14.520591 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:55:14.522189 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:14.520704 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:55:14.522189 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:14.520755 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:55:14.814728 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:14.814600 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" event={"ID":"52b4bc93-af99-4c3e-bddb-c087f6c0247c","Type":"ContainerStarted","Data":"5f331ad908d23f77d04b95af2c98dbcb5a505fb0e1361c1097526f4088481afa"} Oct 31 12:55:14.815337 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:14.815312 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" event={"ID":"52b4bc93-af99-4c3e-bddb-c087f6c0247c","Type":"ContainerStarted","Data":"8ea044d3f0163207bd72d3b9db2791c59fc04a586b8b38c642c8b80f8ea14d44"} Oct 31 12:55:15.093014 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:15.092892 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" Oct 31 12:55:15.093407 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:15.093383 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 12:55:15.093619 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:15.093581 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" podUID="52b4bc93-af99-4c3e-bddb-c087f6c0247c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 12:55:16.100664 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:16.089791 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 12:55:16.100664 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:16.089888 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" podUID="52b4bc93-af99-4c3e-bddb-c087f6c0247c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 12:55:16.712421 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:16.711735 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" podUID="710a52fa-2bc4-4bb9-96be-4b08dc60de67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:16.737676 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:16.735734 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" podUID="17d56bfd-8cb1-42bf-bb11-05810c46d469" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:16.786675 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:16.784717 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" podUID="8fb66a94-c399-43fb-945c-610ca63f0d76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:16.791694 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:16.790682 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" podUID="66c6ec5a-c0da-4bd7-a314-0c90ccec2425" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:16.842706 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:16.841407 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" podUID="1a2abd3f-fb4f-4990-9a32-5f945a588068" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 31 12:55:17.092323 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:17.092214 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 12:55:17.092621 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:17.092573 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" podUID="52b4bc93-af99-4c3e-bddb-c087f6c0247c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 12:55:17.550323 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:17.550291 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" Oct 31 12:55:17.563442 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:17.563416 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" Oct 31 12:55:17.597668 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:17.596339 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" podStartSLOduration=4.596318163 podStartE2EDuration="4.596318163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:55:14.843560595 +0000 UTC m=+1365.551146891" watchObservedRunningTime="2026-10-31 12:55:17.596318163 +0000 UTC m=+1368.303904459" Oct 31 12:55:17.871505 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:17.871429 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" Oct 31 12:55:18.098667 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:18.098085 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 12:55:18.098667 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:18.098151 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" podUID="52b4bc93-af99-4c3e-bddb-c087f6c0247c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 12:55:18.711415 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:55:18.711382 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:55:19.089117 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:19.089021 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 12:55:19.089396 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:19.089364 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" podUID="52b4bc93-af99-4c3e-bddb-c087f6c0247c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 12:55:19.453503 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:19.453465 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5ccdc5599d-mz2k5" podUID="34abc9d3-a4c7-4f9b-9efd-64e156b45fc1" containerName="payload-processing" containerID="cri-o://1a3f7e84bf8a6c4ae069528325f3a479b8352322a67c9e656bb09870a3e397c4" gracePeriod=25 Oct 31 12:55:19.769560 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:19.769534 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5ccdc5599d-mz2k5" Oct 31 12:55:19.873568 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:19.873534 5693 generic.go:356] "Generic (PLEG): container finished" podID="34abc9d3-a4c7-4f9b-9efd-64e156b45fc1" containerID="1a3f7e84bf8a6c4ae069528325f3a479b8352322a67c9e656bb09870a3e397c4" exitCode=0 Oct 31 12:55:19.873743 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:19.873593 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5ccdc5599d-mz2k5" Oct 31 12:55:19.873743 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:19.873624 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5ccdc5599d-mz2k5" event={"ID":"34abc9d3-a4c7-4f9b-9efd-64e156b45fc1","Type":"ContainerDied","Data":"1a3f7e84bf8a6c4ae069528325f3a479b8352322a67c9e656bb09870a3e397c4"} Oct 31 12:55:19.873743 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:19.873696 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5ccdc5599d-mz2k5" event={"ID":"34abc9d3-a4c7-4f9b-9efd-64e156b45fc1","Type":"ContainerDied","Data":"083a27bc8318bf03603afd98cca31cb883ef4732fdfbfd92298f412a873933c3"} Oct 31 12:55:19.873743 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:19.873720 5693 scope.go:122] "RemoveContainer" containerID="1a3f7e84bf8a6c4ae069528325f3a479b8352322a67c9e656bb09870a3e397c4" Oct 31 12:55:19.911742 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:19.911719 5693 scope.go:122] "RemoveContainer" containerID="1a3f7e84bf8a6c4ae069528325f3a479b8352322a67c9e656bb09870a3e397c4" Oct 31 12:55:19.912152 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:55:19.912130 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a3f7e84bf8a6c4ae069528325f3a479b8352322a67c9e656bb09870a3e397c4\": container with ID starting with 1a3f7e84bf8a6c4ae069528325f3a479b8352322a67c9e656bb09870a3e397c4 not found: ID does not exist" containerID="1a3f7e84bf8a6c4ae069528325f3a479b8352322a67c9e656bb09870a3e397c4" Oct 31 12:55:19.912249 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:19.912159 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a3f7e84bf8a6c4ae069528325f3a479b8352322a67c9e656bb09870a3e397c4"} err="failed to get container status \"1a3f7e84bf8a6c4ae069528325f3a479b8352322a67c9e656bb09870a3e397c4\": rpc error: code = NotFound desc = could not find container \"1a3f7e84bf8a6c4ae069528325f3a479b8352322a67c9e656bb09870a3e397c4\": container with ID starting with 1a3f7e84bf8a6c4ae069528325f3a479b8352322a67c9e656bb09870a3e397c4 not found: ID does not exist" Oct 31 12:55:19.936791 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:19.936763 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34abc9d3-a4c7-4f9b-9efd-64e156b45fc1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34abc9d3-a4c7-4f9b-9efd-64e156b45fc1-routing-overlay-volume\") pod \"34abc9d3-a4c7-4f9b-9efd-64e156b45fc1\" (UID: \"34abc9d3-a4c7-4f9b-9efd-64e156b45fc1\") " Oct 31 12:55:19.936939 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:19.936921 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34abc9d3-a4c7-4f9b-9efd-64e156b45fc1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34abc9d3-a4c7-4f9b-9efd-64e156b45fc1-plugins-config-volume\") pod \"34abc9d3-a4c7-4f9b-9efd-64e156b45fc1\" (UID: \"34abc9d3-a4c7-4f9b-9efd-64e156b45fc1\") " Oct 31 12:55:19.936985 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:19.936976 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34abc9d3-a4c7-4f9b-9efd-64e156b45fc1-kube-api-access-nj97v\" (UniqueName: \"kubernetes.io/projected/34abc9d3-a4c7-4f9b-9efd-64e156b45fc1-kube-api-access-nj97v\") pod \"34abc9d3-a4c7-4f9b-9efd-64e156b45fc1\" (UID: \"34abc9d3-a4c7-4f9b-9efd-64e156b45fc1\") " Oct 31 12:55:19.937017 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:19.937005 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34abc9d3-a4c7-4f9b-9efd-64e156b45fc1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/34abc9d3-a4c7-4f9b-9efd-64e156b45fc1-provider-credentials\") pod \"34abc9d3-a4c7-4f9b-9efd-64e156b45fc1\" (UID: \"34abc9d3-a4c7-4f9b-9efd-64e156b45fc1\") " Oct 31 12:55:19.937140 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:19.937110 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34abc9d3-a4c7-4f9b-9efd-64e156b45fc1-routing-overlay-volume" pod "34abc9d3-a4c7-4f9b-9efd-64e156b45fc1" (UID: "34abc9d3-a4c7-4f9b-9efd-64e156b45fc1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:55:19.937268 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:19.937248 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34abc9d3-a4c7-4f9b-9efd-64e156b45fc1-plugins-config-volume" pod "34abc9d3-a4c7-4f9b-9efd-64e156b45fc1" (UID: "34abc9d3-a4c7-4f9b-9efd-64e156b45fc1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:55:19.937363 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:19.937350 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34abc9d3-a4c7-4f9b-9efd-64e156b45fc1-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:19.937418 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:19.937369 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34abc9d3-a4c7-4f9b-9efd-64e156b45fc1-routing-overlay-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:19.939183 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:19.939161 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34abc9d3-a4c7-4f9b-9efd-64e156b45fc1-kube-api-access-nj97v" pod "34abc9d3-a4c7-4f9b-9efd-64e156b45fc1" (UID: "34abc9d3-a4c7-4f9b-9efd-64e156b45fc1"). InnerVolumeSpecName "kube-api-access-nj97v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:55:19.939419 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:19.939390 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34abc9d3-a4c7-4f9b-9efd-64e156b45fc1-provider-credentials" pod "34abc9d3-a4c7-4f9b-9efd-64e156b45fc1" (UID: "34abc9d3-a4c7-4f9b-9efd-64e156b45fc1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:55:20.038253 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:20.038162 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj97v\" (UniqueName: \"kubernetes.io/projected/34abc9d3-a4c7-4f9b-9efd-64e156b45fc1-kube-api-access-nj97v\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:20.038253 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:20.038196 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/34abc9d3-a4c7-4f9b-9efd-64e156b45fc1-provider-credentials\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:20.096666 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:20.093069 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 12:55:20.096666 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:20.093135 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" podUID="52b4bc93-af99-4c3e-bddb-c087f6c0247c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 12:55:20.220531 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:20.220499 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5ccdc5599d-mz2k5"] Oct 31 12:55:20.224359 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:20.224326 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5ccdc5599d-mz2k5"] Oct 31 12:55:20.352542 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:20.352469 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt"] Oct 31 12:55:20.353035 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:20.353003 5693 cpu_manager.go:414] "containerMap: removing container" podUID="34abc9d3-a4c7-4f9b-9efd-64e156b45fc1" containerName="payload-processing" Oct 31 12:55:20.353035 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:20.353022 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34abc9d3-a4c7-4f9b-9efd-64e156b45fc1" containerName="payload-processing" Oct 31 12:55:20.353243 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:20.353168 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="34abc9d3-a4c7-4f9b-9efd-64e156b45fc1" containerName="payload-processing" Oct 31 12:55:20.357705 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:20.357683 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" Oct 31 12:55:20.367001 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:20.366975 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-3dd1e5\"/\"distinct-2-w2-3dd1e5-kserve-self-signed-certs\"" Oct 31 12:55:20.367835 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:20.367813 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt"] Oct 31 12:55:20.544275 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:20.544240 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff9a9909-6fc3-475b-a9fe-562ec150c76a-tls-certs\") pod \"distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt\" (UID: \"ff9a9909-6fc3-475b-a9fe-562ec150c76a\") " pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" Oct 31 12:55:20.544454 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:20.544306 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff9a9909-6fc3-475b-a9fe-562ec150c76a-model-cache\") pod \"distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt\" (UID: \"ff9a9909-6fc3-475b-a9fe-562ec150c76a\") " pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" Oct 31 12:55:20.544454 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:20.544372 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff9a9909-6fc3-475b-a9fe-562ec150c76a-home\") pod \"distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt\" (UID: \"ff9a9909-6fc3-475b-a9fe-562ec150c76a\") " pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" Oct 31 12:55:20.544534 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:20.544485 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff9a9909-6fc3-475b-a9fe-562ec150c76a-tmp-dir\") pod \"distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt\" (UID: \"ff9a9909-6fc3-475b-a9fe-562ec150c76a\") " pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" Oct 31 12:55:20.544578 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:20.544532 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff9a9909-6fc3-475b-a9fe-562ec150c76a-dshm\") pod \"distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt\" (UID: \"ff9a9909-6fc3-475b-a9fe-562ec150c76a\") " pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" Oct 31 12:55:20.544616 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:20.544581 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wptc9\" (UniqueName: \"kubernetes.io/projected/ff9a9909-6fc3-475b-a9fe-562ec150c76a-kube-api-access-wptc9\") pod \"distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt\" (UID: \"ff9a9909-6fc3-475b-a9fe-562ec150c76a\") " pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" Oct 31 12:55:20.645205 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:20.645174 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff9a9909-6fc3-475b-a9fe-562ec150c76a-home\") pod \"distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt\" (UID: \"ff9a9909-6fc3-475b-a9fe-562ec150c76a\") " pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" Oct 31 12:55:20.645392 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:20.645237 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff9a9909-6fc3-475b-a9fe-562ec150c76a-tmp-dir\") pod \"distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt\" (UID: \"ff9a9909-6fc3-475b-a9fe-562ec150c76a\") " pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" Oct 31 12:55:20.645392 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:20.645272 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff9a9909-6fc3-475b-a9fe-562ec150c76a-dshm\") pod \"distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt\" (UID: \"ff9a9909-6fc3-475b-a9fe-562ec150c76a\") " pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" Oct 31 12:55:20.645392 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:20.645293 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wptc9\" (UniqueName: \"kubernetes.io/projected/ff9a9909-6fc3-475b-a9fe-562ec150c76a-kube-api-access-wptc9\") pod \"distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt\" (UID: \"ff9a9909-6fc3-475b-a9fe-562ec150c76a\") " pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" Oct 31 12:55:20.645392 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:20.645388 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff9a9909-6fc3-475b-a9fe-562ec150c76a-tls-certs\") pod \"distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt\" (UID: \"ff9a9909-6fc3-475b-a9fe-562ec150c76a\") " pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" Oct 31 12:55:20.645626 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:20.645439 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff9a9909-6fc3-475b-a9fe-562ec150c76a-model-cache\") pod \"distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt\" (UID: \"ff9a9909-6fc3-475b-a9fe-562ec150c76a\") " pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" Oct 31 12:55:20.645626 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:20.645574 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff9a9909-6fc3-475b-a9fe-562ec150c76a-home\") pod \"distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt\" (UID: \"ff9a9909-6fc3-475b-a9fe-562ec150c76a\") " pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" Oct 31 12:55:20.645786 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:20.645708 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff9a9909-6fc3-475b-a9fe-562ec150c76a-tmp-dir\") pod \"distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt\" (UID: \"ff9a9909-6fc3-475b-a9fe-562ec150c76a\") " pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" Oct 31 12:55:20.645837 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:20.645821 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff9a9909-6fc3-475b-a9fe-562ec150c76a-model-cache\") pod \"distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt\" (UID: \"ff9a9909-6fc3-475b-a9fe-562ec150c76a\") " pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" Oct 31 12:55:20.647747 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:20.647727 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff9a9909-6fc3-475b-a9fe-562ec150c76a-dshm\") pod \"distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt\" (UID: \"ff9a9909-6fc3-475b-a9fe-562ec150c76a\") " pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" Oct 31 12:55:20.648053 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:20.648034 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff9a9909-6fc3-475b-a9fe-562ec150c76a-tls-certs\") pod \"distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt\" (UID: \"ff9a9909-6fc3-475b-a9fe-562ec150c76a\") " pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" Oct 31 12:55:20.657784 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:20.657759 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wptc9\" (UniqueName: \"kubernetes.io/projected/ff9a9909-6fc3-475b-a9fe-562ec150c76a-kube-api-access-wptc9\") pod \"distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt\" (UID: \"ff9a9909-6fc3-475b-a9fe-562ec150c76a\") " pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" Oct 31 12:55:20.671202 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:20.671175 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" Oct 31 12:55:21.003131 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:20.989622 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt"] Oct 31 12:55:21.006375 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:55:21.006334 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff9a9909_6fc3_475b_a9fe_562ec150c76a.slice/crio-e0350c6e7ea27c82c582d71851551ce39d4d62adcb3162615022e80ec2aff666 WatchSource:0}: Error finding container e0350c6e7ea27c82c582d71851551ce39d4d62adcb3162615022e80ec2aff666: Status 404 returned error can't find the container with id e0350c6e7ea27c82c582d71851551ce39d4d62adcb3162615022e80ec2aff666 Oct 31 12:55:21.092667 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:21.091055 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 12:55:21.092667 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:21.091110 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" podUID="52b4bc93-af99-4c3e-bddb-c087f6c0247c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 12:55:21.408715 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:21.408683 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34abc9d3-a4c7-4f9b-9efd-64e156b45fc1" path="/var/lib/kubelet/pods/34abc9d3-a4c7-4f9b-9efd-64e156b45fc1/volumes" Oct 31 12:55:21.910619 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:21.910584 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" event={"ID":"ff9a9909-6fc3-475b-a9fe-562ec150c76a","Type":"ContainerStarted","Data":"f1cc3f00414a45bb1a71fb9d0d22a83d2534cdc0a46a25a58ff116ddfbd9988c"} Oct 31 12:55:21.910831 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:21.910629 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" event={"ID":"ff9a9909-6fc3-475b-a9fe-562ec150c76a","Type":"ContainerStarted","Data":"e0350c6e7ea27c82c582d71851551ce39d4d62adcb3162615022e80ec2aff666"} Oct 31 12:55:21.934761 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:21.934713 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" podStartSLOduration=1.934698174 podStartE2EDuration="1.934698174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:55:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:55:21.93104113 +0000 UTC m=+1372.638627444" watchObservedRunningTime="2026-10-31 12:55:21.934698174 +0000 UTC m=+1372.642284468" Oct 31 12:55:21.946718 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:21.946685 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg"] Oct 31 12:55:21.951271 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:21.951245 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" Oct 31 12:55:21.961795 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:21.961772 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-7ae9e7-openshift-default-dockercfg-pplr5\"" Oct 31 12:55:21.964457 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:21.964415 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg"] Oct 31 12:55:22.061331 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:22.061292 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh2kd\" (UniqueName: \"kubernetes.io/projected/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-kube-api-access-hh2kd\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg\" (UID: \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" Oct 31 12:55:22.061815 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:22.061350 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg\" (UID: \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" Oct 31 12:55:22.061815 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:22.061459 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg\" (UID: \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" Oct 31 12:55:22.061815 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:22.061539 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg\" (UID: \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" Oct 31 12:55:22.061815 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:22.061567 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg\" (UID: \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" Oct 31 12:55:22.061815 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:22.061726 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg\" (UID: \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" Oct 31 12:55:22.061994 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:22.061889 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg\" (UID: \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" Oct 31 12:55:22.062066 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:22.062041 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg\" (UID: \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" Oct 31 12:55:22.062213 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:22.062165 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg\" (UID: \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" Oct 31 12:55:22.088179 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:22.088144 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 12:55:22.088379 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:22.088211 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" podUID="52b4bc93-af99-4c3e-bddb-c087f6c0247c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 12:55:22.163626 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:22.163519 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg\" (UID: \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" Oct 31 12:55:22.163626 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:22.163601 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg\" (UID: \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" Oct 31 12:55:22.163913 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:22.163735 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg\" (UID: \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" Oct 31 12:55:22.163913 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:22.163773 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh2kd\" (UniqueName: \"kubernetes.io/projected/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-kube-api-access-hh2kd\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg\" (UID: \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" Oct 31 12:55:22.163913 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:22.163803 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg\" (UID: \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" Oct 31 12:55:22.163913 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:22.163842 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg\" (UID: \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" Oct 31 12:55:22.163913 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:22.163896 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg\" (UID: \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" Oct 31 12:55:22.164164 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:22.163921 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg\" (UID: \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" Oct 31 12:55:22.164164 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:22.164001 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg\" (UID: \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" Oct 31 12:55:22.164164 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:22.164124 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg\" (UID: \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" Oct 31 12:55:22.164550 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:22.164525 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg\" (UID: \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" Oct 31 12:55:22.164904 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:22.164880 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg\" (UID: \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" Oct 31 12:55:22.165283 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:22.164989 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg\" (UID: \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" Oct 31 12:55:22.165283 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:22.165240 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg\" (UID: \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" Oct 31 12:55:22.166243 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:22.166219 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg\" (UID: \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" Oct 31 12:55:22.166606 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:22.166578 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg\" (UID: \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" Oct 31 12:55:22.176522 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:22.176470 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg\" (UID: \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" Oct 31 12:55:22.176758 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:22.176734 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh2kd\" (UniqueName: \"kubernetes.io/projected/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-kube-api-access-hh2kd\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg\" (UID: \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" Oct 31 12:55:22.267302 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:22.267269 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" Oct 31 12:55:22.565475 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:22.564027 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg"] Oct 31 12:55:22.567385 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:55:22.567350 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c36f54_5f25_45ea_b887_c0d2e0f53cfa.slice/crio-7b6a91b171c5591863e17993b19ae50df2f286eb6d4f8815a28168e2a4e3b223 WatchSource:0}: Error finding container 7b6a91b171c5591863e17993b19ae50df2f286eb6d4f8815a28168e2a4e3b223: Status 404 returned error can't find the container with id 7b6a91b171c5591863e17993b19ae50df2f286eb6d4f8815a28168e2a4e3b223 Oct 31 12:55:22.570687 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:22.570152 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:55:22.570687 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:22.570260 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:55:22.570687 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:22.570313 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:55:22.671408 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:22.671369 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" Oct 31 12:55:22.920030 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:22.919989 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" event={"ID":"69c36f54-5f25-45ea-b887-c0d2e0f53cfa","Type":"ContainerStarted","Data":"9bf4e45d90ab2d1b7557d892269a9fde0dac279fb391ae18035bc27f3743bb2c"} Oct 31 12:55:22.920217 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:22.920095 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" event={"ID":"69c36f54-5f25-45ea-b887-c0d2e0f53cfa","Type":"ContainerStarted","Data":"7b6a91b171c5591863e17993b19ae50df2f286eb6d4f8815a28168e2a4e3b223"} Oct 31 12:55:22.944479 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:22.944411 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" podStartSLOduration=1.944393984 podStartE2EDuration="1.944393984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:55:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:55:22.94256867 +0000 UTC m=+1373.650154966" watchObservedRunningTime="2026-10-31 12:55:22.944393984 +0000 UTC m=+1373.651980280" Oct 31 12:55:23.102419 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:23.102368 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" Oct 31 12:55:23.103812 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:23.103785 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" Oct 31 12:55:23.104673 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:23.104597 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" Oct 31 12:55:23.267601 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:23.267524 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" Oct 31 12:55:23.269068 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:23.269042 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 12:55:23.269237 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:23.269113 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" podUID="69c36f54-5f25-45ea-b887-c0d2e0f53cfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 12:55:24.268016 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:24.267984 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 12:55:24.268416 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:24.268049 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" podUID="69c36f54-5f25-45ea-b887-c0d2e0f53cfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 12:55:25.268612 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:25.268578 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 12:55:25.269031 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:25.268662 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" podUID="69c36f54-5f25-45ea-b887-c0d2e0f53cfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 12:55:25.320907 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:25.320874 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6zvmg"] Oct 31 12:55:25.325039 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:25.325018 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6zvmg" Oct 31 12:55:25.339351 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:25.339327 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 12:55:25.339476 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:25.339364 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 12:55:25.340349 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:25.340319 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 12:55:25.340451 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:25.340386 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xtsw2\"" Oct 31 12:55:25.343479 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:25.343460 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6zvmg"] Oct 31 12:55:25.441327 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:25.441290 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f02c2b48-73f3-45a7-a329-1b797c681c6a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6zvmg\" (UID: \"f02c2b48-73f3-45a7-a329-1b797c681c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6zvmg" Oct 31 12:55:25.441327 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:25.441327 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f02c2b48-73f3-45a7-a329-1b797c681c6a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6zvmg\" (UID: \"f02c2b48-73f3-45a7-a329-1b797c681c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6zvmg" Oct 31 12:55:25.441521 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:25.441426 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f02c2b48-73f3-45a7-a329-1b797c681c6a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6zvmg\" (UID: \"f02c2b48-73f3-45a7-a329-1b797c681c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6zvmg" Oct 31 12:55:25.441521 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:25.441459 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f02c2b48-73f3-45a7-a329-1b797c681c6a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6zvmg\" (UID: \"f02c2b48-73f3-45a7-a329-1b797c681c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6zvmg" Oct 31 12:55:25.441521 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:25.441482 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gckxb\" (UniqueName: \"kubernetes.io/projected/f02c2b48-73f3-45a7-a329-1b797c681c6a-kube-api-access-gckxb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6zvmg\" (UID: \"f02c2b48-73f3-45a7-a329-1b797c681c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6zvmg" Oct 31 12:55:25.441521 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:25.441503 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f02c2b48-73f3-45a7-a329-1b797c681c6a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6zvmg\" (UID: \"f02c2b48-73f3-45a7-a329-1b797c681c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6zvmg" Oct 31 12:55:25.441721 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:25.441696 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f02c2b48-73f3-45a7-a329-1b797c681c6a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6zvmg\" (UID: \"f02c2b48-73f3-45a7-a329-1b797c681c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6zvmg" Oct 31 12:55:25.543330 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:25.543245 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f02c2b48-73f3-45a7-a329-1b797c681c6a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6zvmg\" (UID: \"f02c2b48-73f3-45a7-a329-1b797c681c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6zvmg" Oct 31 12:55:25.543330 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:25.543301 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f02c2b48-73f3-45a7-a329-1b797c681c6a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6zvmg\" (UID: \"f02c2b48-73f3-45a7-a329-1b797c681c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6zvmg" Oct 31 12:55:25.543517 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:25.543342 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f02c2b48-73f3-45a7-a329-1b797c681c6a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6zvmg\" (UID: \"f02c2b48-73f3-45a7-a329-1b797c681c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6zvmg" Oct 31 12:55:25.543517 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:25.543405 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f02c2b48-73f3-45a7-a329-1b797c681c6a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6zvmg\" (UID: \"f02c2b48-73f3-45a7-a329-1b797c681c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6zvmg" Oct 31 12:55:25.543517 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:25.543434 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f02c2b48-73f3-45a7-a329-1b797c681c6a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6zvmg\" (UID: \"f02c2b48-73f3-45a7-a329-1b797c681c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6zvmg" Oct 31 12:55:25.543517 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:25.543470 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gckxb\" (UniqueName: \"kubernetes.io/projected/f02c2b48-73f3-45a7-a329-1b797c681c6a-kube-api-access-gckxb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6zvmg\" (UID: \"f02c2b48-73f3-45a7-a329-1b797c681c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6zvmg" Oct 31 12:55:25.543517 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:25.543499 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f02c2b48-73f3-45a7-a329-1b797c681c6a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6zvmg\" (UID: \"f02c2b48-73f3-45a7-a329-1b797c681c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6zvmg" Oct 31 12:55:25.544285 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:25.544254 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f02c2b48-73f3-45a7-a329-1b797c681c6a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6zvmg\" (UID: \"f02c2b48-73f3-45a7-a329-1b797c681c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6zvmg" Oct 31 12:55:25.546053 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:25.546030 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f02c2b48-73f3-45a7-a329-1b797c681c6a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6zvmg\" (UID: \"f02c2b48-73f3-45a7-a329-1b797c681c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6zvmg" Oct 31 12:55:25.546142 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:25.546050 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f02c2b48-73f3-45a7-a329-1b797c681c6a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6zvmg\" (UID: \"f02c2b48-73f3-45a7-a329-1b797c681c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6zvmg" Oct 31 12:55:25.546142 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:25.546048 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f02c2b48-73f3-45a7-a329-1b797c681c6a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6zvmg\" (UID: \"f02c2b48-73f3-45a7-a329-1b797c681c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6zvmg" Oct 31 12:55:25.546310 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:25.546295 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f02c2b48-73f3-45a7-a329-1b797c681c6a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6zvmg\" (UID: \"f02c2b48-73f3-45a7-a329-1b797c681c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6zvmg" Oct 31 12:55:25.557248 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:25.557223 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f02c2b48-73f3-45a7-a329-1b797c681c6a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6zvmg\" (UID: \"f02c2b48-73f3-45a7-a329-1b797c681c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6zvmg" Oct 31 12:55:25.558439 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:25.558417 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gckxb\" (UniqueName: \"kubernetes.io/projected/f02c2b48-73f3-45a7-a329-1b797c681c6a-kube-api-access-gckxb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6zvmg\" (UID: \"f02c2b48-73f3-45a7-a329-1b797c681c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6zvmg" Oct 31 12:55:25.638026 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:25.637994 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6zvmg" Oct 31 12:55:25.957702 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:55:25.957670 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf02c2b48_73f3_45a7_a329_1b797c681c6a.slice/crio-cb14d8e8b04a819764d232f011b341537ff7c614c0d15bcd0f0bfb8eaa8f7d34 WatchSource:0}: Error finding container cb14d8e8b04a819764d232f011b341537ff7c614c0d15bcd0f0bfb8eaa8f7d34: Status 404 returned error can't find the container with id cb14d8e8b04a819764d232f011b341537ff7c614c0d15bcd0f0bfb8eaa8f7d34 Oct 31 12:55:25.959859 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:25.959832 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6zvmg"] Oct 31 12:55:25.959944 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:25.959910 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:55:25.959989 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:25.959970 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:55:26.269609 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:26.269516 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 12:55:26.269609 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:26.269578 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" podUID="69c36f54-5f25-45ea-b887-c0d2e0f53cfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 12:55:26.947203 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:26.947169 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6zvmg" event={"ID":"f02c2b48-73f3-45a7-a329-1b797c681c6a","Type":"ContainerStarted","Data":"bc5f9faf79375e3c242a937d298938cb09804a3447f07697a19edf70fb0fc59b"} Oct 31 12:55:26.947354 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:26.947233 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6zvmg" event={"ID":"f02c2b48-73f3-45a7-a329-1b797c681c6a","Type":"ContainerStarted","Data":"cb14d8e8b04a819764d232f011b341537ff7c614c0d15bcd0f0bfb8eaa8f7d34"} Oct 31 12:55:26.948042 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:26.948020 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6zvmg" Oct 31 12:55:26.996683 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:26.996603 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6zvmg" podStartSLOduration=1.99657221 podStartE2EDuration="1.99657221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:55:26.974698329 +0000 UTC m=+1377.682284620" watchObservedRunningTime="2026-10-31 12:55:26.99657221 +0000 UTC m=+1377.704158505" Oct 31 12:55:27.267721 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:27.267607 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 12:55:27.267721 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:27.267710 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" podUID="69c36f54-5f25-45ea-b887-c0d2e0f53cfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 12:55:27.957308 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:27.957267 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6zvmg" Oct 31 12:55:28.268008 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:28.267923 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 12:55:28.268008 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:28.267981 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" podUID="69c36f54-5f25-45ea-b887-c0d2e0f53cfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 12:55:29.267910 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:29.267867 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 12:55:29.268347 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:29.267925 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" podUID="69c36f54-5f25-45ea-b887-c0d2e0f53cfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 12:55:30.272311 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:30.272279 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" Oct 31 12:55:30.272723 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:30.272684 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" Oct 31 12:55:30.273458 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:30.273440 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" Oct 31 12:55:30.677665 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:30.673455 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" Oct 31 12:55:30.706002 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:30.705973 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" Oct 31 12:55:30.984706 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:30.984620 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" Oct 31 12:55:32.146425 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:32.146391 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp"] Oct 31 12:55:36.993830 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:36.993798 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp"] Oct 31 12:55:37.027331 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:37.027292 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" Oct 31 12:55:37.029943 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:37.029907 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp"] Oct 31 12:55:37.132167 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:37.132132 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c1be59-5195-45f3-a004-820466d70099-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp\" (UID: \"95c1be59-5195-45f3-a004-820466d70099\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" Oct 31 12:55:37.132312 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:37.132192 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95c1be59-5195-45f3-a004-820466d70099-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp\" (UID: \"95c1be59-5195-45f3-a004-820466d70099\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" Oct 31 12:55:37.132312 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:37.132264 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c1be59-5195-45f3-a004-820466d70099-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp\" (UID: \"95c1be59-5195-45f3-a004-820466d70099\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" Oct 31 12:55:37.132312 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:37.132305 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95c1be59-5195-45f3-a004-820466d70099-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp\" (UID: \"95c1be59-5195-45f3-a004-820466d70099\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" Oct 31 12:55:37.132447 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:37.132331 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95c1be59-5195-45f3-a004-820466d70099-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp\" (UID: \"95c1be59-5195-45f3-a004-820466d70099\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" Oct 31 12:55:37.132447 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:37.132421 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5977\" (UniqueName: \"kubernetes.io/projected/95c1be59-5195-45f3-a004-820466d70099-kube-api-access-j5977\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp\" (UID: \"95c1be59-5195-45f3-a004-820466d70099\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" Oct 31 12:55:37.132509 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:37.132453 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95c1be59-5195-45f3-a004-820466d70099-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp\" (UID: \"95c1be59-5195-45f3-a004-820466d70099\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" Oct 31 12:55:37.132509 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:37.132483 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95c1be59-5195-45f3-a004-820466d70099-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp\" (UID: \"95c1be59-5195-45f3-a004-820466d70099\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" Oct 31 12:55:37.132619 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:37.132601 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95c1be59-5195-45f3-a004-820466d70099-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp\" (UID: \"95c1be59-5195-45f3-a004-820466d70099\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" Oct 31 12:55:37.233543 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:37.233497 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c1be59-5195-45f3-a004-820466d70099-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp\" (UID: \"95c1be59-5195-45f3-a004-820466d70099\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" Oct 31 12:55:37.233774 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:37.233564 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95c1be59-5195-45f3-a004-820466d70099-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp\" (UID: \"95c1be59-5195-45f3-a004-820466d70099\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" Oct 31 12:55:37.233774 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:37.233593 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c1be59-5195-45f3-a004-820466d70099-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp\" (UID: \"95c1be59-5195-45f3-a004-820466d70099\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" Oct 31 12:55:37.233774 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:37.233666 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95c1be59-5195-45f3-a004-820466d70099-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp\" (UID: \"95c1be59-5195-45f3-a004-820466d70099\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" Oct 31 12:55:37.233962 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:37.233799 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95c1be59-5195-45f3-a004-820466d70099-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp\" (UID: \"95c1be59-5195-45f3-a004-820466d70099\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" Oct 31 12:55:37.233962 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:37.233863 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5977\" (UniqueName: \"kubernetes.io/projected/95c1be59-5195-45f3-a004-820466d70099-kube-api-access-j5977\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp\" (UID: \"95c1be59-5195-45f3-a004-820466d70099\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" Oct 31 12:55:37.233962 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:37.233912 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95c1be59-5195-45f3-a004-820466d70099-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp\" (UID: \"95c1be59-5195-45f3-a004-820466d70099\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" Oct 31 12:55:37.234108 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:37.233962 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95c1be59-5195-45f3-a004-820466d70099-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp\" (UID: \"95c1be59-5195-45f3-a004-820466d70099\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" Oct 31 12:55:37.234108 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:37.233999 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c1be59-5195-45f3-a004-820466d70099-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp\" (UID: \"95c1be59-5195-45f3-a004-820466d70099\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" Oct 31 12:55:37.234108 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:37.234040 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95c1be59-5195-45f3-a004-820466d70099-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp\" (UID: \"95c1be59-5195-45f3-a004-820466d70099\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" Oct 31 12:55:37.234108 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:37.234065 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c1be59-5195-45f3-a004-820466d70099-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp\" (UID: \"95c1be59-5195-45f3-a004-820466d70099\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" Oct 31 12:55:37.234298 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:37.234265 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95c1be59-5195-45f3-a004-820466d70099-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp\" (UID: \"95c1be59-5195-45f3-a004-820466d70099\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" Oct 31 12:55:37.234451 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:37.234430 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95c1be59-5195-45f3-a004-820466d70099-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp\" (UID: \"95c1be59-5195-45f3-a004-820466d70099\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" Oct 31 12:55:37.234527 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:37.234507 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95c1be59-5195-45f3-a004-820466d70099-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp\" (UID: \"95c1be59-5195-45f3-a004-820466d70099\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" Oct 31 12:55:37.236089 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:37.236058 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95c1be59-5195-45f3-a004-820466d70099-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp\" (UID: \"95c1be59-5195-45f3-a004-820466d70099\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" Oct 31 12:55:37.236542 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:37.236523 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95c1be59-5195-45f3-a004-820466d70099-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp\" (UID: \"95c1be59-5195-45f3-a004-820466d70099\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" Oct 31 12:55:37.255014 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:37.254959 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95c1be59-5195-45f3-a004-820466d70099-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp\" (UID: \"95c1be59-5195-45f3-a004-820466d70099\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" Oct 31 12:55:37.256045 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:37.256018 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5977\" (UniqueName: \"kubernetes.io/projected/95c1be59-5195-45f3-a004-820466d70099-kube-api-access-j5977\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp\" (UID: \"95c1be59-5195-45f3-a004-820466d70099\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" Oct 31 12:55:37.344056 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:37.344019 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" Oct 31 12:55:37.630043 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:37.630014 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp"] Oct 31 12:55:37.631794 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:55:37.631772 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95c1be59_5195_45f3_a004_820466d70099.slice/crio-1a5dc0f3e42dbc4baf4359c5e011d1a7ce97af1163d88337f3496ad876ed6655 WatchSource:0}: Error finding container 1a5dc0f3e42dbc4baf4359c5e011d1a7ce97af1163d88337f3496ad876ed6655: Status 404 returned error can't find the container with id 1a5dc0f3e42dbc4baf4359c5e011d1a7ce97af1163d88337f3496ad876ed6655 Oct 31 12:55:37.634230 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:37.634195 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:55:37.634337 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:37.634261 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:55:37.634337 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:37.634288 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:55:38.038384 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:38.038350 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" event={"ID":"95c1be59-5195-45f3-a004-820466d70099","Type":"ContainerStarted","Data":"a9cadc7cbdc2479d2c4971964da2417c7a8e0249068f4a966f678047d2457e2f"} Oct 31 12:55:38.039035 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:38.039008 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" event={"ID":"95c1be59-5195-45f3-a004-820466d70099","Type":"ContainerStarted","Data":"1a5dc0f3e42dbc4baf4359c5e011d1a7ce97af1163d88337f3496ad876ed6655"} Oct 31 12:55:38.065154 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:38.065109 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" podStartSLOduration=2.06509229 podStartE2EDuration="2.06509229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:55:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:55:38.062695043 +0000 UTC m=+1388.770281340" watchObservedRunningTime="2026-10-31 12:55:38.06509229 +0000 UTC m=+1388.772678585" Oct 31 12:55:38.345219 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:38.345126 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" Oct 31 12:55:38.346803 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:38.346774 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 12:55:38.346956 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:38.346831 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" podUID="95c1be59-5195-45f3-a004-820466d70099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 12:55:39.345287 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:39.345249 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 12:55:39.345817 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:39.345302 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" podUID="95c1be59-5195-45f3-a004-820466d70099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 12:55:40.345033 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:40.344994 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 12:55:40.345251 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:40.345057 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" podUID="95c1be59-5195-45f3-a004-820466d70099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 12:55:41.349667 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:41.347731 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 12:55:41.349667 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:41.347806 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" podUID="95c1be59-5195-45f3-a004-820466d70099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 12:55:42.348666 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:42.348104 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 12:55:42.348666 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:42.348182 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" podUID="95c1be59-5195-45f3-a004-820466d70099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 12:55:42.568667 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:42.567730 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp" podUID="800da7e5-09e1-4844-a967-13258ecb47cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:42.644728 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:42.643801 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" podUID="e5a40b79-9879-4981-ab9f-446e1750a013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:42.706793 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:42.706747 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" podUID="710a52fa-2bc4-4bb9-96be-4b08dc60de67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:42.736573 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:42.736517 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" podUID="17d56bfd-8cb1-42bf-bb11-05810c46d469" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:42.788205 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:42.788153 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" podUID="8fb66a94-c399-43fb-945c-610ca63f0d76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:42.792495 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:42.792455 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" podUID="66c6ec5a-c0da-4bd7-a314-0c90ccec2425" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:42.869667 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:42.867801 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" podUID="1a2abd3f-fb4f-4990-9a32-5f945a588068" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:42.985531 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:42.984076 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" podUID="ff9a9909-6fc3-475b-a9fe-562ec150c76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:43.344718 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:43.344523 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 12:55:43.344718 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:43.344582 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" podUID="95c1be59-5195-45f3-a004-820466d70099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 12:55:43.568784 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:43.568733 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp" podUID="800da7e5-09e1-4844-a967-13258ecb47cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:43.570114 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:43.570091 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp" Oct 31 12:55:43.646861 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:43.646816 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" podUID="e5a40b79-9879-4981-ab9f-446e1750a013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:43.647721 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:43.647697 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" Oct 31 12:55:43.738673 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:43.738264 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" podUID="17d56bfd-8cb1-42bf-bb11-05810c46d469" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:43.765664 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:43.739395 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" Oct 31 12:55:43.811779 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:43.811731 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" podUID="8fb66a94-c399-43fb-945c-610ca63f0d76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:43.846667 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:43.816097 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" Oct 31 12:55:43.853668 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:43.852732 5693 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" podUID="8fb66a94-c399-43fb-945c-610ca63f0d76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:43.853668 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:43.852941 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" podUID="66c6ec5a-c0da-4bd7-a314-0c90ccec2425" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:43.853668 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:43.853088 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" Oct 31 12:55:43.868866 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:43.868825 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" podUID="1a2abd3f-fb4f-4990-9a32-5f945a588068" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:43.869504 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:43.869484 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" Oct 31 12:55:43.910725 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:43.893711 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" podUID="f2b3766f-058c-40e7-9a2d-a7f5028e2b1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Oct 31 12:55:44.070879 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:44.070818 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" podUID="ff9a9909-6fc3-475b-a9fe-562ec150c76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:44.071602 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:44.071582 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" Oct 31 12:55:44.383887 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:44.383790 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 12:55:44.384961 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:44.384926 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" podUID="95c1be59-5195-45f3-a004-820466d70099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 12:55:44.471960 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:44.470514 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" Oct 31 12:55:44.575686 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:44.574439 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp" podUID="800da7e5-09e1-4844-a967-13258ecb47cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:44.666128 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:44.666084 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" podUID="e5a40b79-9879-4981-ab9f-446e1750a013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:44.697680 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:44.680842 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" Oct 31 12:55:44.697680 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:44.696719 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp" Oct 31 12:55:44.735407 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:44.735378 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" Oct 31 12:55:44.738813 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:44.738781 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" podUID="17d56bfd-8cb1-42bf-bb11-05810c46d469" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 12:55:44.746299 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:44.744139 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" podUID="710a52fa-2bc4-4bb9-96be-4b08dc60de67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:44.770040 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:44.770012 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" Oct 31 12:55:44.778482 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:44.778459 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" Oct 31 12:55:44.792786 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:44.792759 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" Oct 31 12:55:45.345001 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:45.344960 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 12:55:45.345204 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:45.345020 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" podUID="95c1be59-5195-45f3-a004-820466d70099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 12:55:47.168085 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:47.168046 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp" podUID="800da7e5-09e1-4844-a967-13258ecb47cc" containerName="main" containerID="cri-o://d703330527ce9e48594bff3449a2d2d62695aee086fb88e35de0cadd8e49f424" gracePeriod=45 Oct 31 12:55:47.343942 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:47.343902 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" Oct 31 12:55:47.344393 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:47.344366 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" Oct 31 12:55:47.345207 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:47.345187 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" Oct 31 12:55:47.379297 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:47.379265 5693 generic.go:356] "Generic (PLEG): container finished" podID="800da7e5-09e1-4844-a967-13258ecb47cc" containerID="d703330527ce9e48594bff3449a2d2d62695aee086fb88e35de0cadd8e49f424" exitCode=0 Oct 31 12:55:47.379471 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:47.379333 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp" event={"ID":"800da7e5-09e1-4844-a967-13258ecb47cc","Type":"ContainerDied","Data":"d703330527ce9e48594bff3449a2d2d62695aee086fb88e35de0cadd8e49f424"} Oct 31 12:55:47.451669 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:47.451629 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp" Oct 31 12:55:47.545106 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:47.545067 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/800da7e5-09e1-4844-a967-13258ecb47cc-tls-certs\" (UniqueName: \"kubernetes.io/secret/800da7e5-09e1-4844-a967-13258ecb47cc-tls-certs\") pod \"800da7e5-09e1-4844-a967-13258ecb47cc\" (UID: \"800da7e5-09e1-4844-a967-13258ecb47cc\") " Oct 31 12:55:47.545320 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:47.545141 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/800da7e5-09e1-4844-a967-13258ecb47cc-home\" (UniqueName: \"kubernetes.io/empty-dir/800da7e5-09e1-4844-a967-13258ecb47cc-home\") pod \"800da7e5-09e1-4844-a967-13258ecb47cc\" (UID: \"800da7e5-09e1-4844-a967-13258ecb47cc\") " Oct 31 12:55:47.545320 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:47.545175 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/800da7e5-09e1-4844-a967-13258ecb47cc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/800da7e5-09e1-4844-a967-13258ecb47cc-model-cache\") pod \"800da7e5-09e1-4844-a967-13258ecb47cc\" (UID: \"800da7e5-09e1-4844-a967-13258ecb47cc\") " Oct 31 12:55:47.545320 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:47.545270 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/800da7e5-09e1-4844-a967-13258ecb47cc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/800da7e5-09e1-4844-a967-13258ecb47cc-tmp-dir\") pod \"800da7e5-09e1-4844-a967-13258ecb47cc\" (UID: \"800da7e5-09e1-4844-a967-13258ecb47cc\") " Oct 31 12:55:47.545320 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:47.545304 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/800da7e5-09e1-4844-a967-13258ecb47cc-kube-api-access-xhjwx\" (UniqueName: \"kubernetes.io/projected/800da7e5-09e1-4844-a967-13258ecb47cc-kube-api-access-xhjwx\") pod \"800da7e5-09e1-4844-a967-13258ecb47cc\" (UID: \"800da7e5-09e1-4844-a967-13258ecb47cc\") " Oct 31 12:55:47.545561 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:47.545376 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/800da7e5-09e1-4844-a967-13258ecb47cc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/800da7e5-09e1-4844-a967-13258ecb47cc-dshm\") pod \"800da7e5-09e1-4844-a967-13258ecb47cc\" (UID: \"800da7e5-09e1-4844-a967-13258ecb47cc\") " Oct 31 12:55:47.545561 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:47.545469 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/800da7e5-09e1-4844-a967-13258ecb47cc-home" pod "800da7e5-09e1-4844-a967-13258ecb47cc" (UID: "800da7e5-09e1-4844-a967-13258ecb47cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:47.545561 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:47.545529 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/800da7e5-09e1-4844-a967-13258ecb47cc-model-cache" pod "800da7e5-09e1-4844-a967-13258ecb47cc" (UID: "800da7e5-09e1-4844-a967-13258ecb47cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:47.545750 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:47.545674 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/800da7e5-09e1-4844-a967-13258ecb47cc-tmp-dir" pod "800da7e5-09e1-4844-a967-13258ecb47cc" (UID: "800da7e5-09e1-4844-a967-13258ecb47cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:47.546136 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:47.546111 5693 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/800da7e5-09e1-4844-a967-13258ecb47cc-home\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:47.546240 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:47.546139 5693 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/800da7e5-09e1-4844-a967-13258ecb47cc-model-cache\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:47.546240 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:47.546153 5693 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/800da7e5-09e1-4844-a967-13258ecb47cc-tmp-dir\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:47.548047 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:47.548015 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/800da7e5-09e1-4844-a967-13258ecb47cc-tls-certs" pod "800da7e5-09e1-4844-a967-13258ecb47cc" (UID: "800da7e5-09e1-4844-a967-13258ecb47cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:55:47.548178 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:47.548144 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/800da7e5-09e1-4844-a967-13258ecb47cc-dshm" pod "800da7e5-09e1-4844-a967-13258ecb47cc" (UID: "800da7e5-09e1-4844-a967-13258ecb47cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:47.548239 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:47.548216 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/800da7e5-09e1-4844-a967-13258ecb47cc-kube-api-access-xhjwx" pod "800da7e5-09e1-4844-a967-13258ecb47cc" (UID: "800da7e5-09e1-4844-a967-13258ecb47cc"). InnerVolumeSpecName "kube-api-access-xhjwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:55:47.647354 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:47.647321 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhjwx\" (UniqueName: \"kubernetes.io/projected/800da7e5-09e1-4844-a967-13258ecb47cc-kube-api-access-xhjwx\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:47.647354 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:47.647354 5693 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/800da7e5-09e1-4844-a967-13258ecb47cc-dshm\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:47.647560 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:47.647369 5693 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/800da7e5-09e1-4844-a967-13258ecb47cc-tls-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:48.386351 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:48.386321 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp" Oct 31 12:55:48.386866 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:48.386353 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp" event={"ID":"800da7e5-09e1-4844-a967-13258ecb47cc","Type":"ContainerDied","Data":"13dfbf21894342358b6e40ea1d12fc871d791c4ebeda572ff2cb918f032d1d67"} Oct 31 12:55:48.386866 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:48.386405 5693 scope.go:122] "RemoveContainer" containerID="d703330527ce9e48594bff3449a2d2d62695aee086fb88e35de0cadd8e49f424" Oct 31 12:55:48.414215 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:48.414182 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp"] Oct 31 12:55:48.418985 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:48.418956 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-a-2f2da815-kserve-64b7897bf6-dtxsp"] Oct 31 12:55:48.707169 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:55:48.707136 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:55:49.409174 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:49.409142 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="800da7e5-09e1-4844-a967-13258ecb47cc" path="/var/lib/kubelet/pods/800da7e5-09e1-4844-a967-13258ecb47cc/volumes" Oct 31 12:55:54.815671 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:54.782040 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" podUID="ff9a9909-6fc3-475b-a9fe-562ec150c76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:54.815671 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:54.783384 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" podUID="66c6ec5a-c0da-4bd7-a314-0c90ccec2425" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:54.815671 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:54.789726 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" podUID="17d56bfd-8cb1-42bf-bb11-05810c46d469" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:54.865666 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:54.865313 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" podUID="f2b3766f-058c-40e7-9a2d-a7f5028e2b1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:55.473793 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:55.473745 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" podUID="1a2abd3f-fb4f-4990-9a32-5f945a588068" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:55.686791 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:55.686738 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" podUID="e5a40b79-9879-4981-ab9f-446e1750a013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:55.726963 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:55.726866 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" podUID="8fb66a94-c399-43fb-945c-610ca63f0d76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:55.798668 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:55.797539 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" podUID="17d56bfd-8cb1-42bf-bb11-05810c46d469" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:55.800559 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:55.800522 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" podUID="66c6ec5a-c0da-4bd7-a314-0c90ccec2425" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 12:55:55.801623 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:55.801603 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" Oct 31 12:55:55.801832 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:55.801817 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" Oct 31 12:55:56.127771 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:56.127687 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" Oct 31 12:55:56.364006 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:56.363976 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" Oct 31 12:55:56.847666 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:56.846858 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-rskhv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:55:56.847666 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:56.846926 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" podUID="2ae02ff0-1e33-4007-a97a-63a61e8b4bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:57.049855 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:57.049790 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fv482 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:55:57.050173 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:55:57.050142 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" podUID="c0bdd69c-691b-47b2-ba17-2e0ac7185643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:07.030786 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:07.030754 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp"] Oct 31 12:56:07.031396 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:07.031376 5693 cpu_manager.go:414] "containerMap: removing container" podUID="800da7e5-09e1-4844-a967-13258ecb47cc" containerName="main" Oct 31 12:56:07.031396 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:07.031396 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="800da7e5-09e1-4844-a967-13258ecb47cc" containerName="main" Oct 31 12:56:07.031570 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:07.031491 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="800da7e5-09e1-4844-a967-13258ecb47cc" containerName="main" Oct 31 12:56:07.036838 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:07.036814 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" Oct 31 12:56:07.052458 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:07.052425 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp"] Oct 31 12:56:07.092755 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:07.092716 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4b60d0a-c847-439d-a156-d6111ffb40a7-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp\" (UID: \"a4b60d0a-c847-439d-a156-d6111ffb40a7\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" Oct 31 12:56:07.092755 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:07.092759 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4b60d0a-c847-439d-a156-d6111ffb40a7-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp\" (UID: \"a4b60d0a-c847-439d-a156-d6111ffb40a7\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" Oct 31 12:56:07.093218 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:07.092890 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b60d0a-c847-439d-a156-d6111ffb40a7-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp\" (UID: \"a4b60d0a-c847-439d-a156-d6111ffb40a7\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" Oct 31 12:56:07.093218 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:07.092949 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nq6w\" (UniqueName: \"kubernetes.io/projected/a4b60d0a-c847-439d-a156-d6111ffb40a7-kube-api-access-4nq6w\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp\" (UID: \"a4b60d0a-c847-439d-a156-d6111ffb40a7\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" Oct 31 12:56:07.093218 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:07.093025 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4b60d0a-c847-439d-a156-d6111ffb40a7-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp\" (UID: \"a4b60d0a-c847-439d-a156-d6111ffb40a7\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" Oct 31 12:56:07.093218 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:07.093073 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4b60d0a-c847-439d-a156-d6111ffb40a7-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp\" (UID: \"a4b60d0a-c847-439d-a156-d6111ffb40a7\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" Oct 31 12:56:07.093218 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:07.093157 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4b60d0a-c847-439d-a156-d6111ffb40a7-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp\" (UID: \"a4b60d0a-c847-439d-a156-d6111ffb40a7\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" Oct 31 12:56:07.093218 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:07.093189 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4b60d0a-c847-439d-a156-d6111ffb40a7-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp\" (UID: \"a4b60d0a-c847-439d-a156-d6111ffb40a7\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" Oct 31 12:56:07.093218 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:07.093214 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b60d0a-c847-439d-a156-d6111ffb40a7-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp\" (UID: \"a4b60d0a-c847-439d-a156-d6111ffb40a7\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" Oct 31 12:56:07.194240 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:07.194207 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b60d0a-c847-439d-a156-d6111ffb40a7-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp\" (UID: \"a4b60d0a-c847-439d-a156-d6111ffb40a7\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" Oct 31 12:56:07.194432 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:07.194269 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4b60d0a-c847-439d-a156-d6111ffb40a7-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp\" (UID: \"a4b60d0a-c847-439d-a156-d6111ffb40a7\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" Oct 31 12:56:07.194432 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:07.194302 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4b60d0a-c847-439d-a156-d6111ffb40a7-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp\" (UID: \"a4b60d0a-c847-439d-a156-d6111ffb40a7\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" Oct 31 12:56:07.194432 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:07.194381 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b60d0a-c847-439d-a156-d6111ffb40a7-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp\" (UID: \"a4b60d0a-c847-439d-a156-d6111ffb40a7\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" Oct 31 12:56:07.194611 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:07.194441 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nq6w\" (UniqueName: \"kubernetes.io/projected/a4b60d0a-c847-439d-a156-d6111ffb40a7-kube-api-access-4nq6w\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp\" (UID: \"a4b60d0a-c847-439d-a156-d6111ffb40a7\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" Oct 31 12:56:07.194611 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:07.194472 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4b60d0a-c847-439d-a156-d6111ffb40a7-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp\" (UID: \"a4b60d0a-c847-439d-a156-d6111ffb40a7\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" Oct 31 12:56:07.194611 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:07.194536 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4b60d0a-c847-439d-a156-d6111ffb40a7-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp\" (UID: \"a4b60d0a-c847-439d-a156-d6111ffb40a7\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" Oct 31 12:56:07.194812 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:07.194614 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4b60d0a-c847-439d-a156-d6111ffb40a7-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp\" (UID: \"a4b60d0a-c847-439d-a156-d6111ffb40a7\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" Oct 31 12:56:07.194812 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:07.194673 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b60d0a-c847-439d-a156-d6111ffb40a7-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp\" (UID: \"a4b60d0a-c847-439d-a156-d6111ffb40a7\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" Oct 31 12:56:07.194812 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:07.194687 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4b60d0a-c847-439d-a156-d6111ffb40a7-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp\" (UID: \"a4b60d0a-c847-439d-a156-d6111ffb40a7\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" Oct 31 12:56:07.195136 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:07.194970 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4b60d0a-c847-439d-a156-d6111ffb40a7-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp\" (UID: \"a4b60d0a-c847-439d-a156-d6111ffb40a7\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" Oct 31 12:56:07.195136 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:07.195012 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b60d0a-c847-439d-a156-d6111ffb40a7-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp\" (UID: \"a4b60d0a-c847-439d-a156-d6111ffb40a7\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" Oct 31 12:56:07.195136 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:07.195041 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4b60d0a-c847-439d-a156-d6111ffb40a7-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp\" (UID: \"a4b60d0a-c847-439d-a156-d6111ffb40a7\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" Oct 31 12:56:07.195136 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:07.195094 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4b60d0a-c847-439d-a156-d6111ffb40a7-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp\" (UID: \"a4b60d0a-c847-439d-a156-d6111ffb40a7\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" Oct 31 12:56:07.196748 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:07.196706 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4b60d0a-c847-439d-a156-d6111ffb40a7-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp\" (UID: \"a4b60d0a-c847-439d-a156-d6111ffb40a7\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" Oct 31 12:56:07.197139 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:07.197117 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4b60d0a-c847-439d-a156-d6111ffb40a7-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp\" (UID: \"a4b60d0a-c847-439d-a156-d6111ffb40a7\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" Oct 31 12:56:07.206528 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:07.206506 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nq6w\" (UniqueName: \"kubernetes.io/projected/a4b60d0a-c847-439d-a156-d6111ffb40a7-kube-api-access-4nq6w\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp\" (UID: \"a4b60d0a-c847-439d-a156-d6111ffb40a7\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" Oct 31 12:56:07.206528 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:07.206523 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4b60d0a-c847-439d-a156-d6111ffb40a7-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp\" (UID: \"a4b60d0a-c847-439d-a156-d6111ffb40a7\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" Oct 31 12:56:07.354429 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:07.354329 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" Oct 31 12:56:07.656940 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:07.653429 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp"] Oct 31 12:56:07.656940 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:56:07.655546 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4b60d0a_c847_439d_a156_d6111ffb40a7.slice/crio-fb2ec01e46c607c5fbbc0cb95e33813144be8f205ff5ee8f8002f37d44d59647 WatchSource:0}: Error finding container fb2ec01e46c607c5fbbc0cb95e33813144be8f205ff5ee8f8002f37d44d59647: Status 404 returned error can't find the container with id fb2ec01e46c607c5fbbc0cb95e33813144be8f205ff5ee8f8002f37d44d59647 Oct 31 12:56:07.658460 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:07.658424 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:56:07.658583 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:07.658503 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:56:07.658583 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:07.658548 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:56:08.642665 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:08.642472 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" event={"ID":"a4b60d0a-c847-439d-a156-d6111ffb40a7","Type":"ContainerStarted","Data":"fadceadd0330b24524f6a9ee5bcdb5925def479bd27453d8c1c0d45bde3b934d"} Oct 31 12:56:08.642665 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:08.642542 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" event={"ID":"a4b60d0a-c847-439d-a156-d6111ffb40a7","Type":"ContainerStarted","Data":"fb2ec01e46c607c5fbbc0cb95e33813144be8f205ff5ee8f8002f37d44d59647"} Oct 31 12:56:09.355678 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:09.354761 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" Oct 31 12:56:09.364674 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:09.364471 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 12:56:09.364674 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:09.364529 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" podUID="a4b60d0a-c847-439d-a156-d6111ffb40a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 12:56:10.304666 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:10.303814 5693 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" podUID="1a2abd3f-fb4f-4990-9a32-5f945a588068" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:10.358665 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:10.358106 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 12:56:10.358665 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:10.358185 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" podUID="a4b60d0a-c847-439d-a156-d6111ffb40a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 12:56:10.438668 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:10.433752 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" podUID="17d56bfd-8cb1-42bf-bb11-05810c46d469" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:10.487666 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:10.482732 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" podUID="1a2abd3f-fb4f-4990-9a32-5f945a588068" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:10.681665 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:10.674408 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" podUID="e5a40b79-9879-4981-ab9f-446e1750a013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:10.686666 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:10.684902 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" podUID="8fb66a94-c399-43fb-945c-610ca63f0d76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:10.781716 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:10.779709 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" podUID="710a52fa-2bc4-4bb9-96be-4b08dc60de67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 31 12:56:10.816678 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:10.804021 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" podUID="ff9a9909-6fc3-475b-a9fe-562ec150c76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:10.991075 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:10.989845 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" podUID="f2b3766f-058c-40e7-9a2d-a7f5028e2b1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:11.355130 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:11.355000 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 12:56:11.355130 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:11.355095 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" podUID="a4b60d0a-c847-439d-a156-d6111ffb40a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 12:56:12.355135 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:12.355101 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 12:56:12.355526 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:12.355156 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" podUID="a4b60d0a-c847-439d-a156-d6111ffb40a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 12:56:13.355560 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:13.355527 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 12:56:13.356034 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:13.355586 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" podUID="a4b60d0a-c847-439d-a156-d6111ffb40a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 12:56:13.792698 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:13.792618 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" podStartSLOduration=6.79259883 podStartE2EDuration="6.79259883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:56:08.671458756 +0000 UTC m=+1419.379045107" watchObservedRunningTime="2026-10-31 12:56:13.79259883 +0000 UTC m=+1424.500185126" Oct 31 12:56:13.804406 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:13.798885 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs"] Oct 31 12:56:13.812855 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:13.812824 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs"] Oct 31 12:56:13.813053 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:13.812955 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" Oct 31 12:56:13.948704 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:13.948668 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d9b984c-d94b-4d88-8b4b-48375154dfaa-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs\" (UID: \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" Oct 31 12:56:13.948879 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:13.948735 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d9b984c-d94b-4d88-8b4b-48375154dfaa-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs\" (UID: \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" Oct 31 12:56:13.948879 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:13.948819 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d9b984c-d94b-4d88-8b4b-48375154dfaa-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs\" (UID: \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" Oct 31 12:56:13.948879 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:13.948865 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9rwt\" (UniqueName: \"kubernetes.io/projected/6d9b984c-d94b-4d88-8b4b-48375154dfaa-kube-api-access-h9rwt\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs\" (UID: \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" Oct 31 12:56:13.948991 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:13.948901 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d9b984c-d94b-4d88-8b4b-48375154dfaa-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs\" (UID: \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" Oct 31 12:56:13.948991 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:13.948916 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d9b984c-d94b-4d88-8b4b-48375154dfaa-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs\" (UID: \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" Oct 31 12:56:13.948991 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:13.948960 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d9b984c-d94b-4d88-8b4b-48375154dfaa-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs\" (UID: \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" Oct 31 12:56:13.948991 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:13.948982 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d9b984c-d94b-4d88-8b4b-48375154dfaa-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs\" (UID: \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" Oct 31 12:56:13.949119 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:13.949022 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d9b984c-d94b-4d88-8b4b-48375154dfaa-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs\" (UID: \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" Oct 31 12:56:14.050465 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:14.050378 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d9b984c-d94b-4d88-8b4b-48375154dfaa-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs\" (UID: \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" Oct 31 12:56:14.050465 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:14.050426 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d9b984c-d94b-4d88-8b4b-48375154dfaa-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs\" (UID: \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" Oct 31 12:56:14.050733 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:14.050512 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d9b984c-d94b-4d88-8b4b-48375154dfaa-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs\" (UID: \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" Oct 31 12:56:14.050733 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:14.050546 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d9b984c-d94b-4d88-8b4b-48375154dfaa-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs\" (UID: \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" Oct 31 12:56:14.050733 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:14.050584 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d9b984c-d94b-4d88-8b4b-48375154dfaa-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs\" (UID: \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" Oct 31 12:56:14.050733 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:14.050700 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d9b984c-d94b-4d88-8b4b-48375154dfaa-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs\" (UID: \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" Oct 31 12:56:14.050968 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:14.050774 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9rwt\" (UniqueName: \"kubernetes.io/projected/6d9b984c-d94b-4d88-8b4b-48375154dfaa-kube-api-access-h9rwt\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs\" (UID: \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" Oct 31 12:56:14.050968 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:14.050827 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d9b984c-d94b-4d88-8b4b-48375154dfaa-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs\" (UID: \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" Oct 31 12:56:14.050968 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:14.050851 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d9b984c-d94b-4d88-8b4b-48375154dfaa-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs\" (UID: \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" Oct 31 12:56:14.050968 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:14.050903 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d9b984c-d94b-4d88-8b4b-48375154dfaa-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs\" (UID: \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" Oct 31 12:56:14.051265 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:14.051240 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d9b984c-d94b-4d88-8b4b-48375154dfaa-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs\" (UID: \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" Oct 31 12:56:14.051440 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:14.051417 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d9b984c-d94b-4d88-8b4b-48375154dfaa-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs\" (UID: \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" Oct 31 12:56:14.051582 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:14.051562 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d9b984c-d94b-4d88-8b4b-48375154dfaa-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs\" (UID: \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" Oct 31 12:56:14.051719 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:14.051631 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d9b984c-d94b-4d88-8b4b-48375154dfaa-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs\" (UID: \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" Oct 31 12:56:14.053632 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:14.053610 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d9b984c-d94b-4d88-8b4b-48375154dfaa-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs\" (UID: \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" Oct 31 12:56:14.054130 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:14.054109 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d9b984c-d94b-4d88-8b4b-48375154dfaa-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs\" (UID: \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" Oct 31 12:56:14.067960 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:14.067935 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d9b984c-d94b-4d88-8b4b-48375154dfaa-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs\" (UID: \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" Oct 31 12:56:14.068724 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:14.068705 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9rwt\" (UniqueName: \"kubernetes.io/projected/6d9b984c-d94b-4d88-8b4b-48375154dfaa-kube-api-access-h9rwt\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs\" (UID: \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" Oct 31 12:56:14.132921 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:14.132881 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" Oct 31 12:56:14.355598 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:14.355501 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 12:56:14.355598 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:14.355557 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" podUID="a4b60d0a-c847-439d-a156-d6111ffb40a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 12:56:14.456239 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:14.456209 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs"] Oct 31 12:56:14.458124 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:56:14.458096 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d9b984c_d94b_4d88_8b4b_48375154dfaa.slice/crio-3965df2c000f19af5a63179e5eac3c501927a2691b89e18e09adb8695d0195e9 WatchSource:0}: Error finding container 3965df2c000f19af5a63179e5eac3c501927a2691b89e18e09adb8695d0195e9: Status 404 returned error can't find the container with id 3965df2c000f19af5a63179e5eac3c501927a2691b89e18e09adb8695d0195e9 Oct 31 12:56:14.461271 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:14.461228 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:56:14.461346 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:14.461322 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:56:14.461388 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:14.461372 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:56:15.137345 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:15.137309 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" event={"ID":"6d9b984c-d94b-4d88-8b4b-48375154dfaa","Type":"ContainerStarted","Data":"8c6bcb7b3aa1e39e060d1e0dda5998d1b8b9b4efe60ee32bbfd8eebb76e4e099"} Oct 31 12:56:15.137572 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:15.137372 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" event={"ID":"6d9b984c-d94b-4d88-8b4b-48375154dfaa","Type":"ContainerStarted","Data":"3965df2c000f19af5a63179e5eac3c501927a2691b89e18e09adb8695d0195e9"} Oct 31 12:56:15.166585 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:15.166530 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" podStartSLOduration=2.166516611 podStartE2EDuration="2.166516611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:56:15.163713109 +0000 UTC m=+1425.871299405" watchObservedRunningTime="2026-10-31 12:56:15.166516611 +0000 UTC m=+1425.874102905" Oct 31 12:56:16.136282 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:16.136244 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" Oct 31 12:56:16.137768 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:16.137739 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 12:56:16.137943 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:16.137798 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" podUID="6d9b984c-d94b-4d88-8b4b-48375154dfaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 12:56:16.355797 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:16.355746 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:56:16.355999 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:16.355807 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" podUID="a4b60d0a-c847-439d-a156-d6111ffb40a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:17.136034 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:17.136003 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 12:56:17.136301 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:17.136275 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" podUID="6d9b984c-d94b-4d88-8b4b-48375154dfaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 12:56:17.243228 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:17.243188 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" Oct 31 12:56:17.243668 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:17.243635 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" Oct 31 12:56:17.244604 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:17.244582 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" Oct 31 12:56:18.066530 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:18.066495 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp"] Oct 31 12:56:18.070408 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:18.070384 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" Oct 31 12:56:18.120841 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:18.120813 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp"] Oct 31 12:56:18.135621 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:18.135573 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 12:56:18.135819 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:18.135681 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" podUID="6d9b984c-d94b-4d88-8b4b-48375154dfaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 12:56:18.138875 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:18.138840 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd8946ab-9867-4565-98e0-9d2c62eaff29-workload-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp\" (UID: \"dd8946ab-9867-4565-98e0-9d2c62eaff29\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" Oct 31 12:56:18.139238 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:18.138907 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2rl6\" (UniqueName: \"kubernetes.io/projected/dd8946ab-9867-4565-98e0-9d2c62eaff29-kube-api-access-d2rl6\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp\" (UID: \"dd8946ab-9867-4565-98e0-9d2c62eaff29\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" Oct 31 12:56:18.139238 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:18.138963 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd8946ab-9867-4565-98e0-9d2c62eaff29-istio-podinfo\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp\" (UID: \"dd8946ab-9867-4565-98e0-9d2c62eaff29\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" Oct 31 12:56:18.139238 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:18.139043 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd8946ab-9867-4565-98e0-9d2c62eaff29-istiod-ca-cert\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp\" (UID: \"dd8946ab-9867-4565-98e0-9d2c62eaff29\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" Oct 31 12:56:18.139238 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:18.139113 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd8946ab-9867-4565-98e0-9d2c62eaff29-istio-envoy\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp\" (UID: \"dd8946ab-9867-4565-98e0-9d2c62eaff29\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" Oct 31 12:56:18.139238 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:18.139160 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd8946ab-9867-4565-98e0-9d2c62eaff29-credential-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp\" (UID: \"dd8946ab-9867-4565-98e0-9d2c62eaff29\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" Oct 31 12:56:18.139238 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:18.139210 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd8946ab-9867-4565-98e0-9d2c62eaff29-workload-certs\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp\" (UID: \"dd8946ab-9867-4565-98e0-9d2c62eaff29\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" Oct 31 12:56:18.139238 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:18.139235 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd8946ab-9867-4565-98e0-9d2c62eaff29-istio-data\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp\" (UID: \"dd8946ab-9867-4565-98e0-9d2c62eaff29\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" Oct 31 12:56:18.139485 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:18.139277 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd8946ab-9867-4565-98e0-9d2c62eaff29-istio-token\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp\" (UID: \"dd8946ab-9867-4565-98e0-9d2c62eaff29\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" Oct 31 12:56:18.240010 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:18.239975 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd8946ab-9867-4565-98e0-9d2c62eaff29-workload-certs\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp\" (UID: \"dd8946ab-9867-4565-98e0-9d2c62eaff29\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" Oct 31 12:56:18.240010 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:18.240014 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd8946ab-9867-4565-98e0-9d2c62eaff29-istio-data\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp\" (UID: \"dd8946ab-9867-4565-98e0-9d2c62eaff29\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" Oct 31 12:56:18.240265 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:18.240066 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd8946ab-9867-4565-98e0-9d2c62eaff29-istio-token\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp\" (UID: \"dd8946ab-9867-4565-98e0-9d2c62eaff29\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" Oct 31 12:56:18.240265 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:18.240100 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd8946ab-9867-4565-98e0-9d2c62eaff29-workload-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp\" (UID: \"dd8946ab-9867-4565-98e0-9d2c62eaff29\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" Oct 31 12:56:18.240265 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:18.240147 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2rl6\" (UniqueName: \"kubernetes.io/projected/dd8946ab-9867-4565-98e0-9d2c62eaff29-kube-api-access-d2rl6\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp\" (UID: \"dd8946ab-9867-4565-98e0-9d2c62eaff29\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" Oct 31 12:56:18.240409 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:18.240327 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd8946ab-9867-4565-98e0-9d2c62eaff29-istio-podinfo\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp\" (UID: \"dd8946ab-9867-4565-98e0-9d2c62eaff29\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" Oct 31 12:56:18.240467 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:18.240443 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd8946ab-9867-4565-98e0-9d2c62eaff29-istio-data\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp\" (UID: \"dd8946ab-9867-4565-98e0-9d2c62eaff29\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" Oct 31 12:56:18.240522 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:18.240480 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd8946ab-9867-4565-98e0-9d2c62eaff29-istiod-ca-cert\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp\" (UID: \"dd8946ab-9867-4565-98e0-9d2c62eaff29\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" Oct 31 12:56:18.240587 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:18.240529 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd8946ab-9867-4565-98e0-9d2c62eaff29-workload-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp\" (UID: \"dd8946ab-9867-4565-98e0-9d2c62eaff29\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" Oct 31 12:56:18.240689 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:18.240623 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd8946ab-9867-4565-98e0-9d2c62eaff29-istio-envoy\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp\" (UID: \"dd8946ab-9867-4565-98e0-9d2c62eaff29\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" Oct 31 12:56:18.240770 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:18.240708 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd8946ab-9867-4565-98e0-9d2c62eaff29-credential-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp\" (UID: \"dd8946ab-9867-4565-98e0-9d2c62eaff29\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" Oct 31 12:56:18.240837 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:18.240809 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd8946ab-9867-4565-98e0-9d2c62eaff29-workload-certs\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp\" (UID: \"dd8946ab-9867-4565-98e0-9d2c62eaff29\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" Oct 31 12:56:18.241260 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:18.241242 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd8946ab-9867-4565-98e0-9d2c62eaff29-credential-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp\" (UID: \"dd8946ab-9867-4565-98e0-9d2c62eaff29\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" Oct 31 12:56:18.241430 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:18.241240 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd8946ab-9867-4565-98e0-9d2c62eaff29-istiod-ca-cert\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp\" (UID: \"dd8946ab-9867-4565-98e0-9d2c62eaff29\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" Oct 31 12:56:18.243009 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:18.242990 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd8946ab-9867-4565-98e0-9d2c62eaff29-istio-envoy\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp\" (UID: \"dd8946ab-9867-4565-98e0-9d2c62eaff29\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" Oct 31 12:56:18.243150 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:18.243131 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd8946ab-9867-4565-98e0-9d2c62eaff29-istio-podinfo\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp\" (UID: \"dd8946ab-9867-4565-98e0-9d2c62eaff29\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" Oct 31 12:56:18.254917 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:18.254851 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd8946ab-9867-4565-98e0-9d2c62eaff29-istio-token\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp\" (UID: \"dd8946ab-9867-4565-98e0-9d2c62eaff29\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" Oct 31 12:56:18.258161 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:18.258090 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2rl6\" (UniqueName: \"kubernetes.io/projected/dd8946ab-9867-4565-98e0-9d2c62eaff29-kube-api-access-d2rl6\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp\" (UID: \"dd8946ab-9867-4565-98e0-9d2c62eaff29\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" Oct 31 12:56:18.426198 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:18.426167 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" Oct 31 12:56:18.717879 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:56:18.717754 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:56:18.739438 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:18.739411 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp"] Oct 31 12:56:18.741021 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:56:18.740977 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd8946ab_9867_4565_98e0_9d2c62eaff29.slice/crio-fe40682a5c254ae98de9435427da2bcd4c215bbf88564067eabab19f260b2fa4 WatchSource:0}: Error finding container fe40682a5c254ae98de9435427da2bcd4c215bbf88564067eabab19f260b2fa4: Status 404 returned error can't find the container with id fe40682a5c254ae98de9435427da2bcd4c215bbf88564067eabab19f260b2fa4 Oct 31 12:56:18.744163 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:18.744118 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:56:18.744238 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:18.744208 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:56:18.744282 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:18.744249 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:56:19.135396 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:19.135316 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 12:56:19.135396 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:19.135367 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" podUID="6d9b984c-d94b-4d88-8b4b-48375154dfaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 12:56:19.165327 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:19.165292 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" event={"ID":"dd8946ab-9867-4565-98e0-9d2c62eaff29","Type":"ContainerStarted","Data":"89cdb613060a8d3b421a700ead246bffd084515f0a6092a1d91fd0adf90d8659"} Oct 31 12:56:19.165830 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:19.165346 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" event={"ID":"dd8946ab-9867-4565-98e0-9d2c62eaff29","Type":"ContainerStarted","Data":"fe40682a5c254ae98de9435427da2bcd4c215bbf88564067eabab19f260b2fa4"} Oct 31 12:56:19.426515 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:19.426430 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" Oct 31 12:56:19.427933 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:19.427907 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 12:56:19.427998 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:19.427971 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" podUID="dd8946ab-9867-4565-98e0-9d2c62eaff29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 12:56:20.136487 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:20.136453 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 12:56:20.136705 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:20.136508 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" podUID="6d9b984c-d94b-4d88-8b4b-48375154dfaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 12:56:20.430166 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:20.430128 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 12:56:20.430860 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:20.430828 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" podUID="dd8946ab-9867-4565-98e0-9d2c62eaff29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 12:56:21.135714 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:21.135679 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 12:56:21.135909 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:21.135733 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" podUID="6d9b984c-d94b-4d88-8b4b-48375154dfaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 12:56:21.432670 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:21.432075 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 12:56:21.432670 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:21.432140 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" podUID="dd8946ab-9867-4565-98e0-9d2c62eaff29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 12:56:22.430063 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:22.430023 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 12:56:22.430360 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:22.430330 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" podUID="dd8946ab-9867-4565-98e0-9d2c62eaff29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 12:56:22.933665 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:22.932981 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" Oct 31 12:56:22.939665 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:22.934481 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" Oct 31 12:56:22.940743 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:22.940705 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" Oct 31 12:56:22.981676 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:22.978407 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" podStartSLOduration=4.978383957 podStartE2EDuration="4.978383957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:56:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:56:19.199983094 +0000 UTC m=+1429.907569393" watchObservedRunningTime="2026-10-31 12:56:22.978383957 +0000 UTC m=+1433.685970252" Oct 31 12:56:23.459697 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:23.441909 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 12:56:23.459697 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:23.441980 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" podUID="dd8946ab-9867-4565-98e0-9d2c62eaff29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 12:56:23.480751 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:23.474772 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" podUID="1a2abd3f-fb4f-4990-9a32-5f945a588068" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:23.669853 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:23.669806 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" podUID="e5a40b79-9879-4981-ab9f-446e1750a013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:23.701560 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:23.701489 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" podUID="8fb66a94-c399-43fb-945c-610ca63f0d76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:23.850182 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:23.844564 5693 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" podUID="8fb66a94-c399-43fb-945c-610ca63f0d76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded" Oct 31 12:56:24.428103 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:24.428064 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 12:56:24.428786 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:24.428741 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" podUID="dd8946ab-9867-4565-98e0-9d2c62eaff29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 12:56:25.427085 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:25.427054 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 12:56:25.427277 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:25.427105 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" podUID="dd8946ab-9867-4565-98e0-9d2c62eaff29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 12:56:26.426982 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:26.426941 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 12:56:26.427455 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:26.427008 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" podUID="dd8946ab-9867-4565-98e0-9d2c62eaff29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 12:56:28.365491 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:28.365459 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" Oct 31 12:56:28.365966 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:28.365894 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" Oct 31 12:56:28.367257 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:28.367233 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" Oct 31 12:56:39.698844 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:39.686857 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" podUID="8fb66a94-c399-43fb-945c-610ca63f0d76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:39.740681 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:39.716795 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" podUID="710a52fa-2bc4-4bb9-96be-4b08dc60de67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:39.740681 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:39.723778 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" podUID="e5a40b79-9879-4981-ab9f-446e1750a013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:39.794661 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:39.784768 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" podUID="ff9a9909-6fc3-475b-a9fe-562ec150c76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:39.871666 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:39.861068 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" podUID="f2b3766f-058c-40e7-9a2d-a7f5028e2b1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:40.128884 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:40.128764 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" podUID="66c6ec5a-c0da-4bd7-a314-0c90ccec2425" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:40.252787 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:40.252739 5693 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" podUID="1a2abd3f-fb4f-4990-9a32-5f945a588068" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:40.497669 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:40.491326 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" podUID="1a2abd3f-fb4f-4990-9a32-5f945a588068" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:40.503667 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:40.498098 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" podUID="17d56bfd-8cb1-42bf-bb11-05810c46d469" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:40.699117 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:40.699065 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" podUID="8fb66a94-c399-43fb-945c-610ca63f0d76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 12:56:40.700314 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:40.700290 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" Oct 31 12:56:40.718145 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:40.718102 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" podUID="710a52fa-2bc4-4bb9-96be-4b08dc60de67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:40.718803 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:40.718779 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" Oct 31 12:56:40.725210 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:40.725177 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" podUID="e5a40b79-9879-4981-ab9f-446e1750a013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:40.725785 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:40.725766 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" Oct 31 12:56:40.935707 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:40.913826 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" podUID="f2b3766f-058c-40e7-9a2d-a7f5028e2b1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Oct 31 12:56:41.078543 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:40.939586 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" podUID="ff9a9909-6fc3-475b-a9fe-562ec150c76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 31 12:56:41.079305 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:41.079277 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" Oct 31 12:56:41.127681 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:41.101138 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" Oct 31 12:56:41.154701 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:41.151021 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" podUID="66c6ec5a-c0da-4bd7-a314-0c90ccec2425" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 12:56:41.198400 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:41.198307 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" Oct 31 12:56:41.539480 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:41.539379 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" podUID="1a2abd3f-fb4f-4990-9a32-5f945a588068" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 31 12:56:41.541930 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:41.539719 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" podUID="17d56bfd-8cb1-42bf-bb11-05810c46d469" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:41.710307 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:41.710262 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" podUID="8fb66a94-c399-43fb-945c-610ca63f0d76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:41.780736 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:41.771571 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" podUID="710a52fa-2bc4-4bb9-96be-4b08dc60de67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:41.807223 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:41.805877 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" podUID="e5a40b79-9879-4981-ab9f-446e1750a013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:41.899667 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:41.895916 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" Oct 31 12:56:41.899667 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:41.895981 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" Oct 31 12:56:41.899667 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:41.896012 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" Oct 31 12:56:41.899667 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:41.896037 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" Oct 31 12:56:41.978695 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:41.967765 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-rskhv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:56:41.978695 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:41.967830 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" podUID="2ae02ff0-1e33-4007-a97a-63a61e8b4bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:41.993876 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:41.993742 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fv482 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:56:41.993876 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:41.993805 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" podUID="c0bdd69c-691b-47b2-ba17-2e0ac7185643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:42.092863 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:42.092732 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" podUID="f2b3766f-058c-40e7-9a2d-a7f5028e2b1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:42.202075 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:42.202036 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" podUID="66c6ec5a-c0da-4bd7-a314-0c90ccec2425" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:42.413752 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:42.413714 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" Oct 31 12:56:42.482666 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:42.441400 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" Oct 31 12:56:42.590663 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:42.590237 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" podUID="17d56bfd-8cb1-42bf-bb11-05810c46d469" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:42.719771 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:42.719678 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" Oct 31 12:56:42.862489 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:42.861525 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" Oct 31 12:56:42.912086 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:42.912039 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" Oct 31 12:56:43.022721 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:43.020154 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" Oct 31 12:56:47.965378 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:47.965338 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4"] Oct 31 12:56:47.969334 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:47.969313 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" Oct 31 12:56:47.986339 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:47.986309 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4"] Oct 31 12:56:48.028593 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:48.028558 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/024bc3d1-116f-4d0d-b8e6-d37839429e0e-istio-token\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4\" (UID: \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" Oct 31 12:56:48.028593 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:48.028596 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/024bc3d1-116f-4d0d-b8e6-d37839429e0e-workload-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4\" (UID: \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" Oct 31 12:56:48.028914 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:48.028623 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/024bc3d1-116f-4d0d-b8e6-d37839429e0e-istio-envoy\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4\" (UID: \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" Oct 31 12:56:48.028914 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:48.028775 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/024bc3d1-116f-4d0d-b8e6-d37839429e0e-workload-certs\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4\" (UID: \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" Oct 31 12:56:48.028914 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:48.028838 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-562z8\" (UniqueName: \"kubernetes.io/projected/024bc3d1-116f-4d0d-b8e6-d37839429e0e-kube-api-access-562z8\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4\" (UID: \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" Oct 31 12:56:48.028914 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:48.028902 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/024bc3d1-116f-4d0d-b8e6-d37839429e0e-istiod-ca-cert\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4\" (UID: \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" Oct 31 12:56:48.029133 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:48.028956 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/024bc3d1-116f-4d0d-b8e6-d37839429e0e-credential-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4\" (UID: \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" Oct 31 12:56:48.029133 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:48.028972 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/024bc3d1-116f-4d0d-b8e6-d37839429e0e-istio-podinfo\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4\" (UID: \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" Oct 31 12:56:48.029133 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:48.029065 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/024bc3d1-116f-4d0d-b8e6-d37839429e0e-istio-data\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4\" (UID: \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" Oct 31 12:56:48.130237 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:48.130101 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/024bc3d1-116f-4d0d-b8e6-d37839429e0e-istiod-ca-cert\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4\" (UID: \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" Oct 31 12:56:48.130431 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:48.130306 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/024bc3d1-116f-4d0d-b8e6-d37839429e0e-credential-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4\" (UID: \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" Oct 31 12:56:48.130431 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:48.130335 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/024bc3d1-116f-4d0d-b8e6-d37839429e0e-istio-podinfo\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4\" (UID: \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" Oct 31 12:56:48.130431 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:48.130380 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/024bc3d1-116f-4d0d-b8e6-d37839429e0e-istio-data\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4\" (UID: \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" Oct 31 12:56:48.130623 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:48.130466 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/024bc3d1-116f-4d0d-b8e6-d37839429e0e-istio-token\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4\" (UID: \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" Oct 31 12:56:48.130623 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:48.130503 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/024bc3d1-116f-4d0d-b8e6-d37839429e0e-workload-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4\" (UID: \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" Oct 31 12:56:48.130623 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:48.130546 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/024bc3d1-116f-4d0d-b8e6-d37839429e0e-istio-envoy\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4\" (UID: \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" Oct 31 12:56:48.130623 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:48.130608 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/024bc3d1-116f-4d0d-b8e6-d37839429e0e-workload-certs\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4\" (UID: \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" Oct 31 12:56:48.130883 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:48.130699 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-562z8\" (UniqueName: \"kubernetes.io/projected/024bc3d1-116f-4d0d-b8e6-d37839429e0e-kube-api-access-562z8\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4\" (UID: \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" Oct 31 12:56:48.131080 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:48.130936 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/024bc3d1-116f-4d0d-b8e6-d37839429e0e-credential-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4\" (UID: \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" Oct 31 12:56:48.131080 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:48.130973 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/024bc3d1-116f-4d0d-b8e6-d37839429e0e-istiod-ca-cert\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4\" (UID: \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" Oct 31 12:56:48.131233 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:48.131165 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/024bc3d1-116f-4d0d-b8e6-d37839429e0e-workload-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4\" (UID: \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" Oct 31 12:56:48.131328 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:48.131307 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/024bc3d1-116f-4d0d-b8e6-d37839429e0e-istio-data\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4\" (UID: \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" Oct 31 12:56:48.131387 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:48.131352 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/024bc3d1-116f-4d0d-b8e6-d37839429e0e-workload-certs\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4\" (UID: \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" Oct 31 12:56:48.133396 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:48.133376 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/024bc3d1-116f-4d0d-b8e6-d37839429e0e-istio-envoy\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4\" (UID: \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" Oct 31 12:56:48.133605 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:48.133588 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/024bc3d1-116f-4d0d-b8e6-d37839429e0e-istio-podinfo\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4\" (UID: \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" Oct 31 12:56:48.145976 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:48.145949 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/024bc3d1-116f-4d0d-b8e6-d37839429e0e-istio-token\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4\" (UID: \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" Oct 31 12:56:48.148454 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:48.148434 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-562z8\" (UniqueName: \"kubernetes.io/projected/024bc3d1-116f-4d0d-b8e6-d37839429e0e-kube-api-access-562z8\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4\" (UID: \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" Oct 31 12:56:48.286290 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:48.286203 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" Oct 31 12:56:48.574238 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:48.574214 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4"] Oct 31 12:56:48.575668 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:56:48.575615 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod024bc3d1_116f_4d0d_b8e6_d37839429e0e.slice/crio-32577a1f15e3d3e6e89f20df5f79820405e6ff95da957c150d0ffd1470d65899 WatchSource:0}: Error finding container 32577a1f15e3d3e6e89f20df5f79820405e6ff95da957c150d0ffd1470d65899: Status 404 returned error can't find the container with id 32577a1f15e3d3e6e89f20df5f79820405e6ff95da957c150d0ffd1470d65899 Oct 31 12:56:48.578058 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:48.578025 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:56:48.578133 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:48.578092 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:56:48.578133 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:48.578120 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:56:48.694682 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:56:48.691925 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:56:48.952937 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:48.952903 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" event={"ID":"024bc3d1-116f-4d0d-b8e6-d37839429e0e","Type":"ContainerStarted","Data":"bfecb7008e5791b0b5c9d75229ff53f49bd5e673582da6fd1630bb7a7bb4ad58"} Oct 31 12:56:48.953149 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:48.952964 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" event={"ID":"024bc3d1-116f-4d0d-b8e6-d37839429e0e","Type":"ContainerStarted","Data":"32577a1f15e3d3e6e89f20df5f79820405e6ff95da957c150d0ffd1470d65899"} Oct 31 12:56:48.986009 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:48.985958 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" podStartSLOduration=1.985944889 podStartE2EDuration="1.985944889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:56:48.982854774 +0000 UTC m=+1459.690441065" watchObservedRunningTime="2026-10-31 12:56:48.985944889 +0000 UTC m=+1459.693531184" Oct 31 12:56:49.294725 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:49.293476 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" Oct 31 12:56:49.296306 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:49.295927 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 12:56:49.296306 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:49.295984 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" podUID="024bc3d1-116f-4d0d-b8e6-d37839429e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 12:56:50.290664 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:50.288129 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 12:56:50.290664 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:50.288197 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" podUID="024bc3d1-116f-4d0d-b8e6-d37839429e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 12:56:51.290102 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:51.290062 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 12:56:51.294664 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:51.290790 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" podUID="024bc3d1-116f-4d0d-b8e6-d37839429e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 12:56:51.904886 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:51.890076 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" podUID="66c6ec5a-c0da-4bd7-a314-0c90ccec2425" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:51.907630 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:51.907588 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" podUID="8fb66a94-c399-43fb-945c-610ca63f0d76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:51.909055 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:51.909024 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" podUID="e5a40b79-9879-4981-ab9f-446e1750a013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:52.071669 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:52.062739 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" podUID="f2b3766f-058c-40e7-9a2d-a7f5028e2b1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:52.287596 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:52.287069 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 12:56:52.287596 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:52.287140 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" podUID="024bc3d1-116f-4d0d-b8e6-d37839429e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 12:56:52.415807 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:52.415763 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" podUID="710a52fa-2bc4-4bb9-96be-4b08dc60de67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:52.480808 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:52.480765 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" podUID="1a2abd3f-fb4f-4990-9a32-5f945a588068" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:52.596800 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:52.596704 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" podUID="17d56bfd-8cb1-42bf-bb11-05810c46d469" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:52.703759 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:52.703712 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" podUID="ff9a9909-6fc3-475b-a9fe-562ec150c76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 31 12:56:52.899962 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:52.899922 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" podUID="66c6ec5a-c0da-4bd7-a314-0c90ccec2425" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 12:56:52.900956 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:52.900932 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" Oct 31 12:56:52.918870 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:52.917128 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" podUID="8fb66a94-c399-43fb-945c-610ca63f0d76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:52.918870 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:52.917565 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" Oct 31 12:56:52.920277 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:52.917041 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" podUID="e5a40b79-9879-4981-ab9f-446e1750a013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 12:56:52.921031 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:52.920700 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" Oct 31 12:56:53.070618 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:53.067721 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" podUID="f2b3766f-058c-40e7-9a2d-a7f5028e2b1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:53.070618 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:53.068429 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" Oct 31 12:56:53.235327 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:53.235241 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:56:53.235570 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:53.235541 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" podUID="595d267e-2643-4570-b24c-8b03e9e34705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:53.387811 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:53.387775 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 12:56:53.388258 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:53.388221 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" podUID="024bc3d1-116f-4d0d-b8e6-d37839429e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 12:56:53.493720 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:53.484967 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" podUID="1a2abd3f-fb4f-4990-9a32-5f945a588068" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:53.707669 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:53.706770 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" podUID="ff9a9909-6fc3-475b-a9fe-562ec150c76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:53.747854 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:53.741598 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" Oct 31 12:56:53.747854 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:53.741712 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" Oct 31 12:56:53.921342 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:53.921308 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" Oct 31 12:56:53.944770 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:53.944726 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" podUID="66c6ec5a-c0da-4bd7-a314-0c90ccec2425" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:53.950811 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:53.950775 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" podUID="8fb66a94-c399-43fb-945c-610ca63f0d76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 12:56:53.978634 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:53.978605 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" Oct 31 12:56:53.995756 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:53.995728 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" Oct 31 12:56:54.030110 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:54.030026 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" Oct 31 12:56:54.035560 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:54.035478 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" Oct 31 12:56:54.053885 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:54.051119 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" Oct 31 12:56:54.330096 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:54.330006 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 12:56:54.330350 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:54.330318 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" podUID="024bc3d1-116f-4d0d-b8e6-d37839429e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 12:56:55.297666 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:55.289043 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 12:56:55.297666 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:55.289103 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" podUID="024bc3d1-116f-4d0d-b8e6-d37839429e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 12:56:56.287076 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:56.287045 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 12:56:56.287246 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:56.287098 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" podUID="024bc3d1-116f-4d0d-b8e6-d37839429e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 12:56:58.293985 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:58.293944 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:56:58.294609 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:58.294567 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" podUID="024bc3d1-116f-4d0d-b8e6-d37839429e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:58.362734 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:58.362706 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" Oct 31 12:56:58.363043 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:58.363014 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" Oct 31 12:56:58.364370 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:58.364345 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" Oct 31 12:56:59.849545 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:59.849512 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm"] Oct 31 12:56:59.853708 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:59.853691 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" Oct 31 12:56:59.874021 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:59.873997 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-107955-openshift-default-dockercfg-qq5mx\"" Oct 31 12:56:59.875394 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:59.875369 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm"] Oct 31 12:56:59.903013 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:59.902984 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-istiod-ca-cert\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm\" (UID: \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" Oct 31 12:56:59.903177 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:59.903046 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-istio-envoy\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm\" (UID: \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" Oct 31 12:56:59.903177 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:59.903124 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-workload-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm\" (UID: \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" Oct 31 12:56:59.903313 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:59.903174 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrpc4\" (UniqueName: \"kubernetes.io/projected/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-kube-api-access-rrpc4\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm\" (UID: \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" Oct 31 12:56:59.903313 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:59.903208 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-istio-token\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm\" (UID: \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" Oct 31 12:56:59.903313 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:59.903272 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-credential-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm\" (UID: \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" Oct 31 12:56:59.903462 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:59.903442 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-istio-data\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm\" (UID: \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" Oct 31 12:56:59.903531 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:59.903489 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-istio-podinfo\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm\" (UID: \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" Oct 31 12:56:59.903531 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:56:59.903509 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-workload-certs\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm\" (UID: \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" Oct 31 12:57:00.004476 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:00.004445 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-istiod-ca-cert\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm\" (UID: \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" Oct 31 12:57:00.004698 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:00.004494 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-istio-envoy\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm\" (UID: \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" Oct 31 12:57:00.004698 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:00.004522 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-workload-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm\" (UID: \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" Oct 31 12:57:00.004698 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:00.004554 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrpc4\" (UniqueName: \"kubernetes.io/projected/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-kube-api-access-rrpc4\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm\" (UID: \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" Oct 31 12:57:00.004698 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:00.004581 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-istio-token\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm\" (UID: \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" Oct 31 12:57:00.004698 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:00.004632 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-credential-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm\" (UID: \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" Oct 31 12:57:00.004990 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:00.004713 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-istio-data\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm\" (UID: \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" Oct 31 12:57:00.004990 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:00.004755 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-istio-podinfo\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm\" (UID: \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" Oct 31 12:57:00.004990 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:00.004781 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-workload-certs\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm\" (UID: \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" Oct 31 12:57:00.005150 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:00.005071 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-credential-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm\" (UID: \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" Oct 31 12:57:00.005150 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:00.005138 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-istio-data\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm\" (UID: \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" Oct 31 12:57:00.005257 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:00.005201 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-workload-certs\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm\" (UID: \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" Oct 31 12:57:00.005325 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:00.005282 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-istiod-ca-cert\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm\" (UID: \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" Oct 31 12:57:00.005376 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:00.005329 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-workload-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm\" (UID: \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" Oct 31 12:57:00.007324 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:00.007301 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-istio-envoy\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm\" (UID: \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" Oct 31 12:57:00.007608 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:00.007588 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-istio-podinfo\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm\" (UID: \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" Oct 31 12:57:00.018822 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:00.018786 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-istio-token\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm\" (UID: \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" Oct 31 12:57:00.019130 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:00.019108 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrpc4\" (UniqueName: \"kubernetes.io/projected/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-kube-api-access-rrpc4\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm\" (UID: \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" Oct 31 12:57:00.168562 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:00.168529 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" Oct 31 12:57:00.467269 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:00.467236 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm"] Oct 31 12:57:00.467727 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:57:00.467690 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1de21e_24e3_46c6_a3cc_d8c16157ae4d.slice/crio-ad0bc17ead5620d550d5287fa2628550acb7c246ebd3f9b99bd1555f37af0928 WatchSource:0}: Error finding container ad0bc17ead5620d550d5287fa2628550acb7c246ebd3f9b99bd1555f37af0928: Status 404 returned error can't find the container with id ad0bc17ead5620d550d5287fa2628550acb7c246ebd3f9b99bd1555f37af0928 Oct 31 12:57:00.470385 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:00.470344 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:57:00.470463 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:00.470431 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:57:00.470499 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:00.470473 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:57:01.322898 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:01.322859 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" event={"ID":"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d","Type":"ContainerStarted","Data":"eaf95044644324f6393b7ae78b510bcb7e3fcf0be76b5262e677a3aa6393af21"} Oct 31 12:57:01.323555 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:01.323525 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" event={"ID":"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d","Type":"ContainerStarted","Data":"ad0bc17ead5620d550d5287fa2628550acb7c246ebd3f9b99bd1555f37af0928"} Oct 31 12:57:01.349091 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:01.349041 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" podStartSLOduration=2.349026971 podStartE2EDuration="2.349026971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:57:01.347029943 +0000 UTC m=+1472.054616237" watchObservedRunningTime="2026-10-31 12:57:01.349026971 +0000 UTC m=+1472.056613265" Oct 31 12:57:02.170369 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:02.170338 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" Oct 31 12:57:02.174786 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:02.174751 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 12:57:02.175037 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:02.175006 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" podUID="2f1de21e-24e3-46c6-a3cc-d8c16157ae4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 12:57:02.602356 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:02.602256 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" podUID="17d56bfd-8cb1-42bf-bb11-05810c46d469" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:02.966668 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:02.963737 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:57:02.966668 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:02.963819 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" podUID="49a12158-0b73-4dbe-86ee-42b72e883cee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:03.041671 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:03.039844 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" podUID="1a2abd3f-fb4f-4990-9a32-5f945a588068" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:03.175102 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:03.175061 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 12:57:03.175396 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:03.175364 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" podUID="2f1de21e-24e3-46c6-a3cc-d8c16157ae4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 12:57:03.606802 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:03.606756 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" podUID="17d56bfd-8cb1-42bf-bb11-05810c46d469" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 12:57:03.608371 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:03.608348 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" Oct 31 12:57:03.676420 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:03.676328 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" Oct 31 12:57:03.847666 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:03.844769 5693 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" podUID="8fb66a94-c399-43fb-945c-610ca63f0d76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded" Oct 31 12:57:03.999990 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:03.999941 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" podUID="ff9a9909-6fc3-475b-a9fe-562ec150c76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:04.044702 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:04.043927 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" podUID="1a2abd3f-fb4f-4990-9a32-5f945a588068" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:04.044702 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:04.044378 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" Oct 31 12:57:04.049906 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:04.049853 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" podUID="f2b3766f-058c-40e7-9a2d-a7f5028e2b1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:04.073685 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:04.069091 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" podUID="8fb66a94-c399-43fb-945c-610ca63f0d76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:04.222769 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:04.190523 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 12:57:04.222769 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:04.190578 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" podUID="2f1de21e-24e3-46c6-a3cc-d8c16157ae4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 12:57:04.356364 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:04.356282 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" Oct 31 12:57:05.169722 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:05.169689 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 12:57:05.170212 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:05.169749 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" podUID="2f1de21e-24e3-46c6-a3cc-d8c16157ae4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 12:57:06.169043 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:06.169006 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 12:57:06.169263 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:06.169065 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" podUID="2f1de21e-24e3-46c6-a3cc-d8c16157ae4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 12:57:07.169154 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:07.169121 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 12:57:07.169671 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:07.169181 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" podUID="2f1de21e-24e3-46c6-a3cc-d8c16157ae4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 12:57:09.213908 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:09.213860 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:57:09.214421 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:09.213976 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" podUID="2f1de21e-24e3-46c6-a3cc-d8c16157ae4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:09.488760 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:09.488681 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" Oct 31 12:57:09.489132 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:09.489108 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" Oct 31 12:57:09.489810 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:09.489788 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" Oct 31 12:57:18.690661 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:57:18.689341 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:57:19.396686 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:19.396620 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw"] Oct 31 12:57:19.432739 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:19.417839 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw"] Oct 31 12:57:19.432739 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:19.417953 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" Oct 31 12:57:19.432739 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:19.432484 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-684kw\" (UID: \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" Oct 31 12:57:19.432739 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:19.432552 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-684kw\" (UID: \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" Oct 31 12:57:19.432739 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:19.432700 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-684kw\" (UID: \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" Oct 31 12:57:19.433054 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:19.432766 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-684kw\" (UID: \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" Oct 31 12:57:19.433054 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:19.432795 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-684kw\" (UID: \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" Oct 31 12:57:19.433054 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:19.432920 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-684kw\" (UID: \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" Oct 31 12:57:19.433054 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:19.432992 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-684kw\" (UID: \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" Oct 31 12:57:19.433181 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:19.433082 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-684kw\" (UID: \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" Oct 31 12:57:19.433181 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:19.433108 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssjmj\" (UniqueName: \"kubernetes.io/projected/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-kube-api-access-ssjmj\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-684kw\" (UID: \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" Oct 31 12:57:19.534290 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:19.534252 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssjmj\" (UniqueName: \"kubernetes.io/projected/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-kube-api-access-ssjmj\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-684kw\" (UID: \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" Oct 31 12:57:19.534494 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:19.534379 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-684kw\" (UID: \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" Oct 31 12:57:19.534494 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:19.534430 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-684kw\" (UID: \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" Oct 31 12:57:19.534494 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:19.534471 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-684kw\" (UID: \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" Oct 31 12:57:19.534704 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:19.534521 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-684kw\" (UID: \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" Oct 31 12:57:19.534704 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:19.534550 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-684kw\" (UID: \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" Oct 31 12:57:19.534704 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:19.534599 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-684kw\" (UID: \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" Oct 31 12:57:19.534974 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:19.534727 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-684kw\" (UID: \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" Oct 31 12:57:19.534974 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:19.534817 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-684kw\" (UID: \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" Oct 31 12:57:19.535101 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:19.535002 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-684kw\" (UID: \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" Oct 31 12:57:19.535218 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:19.535189 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-684kw\" (UID: \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" Oct 31 12:57:19.535375 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:19.535246 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-684kw\" (UID: \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" Oct 31 12:57:19.535447 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:19.535391 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-684kw\" (UID: \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" Oct 31 12:57:19.535565 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:19.535543 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-684kw\" (UID: \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" Oct 31 12:57:19.537528 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:19.537500 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-684kw\" (UID: \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" Oct 31 12:57:19.537629 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:19.537596 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-684kw\" (UID: \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" Oct 31 12:57:19.546495 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:19.546466 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssjmj\" (UniqueName: \"kubernetes.io/projected/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-kube-api-access-ssjmj\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-684kw\" (UID: \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" Oct 31 12:57:19.547164 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:19.547142 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-684kw\" (UID: \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" Oct 31 12:57:19.753208 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:19.753118 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" Oct 31 12:57:20.041501 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:20.041473 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw"] Oct 31 12:57:20.042560 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:57:20.042531 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fe38c78_fe1a_4633_bbc7_696b2e517ea8.slice/crio-6a7a4f59bbed9cc1ac9d5e95eb62068d47c56ef9f5e1a0f3518426a8315db0e1 WatchSource:0}: Error finding container 6a7a4f59bbed9cc1ac9d5e95eb62068d47c56ef9f5e1a0f3518426a8315db0e1: Status 404 returned error can't find the container with id 6a7a4f59bbed9cc1ac9d5e95eb62068d47c56ef9f5e1a0f3518426a8315db0e1 Oct 31 12:57:20.045171 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:20.045132 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:57:20.045253 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:20.045215 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:57:20.045291 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:20.045258 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:57:20.153216 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:20.153180 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-702419-f5484d7cd-4xm5c"] Oct 31 12:57:20.160764 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:20.160727 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-702419-f5484d7cd-4xm5c"] Oct 31 12:57:20.160963 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:20.160852 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-702419-f5484d7cd-4xm5c" Oct 31 12:57:20.164718 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:20.164694 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-702419-serving-cert\"" Oct 31 12:57:20.353257 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:20.353175 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-792r2\" (UniqueName: \"kubernetes.io/projected/32faa6d4-5f7a-449c-973d-819d5aa6e1a6-kube-api-access-792r2\") pod \"maas-api-e2e-api-a-702419-f5484d7cd-4xm5c\" (UID: \"32faa6d4-5f7a-449c-973d-819d5aa6e1a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-702419-f5484d7cd-4xm5c" Oct 31 12:57:20.353448 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:20.353303 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32faa6d4-5f7a-449c-973d-819d5aa6e1a6-maas-api-tls\") pod \"maas-api-e2e-api-a-702419-f5484d7cd-4xm5c\" (UID: \"32faa6d4-5f7a-449c-973d-819d5aa6e1a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-702419-f5484d7cd-4xm5c" Oct 31 12:57:20.353448 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:20.353356 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32faa6d4-5f7a-449c-973d-819d5aa6e1a6-openshift-service-ca\") pod \"maas-api-e2e-api-a-702419-f5484d7cd-4xm5c\" (UID: \"32faa6d4-5f7a-449c-973d-819d5aa6e1a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-702419-f5484d7cd-4xm5c" Oct 31 12:57:20.353567 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:20.353499 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32faa6d4-5f7a-449c-973d-819d5aa6e1a6-metrics-certs\") pod \"maas-api-e2e-api-a-702419-f5484d7cd-4xm5c\" (UID: \"32faa6d4-5f7a-449c-973d-819d5aa6e1a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-702419-f5484d7cd-4xm5c" Oct 31 12:57:20.454522 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:20.454487 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32faa6d4-5f7a-449c-973d-819d5aa6e1a6-maas-api-tls\") pod \"maas-api-e2e-api-a-702419-f5484d7cd-4xm5c\" (UID: \"32faa6d4-5f7a-449c-973d-819d5aa6e1a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-702419-f5484d7cd-4xm5c" Oct 31 12:57:20.454688 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:20.454548 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32faa6d4-5f7a-449c-973d-819d5aa6e1a6-openshift-service-ca\") pod \"maas-api-e2e-api-a-702419-f5484d7cd-4xm5c\" (UID: \"32faa6d4-5f7a-449c-973d-819d5aa6e1a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-702419-f5484d7cd-4xm5c" Oct 31 12:57:20.454688 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:20.454619 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32faa6d4-5f7a-449c-973d-819d5aa6e1a6-metrics-certs\") pod \"maas-api-e2e-api-a-702419-f5484d7cd-4xm5c\" (UID: \"32faa6d4-5f7a-449c-973d-819d5aa6e1a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-702419-f5484d7cd-4xm5c" Oct 31 12:57:20.454822 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:20.454708 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-792r2\" (UniqueName: \"kubernetes.io/projected/32faa6d4-5f7a-449c-973d-819d5aa6e1a6-kube-api-access-792r2\") pod \"maas-api-e2e-api-a-702419-f5484d7cd-4xm5c\" (UID: \"32faa6d4-5f7a-449c-973d-819d5aa6e1a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-702419-f5484d7cd-4xm5c" Oct 31 12:57:20.455300 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:20.455270 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32faa6d4-5f7a-449c-973d-819d5aa6e1a6-openshift-service-ca\") pod \"maas-api-e2e-api-a-702419-f5484d7cd-4xm5c\" (UID: \"32faa6d4-5f7a-449c-973d-819d5aa6e1a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-702419-f5484d7cd-4xm5c" Oct 31 12:57:20.457229 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:20.457205 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32faa6d4-5f7a-449c-973d-819d5aa6e1a6-metrics-certs\") pod \"maas-api-e2e-api-a-702419-f5484d7cd-4xm5c\" (UID: \"32faa6d4-5f7a-449c-973d-819d5aa6e1a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-702419-f5484d7cd-4xm5c" Oct 31 12:57:20.457374 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:20.457351 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32faa6d4-5f7a-449c-973d-819d5aa6e1a6-maas-api-tls\") pod \"maas-api-e2e-api-a-702419-f5484d7cd-4xm5c\" (UID: \"32faa6d4-5f7a-449c-973d-819d5aa6e1a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-702419-f5484d7cd-4xm5c" Oct 31 12:57:20.467221 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:20.467197 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-792r2\" (UniqueName: \"kubernetes.io/projected/32faa6d4-5f7a-449c-973d-819d5aa6e1a6-kube-api-access-792r2\") pod \"maas-api-e2e-api-a-702419-f5484d7cd-4xm5c\" (UID: \"32faa6d4-5f7a-449c-973d-819d5aa6e1a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-702419-f5484d7cd-4xm5c" Oct 31 12:57:20.491097 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:20.491064 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-702419-f5484d7cd-4xm5c" Oct 31 12:57:20.520458 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:20.520425 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" event={"ID":"9fe38c78-fe1a-4633-bbc7-696b2e517ea8","Type":"ContainerStarted","Data":"948fe94efd57ef9f19e41753577a8bdcb0e5b07434eba7dcdd9a7783097fe9ae"} Oct 31 12:57:20.520595 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:20.520476 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" event={"ID":"9fe38c78-fe1a-4633-bbc7-696b2e517ea8","Type":"ContainerStarted","Data":"6a7a4f59bbed9cc1ac9d5e95eb62068d47c56ef9f5e1a0f3518426a8315db0e1"} Oct 31 12:57:20.558238 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:20.556816 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" podStartSLOduration=1.556794639 podStartE2EDuration="1.556794639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:57:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:57:20.556338537 +0000 UTC m=+1491.263924833" watchObservedRunningTime="2026-10-31 12:57:20.556794639 +0000 UTC m=+1491.264380938" Oct 31 12:57:20.756692 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:20.755772 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" Oct 31 12:57:20.814662 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:20.811324 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-702419-f5484d7cd-4xm5c"] Oct 31 12:57:20.832704 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:57:20.832630 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32faa6d4_5f7a_449c_973d_819d5aa6e1a6.slice/crio-b1504661d1e126eafee53f6a8c14df4cb345562d998051af614cf5b882993d03 WatchSource:0}: Error finding container b1504661d1e126eafee53f6a8c14df4cb345562d998051af614cf5b882993d03: Status 404 returned error can't find the container with id b1504661d1e126eafee53f6a8c14df4cb345562d998051af614cf5b882993d03 Oct 31 12:57:20.834668 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:20.834079 5693 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 12:57:21.527863 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:21.527823 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-702419-f5484d7cd-4xm5c" event={"ID":"32faa6d4-5f7a-449c-973d-819d5aa6e1a6","Type":"ContainerStarted","Data":"f1f773e94d2c4d2477ed05408b2b756f1edec796f90ebaf53beae6f15ce8cf15"} Oct 31 12:57:21.528056 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:21.527879 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-702419-f5484d7cd-4xm5c" event={"ID":"32faa6d4-5f7a-449c-973d-819d5aa6e1a6","Type":"ContainerStarted","Data":"b1504661d1e126eafee53f6a8c14df4cb345562d998051af614cf5b882993d03"} Oct 31 12:57:21.528056 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:21.527926 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-702419-f5484d7cd-4xm5c" Oct 31 12:57:21.564997 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:21.564887 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-702419-f5484d7cd-4xm5c" podStartSLOduration=1.093223712 podStartE2EDuration="1.564869835s" totalImagesPullingTime="471.646123ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:57:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:57:21.564003608 +0000 UTC m=+1492.271589904" watchObservedRunningTime="2026-10-31 12:57:21.564869835 +0000 UTC m=+1492.272456130" Oct 31 12:57:21.754965 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:21.754930 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 12:57:21.755266 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:21.755212 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" podUID="9fe38c78-fe1a-4633-bbc7-696b2e517ea8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 12:57:22.760669 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:22.756925 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 12:57:22.760669 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:22.756997 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" podUID="9fe38c78-fe1a-4633-bbc7-696b2e517ea8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 12:57:23.753545 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:23.753508 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 12:57:23.753824 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:23.753795 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" podUID="9fe38c78-fe1a-4633-bbc7-696b2e517ea8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 12:57:24.361668 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:24.361574 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" podUID="1a2abd3f-fb4f-4990-9a32-5f945a588068" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:24.415822 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:24.415768 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" podUID="710a52fa-2bc4-4bb9-96be-4b08dc60de67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:24.798668 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:24.776881 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 12:57:24.798668 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:24.777517 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" podUID="9fe38c78-fe1a-4633-bbc7-696b2e517ea8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 12:57:25.010667 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:25.007920 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" podUID="f2b3766f-058c-40e7-9a2d-a7f5028e2b1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Oct 31 12:57:25.037224 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:25.036728 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" podUID="e5a40b79-9879-4981-ab9f-446e1750a013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:25.096726 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:25.091177 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" podUID="8fb66a94-c399-43fb-945c-610ca63f0d76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:25.433831 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:25.417489 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" podUID="710a52fa-2bc4-4bb9-96be-4b08dc60de67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:25.465912 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:25.465880 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" Oct 31 12:57:25.556037 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:25.555999 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:57:25.556397 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:25.556352 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" podUID="2f1de21e-24e3-46c6-a3cc-d8c16157ae4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:25.741241 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:25.693302 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" podUID="17d56bfd-8cb1-42bf-bb11-05810c46d469" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:25.796805 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:25.775983 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" Oct 31 12:57:25.799740 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:25.799701 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 12:57:25.800012 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:25.799965 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" podUID="9fe38c78-fe1a-4633-bbc7-696b2e517ea8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 12:57:25.954016 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:25.953424 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" podUID="66c6ec5a-c0da-4bd7-a314-0c90ccec2425" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:26.045713 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:26.037262 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" podUID="e5a40b79-9879-4981-ab9f-446e1750a013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:26.045713 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:26.038350 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" Oct 31 12:57:26.067281 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:26.067257 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" Oct 31 12:57:26.754472 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:26.754437 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 12:57:26.754967 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:26.754492 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" podUID="9fe38c78-fe1a-4633-bbc7-696b2e517ea8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 12:57:27.567981 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:27.567950 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-702419-f5484d7cd-4xm5c" Oct 31 12:57:27.924178 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:27.924148 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" Oct 31 12:57:27.925468 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:27.925450 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" Oct 31 12:57:27.926244 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:27.926218 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" Oct 31 12:57:40.582908 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:40.582868 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-73503b-56954d7b47-p8brk"] Oct 31 12:57:40.588014 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:40.587993 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73503b-56954d7b47-p8brk" Oct 31 12:57:40.596400 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:40.596373 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-73503b-dockercfg-fpktf\"" Oct 31 12:57:40.600141 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:40.600111 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-73503b-56954d7b47-p8brk"] Oct 31 12:57:40.695817 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:40.695782 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-73503b\"" Oct 31 12:57:40.696581 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:40.696555 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzqkp\" (UniqueName: \"kubernetes.io/projected/895988e2-2481-4504-99b1-dbc84d806479-kube-api-access-qzqkp\") pod \"payload-pre-processing-e2e-api-b-73503b-56954d7b47-p8brk\" (UID: \"895988e2-2481-4504-99b1-dbc84d806479\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-73503b-56954d7b47-p8brk" Oct 31 12:57:40.696910 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:40.696850 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/895988e2-2481-4504-99b1-dbc84d806479-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-73503b-56954d7b47-p8brk\" (UID: \"895988e2-2481-4504-99b1-dbc84d806479\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-73503b-56954d7b47-p8brk" Oct 31 12:57:40.800822 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:40.800786 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/895988e2-2481-4504-99b1-dbc84d806479-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-73503b-56954d7b47-p8brk\" (UID: \"895988e2-2481-4504-99b1-dbc84d806479\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-73503b-56954d7b47-p8brk" Oct 31 12:57:40.802036 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:40.802014 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzqkp\" (UniqueName: \"kubernetes.io/projected/895988e2-2481-4504-99b1-dbc84d806479-kube-api-access-qzqkp\") pod \"payload-pre-processing-e2e-api-b-73503b-56954d7b47-p8brk\" (UID: \"895988e2-2481-4504-99b1-dbc84d806479\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-73503b-56954d7b47-p8brk" Oct 31 12:57:40.802855 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:40.801929 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/895988e2-2481-4504-99b1-dbc84d806479-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-73503b-56954d7b47-p8brk\" (UID: \"895988e2-2481-4504-99b1-dbc84d806479\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-73503b-56954d7b47-p8brk" Oct 31 12:57:40.845740 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:40.844020 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzqkp\" (UniqueName: \"kubernetes.io/projected/895988e2-2481-4504-99b1-dbc84d806479-kube-api-access-qzqkp\") pod \"payload-pre-processing-e2e-api-b-73503b-56954d7b47-p8brk\" (UID: \"895988e2-2481-4504-99b1-dbc84d806479\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-73503b-56954d7b47-p8brk" Oct 31 12:57:40.911097 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:40.911064 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73503b-56954d7b47-p8brk" Oct 31 12:57:41.197388 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:41.197357 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-73503b-56954d7b47-p8brk"] Oct 31 12:57:41.198269 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:57:41.198247 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod895988e2_2481_4504_99b1_dbc84d806479.slice/crio-e02a486b20a3209840f07fb3d9c13625650e987189ac64859fa1a87f0ff89184 WatchSource:0}: Error finding container e02a486b20a3209840f07fb3d9c13625650e987189ac64859fa1a87f0ff89184: Status 404 returned error can't find the container with id e02a486b20a3209840f07fb3d9c13625650e987189ac64859fa1a87f0ff89184 Oct 31 12:57:41.959396 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:41.959363 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73503b-56954d7b47-p8brk" event={"ID":"895988e2-2481-4504-99b1-dbc84d806479","Type":"ContainerStarted","Data":"a6e336407318b61738a958119decf4f261e2ac3f05679fd33f4282f89c5902a5"} Oct 31 12:57:41.959813 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:41.959415 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73503b-56954d7b47-p8brk" event={"ID":"895988e2-2481-4504-99b1-dbc84d806479","Type":"ContainerStarted","Data":"e02a486b20a3209840f07fb3d9c13625650e987189ac64859fa1a87f0ff89184"} Oct 31 12:57:41.959813 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:41.959460 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73503b-56954d7b47-p8brk" Oct 31 12:57:41.980382 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:41.980339 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73503b-56954d7b47-p8brk" podStartSLOduration=1.9803249040000002 podStartE2EDuration="1.980324904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:57:41.977357277 +0000 UTC m=+1512.684943573" watchObservedRunningTime="2026-10-31 12:57:41.980324904 +0000 UTC m=+1512.687911199" Oct 31 12:57:44.975678 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:44.958543 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942"] Oct 31 12:57:45.022665 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:44.996114 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" Oct 31 12:57:45.163671 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:45.137065 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/530ebf58-0d39-42be-879f-3c16bdf11d58-istio-data\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942\" (UID: \"530ebf58-0d39-42be-879f-3c16bdf11d58\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" Oct 31 12:57:45.163671 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:45.137188 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/530ebf58-0d39-42be-879f-3c16bdf11d58-credential-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942\" (UID: \"530ebf58-0d39-42be-879f-3c16bdf11d58\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" Oct 31 12:57:45.163671 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:45.137220 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df5z2\" (UniqueName: \"kubernetes.io/projected/530ebf58-0d39-42be-879f-3c16bdf11d58-kube-api-access-df5z2\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942\" (UID: \"530ebf58-0d39-42be-879f-3c16bdf11d58\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" Oct 31 12:57:45.163671 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:45.137251 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/530ebf58-0d39-42be-879f-3c16bdf11d58-istio-envoy\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942\" (UID: \"530ebf58-0d39-42be-879f-3c16bdf11d58\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" Oct 31 12:57:45.163671 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:45.137351 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/530ebf58-0d39-42be-879f-3c16bdf11d58-workload-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942\" (UID: \"530ebf58-0d39-42be-879f-3c16bdf11d58\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" Oct 31 12:57:45.163671 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:45.137384 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/530ebf58-0d39-42be-879f-3c16bdf11d58-workload-certs\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942\" (UID: \"530ebf58-0d39-42be-879f-3c16bdf11d58\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" Oct 31 12:57:45.163671 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:45.137470 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/530ebf58-0d39-42be-879f-3c16bdf11d58-istiod-ca-cert\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942\" (UID: \"530ebf58-0d39-42be-879f-3c16bdf11d58\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" Oct 31 12:57:45.163671 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:45.137565 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/530ebf58-0d39-42be-879f-3c16bdf11d58-istio-podinfo\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942\" (UID: \"530ebf58-0d39-42be-879f-3c16bdf11d58\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" Oct 31 12:57:45.163671 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:45.137604 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/530ebf58-0d39-42be-879f-3c16bdf11d58-istio-token\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942\" (UID: \"530ebf58-0d39-42be-879f-3c16bdf11d58\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" Oct 31 12:57:45.217509 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:45.217447 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942"] Oct 31 12:57:45.301143 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:45.269559 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/530ebf58-0d39-42be-879f-3c16bdf11d58-istio-data\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942\" (UID: \"530ebf58-0d39-42be-879f-3c16bdf11d58\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" Oct 31 12:57:45.301143 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:45.269727 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/530ebf58-0d39-42be-879f-3c16bdf11d58-credential-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942\" (UID: \"530ebf58-0d39-42be-879f-3c16bdf11d58\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" Oct 31 12:57:45.301143 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:45.269763 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df5z2\" (UniqueName: \"kubernetes.io/projected/530ebf58-0d39-42be-879f-3c16bdf11d58-kube-api-access-df5z2\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942\" (UID: \"530ebf58-0d39-42be-879f-3c16bdf11d58\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" Oct 31 12:57:45.301143 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:45.269796 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/530ebf58-0d39-42be-879f-3c16bdf11d58-istio-envoy\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942\" (UID: \"530ebf58-0d39-42be-879f-3c16bdf11d58\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" Oct 31 12:57:45.301143 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:45.269911 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/530ebf58-0d39-42be-879f-3c16bdf11d58-workload-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942\" (UID: \"530ebf58-0d39-42be-879f-3c16bdf11d58\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" Oct 31 12:57:45.301143 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:45.269946 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/530ebf58-0d39-42be-879f-3c16bdf11d58-workload-certs\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942\" (UID: \"530ebf58-0d39-42be-879f-3c16bdf11d58\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" Oct 31 12:57:45.301143 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:45.270036 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/530ebf58-0d39-42be-879f-3c16bdf11d58-istiod-ca-cert\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942\" (UID: \"530ebf58-0d39-42be-879f-3c16bdf11d58\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" Oct 31 12:57:45.301143 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:45.270140 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/530ebf58-0d39-42be-879f-3c16bdf11d58-istio-podinfo\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942\" (UID: \"530ebf58-0d39-42be-879f-3c16bdf11d58\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" Oct 31 12:57:45.301143 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:45.270182 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/530ebf58-0d39-42be-879f-3c16bdf11d58-istio-token\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942\" (UID: \"530ebf58-0d39-42be-879f-3c16bdf11d58\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" Oct 31 12:57:45.347297 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:45.346603 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/530ebf58-0d39-42be-879f-3c16bdf11d58-credential-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942\" (UID: \"530ebf58-0d39-42be-879f-3c16bdf11d58\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" Oct 31 12:57:45.376680 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:45.358342 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/530ebf58-0d39-42be-879f-3c16bdf11d58-istio-data\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942\" (UID: \"530ebf58-0d39-42be-879f-3c16bdf11d58\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" Oct 31 12:57:45.442669 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:45.414243 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/530ebf58-0d39-42be-879f-3c16bdf11d58-workload-certs\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942\" (UID: \"530ebf58-0d39-42be-879f-3c16bdf11d58\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" Oct 31 12:57:45.442669 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:45.414568 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/530ebf58-0d39-42be-879f-3c16bdf11d58-workload-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942\" (UID: \"530ebf58-0d39-42be-879f-3c16bdf11d58\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" Oct 31 12:57:45.442669 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:45.415206 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/530ebf58-0d39-42be-879f-3c16bdf11d58-istiod-ca-cert\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942\" (UID: \"530ebf58-0d39-42be-879f-3c16bdf11d58\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" Oct 31 12:57:45.442669 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:45.416995 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/530ebf58-0d39-42be-879f-3c16bdf11d58-istio-token\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942\" (UID: \"530ebf58-0d39-42be-879f-3c16bdf11d58\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" Oct 31 12:57:45.484668 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:45.483552 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df5z2\" (UniqueName: \"kubernetes.io/projected/530ebf58-0d39-42be-879f-3c16bdf11d58-kube-api-access-df5z2\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942\" (UID: \"530ebf58-0d39-42be-879f-3c16bdf11d58\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" Oct 31 12:57:45.513718 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:45.492032 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/530ebf58-0d39-42be-879f-3c16bdf11d58-istio-envoy\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942\" (UID: \"530ebf58-0d39-42be-879f-3c16bdf11d58\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" Oct 31 12:57:45.565285 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:45.560326 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/530ebf58-0d39-42be-879f-3c16bdf11d58-istio-podinfo\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942\" (UID: \"530ebf58-0d39-42be-879f-3c16bdf11d58\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" Oct 31 12:57:45.836949 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:45.832115 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" Oct 31 12:57:45.962963 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:45.962917 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" podUID="66c6ec5a-c0da-4bd7-a314-0c90ccec2425" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 12:57:45.988966 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:45.988908 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" podUID="f2b3766f-058c-40e7-9a2d-a7f5028e2b1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:46.021791 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:46.009019 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" podUID="ff9a9909-6fc3-475b-a9fe-562ec150c76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:46.069666 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:46.066339 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" podUID="8fb66a94-c399-43fb-945c-610ca63f0d76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:46.093998 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:46.093891 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" podUID="e5a40b79-9879-4981-ab9f-446e1750a013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:46.477668 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:46.454783 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:57:46.477668 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:46.454846 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" podUID="69c36f54-5f25-45ea-b887-c0d2e0f53cfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:46.477668 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:46.452813 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" podUID="1a2abd3f-fb4f-4990-9a32-5f945a588068" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:46.692667 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:46.691739 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" podUID="17d56bfd-8cb1-42bf-bb11-05810c46d469" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:46.787869 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:46.787757 5693 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" podUID="e5a40b79-9879-4981-ab9f-446e1750a013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:46.826181 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:46.826132 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" podUID="710a52fa-2bc4-4bb9-96be-4b08dc60de67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:47.033683 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:47.030588 5693 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" podUID="17d56bfd-8cb1-42bf-bb11-05810c46d469" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:47.033683 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:47.030720 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" podUID="66c6ec5a-c0da-4bd7-a314-0c90ccec2425" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:47.043037 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:47.042945 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" Oct 31 12:57:47.064877 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:47.064769 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" podUID="f2b3766f-058c-40e7-9a2d-a7f5028e2b1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:47.107665 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:47.079287 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" podUID="8fb66a94-c399-43fb-945c-610ca63f0d76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:47.107665 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:47.089686 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" podUID="ff9a9909-6fc3-475b-a9fe-562ec150c76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:47.107665 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:47.096747 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" Oct 31 12:57:47.107665 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:47.096811 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" Oct 31 12:57:47.107665 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:47.096859 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" Oct 31 12:57:47.107665 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:47.099963 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" podUID="e5a40b79-9879-4981-ab9f-446e1750a013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:47.107665 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:47.100301 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" Oct 31 12:57:47.562872 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:47.562829 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" podUID="1a2abd3f-fb4f-4990-9a32-5f945a588068" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:47.701556 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:47.701515 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" podUID="17d56bfd-8cb1-42bf-bb11-05810c46d469" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 12:57:47.718672 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:57:47.716897 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod530ebf58_0d39_42be_879f_3c16bdf11d58.slice/crio-2141764fc369b3dc9c53e1be1364089692ab51f87a229b007f3e4858b4b160a6 WatchSource:0}: Error finding container 2141764fc369b3dc9c53e1be1364089692ab51f87a229b007f3e4858b4b160a6: Status 404 returned error can't find the container with id 2141764fc369b3dc9c53e1be1364089692ab51f87a229b007f3e4858b4b160a6 Oct 31 12:57:47.725493 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:47.725454 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:57:47.725729 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:47.725709 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:57:47.725870 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:47.725853 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:57:47.752416 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:47.752389 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" Oct 31 12:57:47.752599 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:47.752443 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" Oct 31 12:57:47.752599 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:47.752473 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" Oct 31 12:57:47.776987 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:47.774250 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942"] Oct 31 12:57:47.776987 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:47.774362 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" Oct 31 12:57:47.778162 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:47.778057 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" Oct 31 12:57:48.074937 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:48.074221 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" Oct 31 12:57:48.074937 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:48.074287 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" Oct 31 12:57:48.100630 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:48.099217 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" podUID="f2b3766f-058c-40e7-9a2d-a7f5028e2b1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:48.100630 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:48.100016 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" podUID="ff9a9909-6fc3-475b-a9fe-562ec150c76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:48.331063 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:48.330712 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" Oct 31 12:57:48.521909 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:48.521864 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" Oct 31 12:57:48.882768 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:57:48.878062 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:57:49.017716 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:48.998491 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" event={"ID":"530ebf58-0d39-42be-879f-3c16bdf11d58","Type":"ContainerStarted","Data":"2141764fc369b3dc9c53e1be1364089692ab51f87a229b007f3e4858b4b160a6"} Oct 31 12:57:49.540429 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:49.540392 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs"] Oct 31 12:57:49.577790 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:49.576368 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" Oct 31 12:57:49.577790 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:49.577618 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs"] Oct 31 12:57:49.716435 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:49.716396 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5780a4a9-4b92-4663-8211-d831e6cfad35-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs\" (UID: \"5780a4a9-4b92-4663-8211-d831e6cfad35\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" Oct 31 12:57:49.716435 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:49.716444 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5780a4a9-4b92-4663-8211-d831e6cfad35-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs\" (UID: \"5780a4a9-4b92-4663-8211-d831e6cfad35\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" Oct 31 12:57:49.716734 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:49.716527 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5780a4a9-4b92-4663-8211-d831e6cfad35-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs\" (UID: \"5780a4a9-4b92-4663-8211-d831e6cfad35\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" Oct 31 12:57:49.716734 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:49.716692 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5780a4a9-4b92-4663-8211-d831e6cfad35-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs\" (UID: \"5780a4a9-4b92-4663-8211-d831e6cfad35\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" Oct 31 12:57:49.716734 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:49.716714 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td26m\" (UniqueName: \"kubernetes.io/projected/5780a4a9-4b92-4663-8211-d831e6cfad35-kube-api-access-td26m\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs\" (UID: \"5780a4a9-4b92-4663-8211-d831e6cfad35\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" Oct 31 12:57:49.716894 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:49.716843 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5780a4a9-4b92-4663-8211-d831e6cfad35-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs\" (UID: \"5780a4a9-4b92-4663-8211-d831e6cfad35\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" Oct 31 12:57:49.716894 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:49.716870 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5780a4a9-4b92-4663-8211-d831e6cfad35-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs\" (UID: \"5780a4a9-4b92-4663-8211-d831e6cfad35\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" Oct 31 12:57:49.717006 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:49.716965 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5780a4a9-4b92-4663-8211-d831e6cfad35-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs\" (UID: \"5780a4a9-4b92-4663-8211-d831e6cfad35\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" Oct 31 12:57:49.717064 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:49.717006 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5780a4a9-4b92-4663-8211-d831e6cfad35-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs\" (UID: \"5780a4a9-4b92-4663-8211-d831e6cfad35\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" Oct 31 12:57:49.818040 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:49.817958 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5780a4a9-4b92-4663-8211-d831e6cfad35-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs\" (UID: \"5780a4a9-4b92-4663-8211-d831e6cfad35\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" Oct 31 12:57:49.818040 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:49.817998 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5780a4a9-4b92-4663-8211-d831e6cfad35-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs\" (UID: \"5780a4a9-4b92-4663-8211-d831e6cfad35\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" Oct 31 12:57:49.818268 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:49.818111 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5780a4a9-4b92-4663-8211-d831e6cfad35-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs\" (UID: \"5780a4a9-4b92-4663-8211-d831e6cfad35\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" Oct 31 12:57:49.818268 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:49.818139 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5780a4a9-4b92-4663-8211-d831e6cfad35-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs\" (UID: \"5780a4a9-4b92-4663-8211-d831e6cfad35\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" Oct 31 12:57:49.818268 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:49.818183 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5780a4a9-4b92-4663-8211-d831e6cfad35-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs\" (UID: \"5780a4a9-4b92-4663-8211-d831e6cfad35\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" Oct 31 12:57:49.818268 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:49.818207 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5780a4a9-4b92-4663-8211-d831e6cfad35-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs\" (UID: \"5780a4a9-4b92-4663-8211-d831e6cfad35\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" Oct 31 12:57:49.818268 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:49.818239 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5780a4a9-4b92-4663-8211-d831e6cfad35-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs\" (UID: \"5780a4a9-4b92-4663-8211-d831e6cfad35\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" Oct 31 12:57:49.818538 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:49.818393 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5780a4a9-4b92-4663-8211-d831e6cfad35-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs\" (UID: \"5780a4a9-4b92-4663-8211-d831e6cfad35\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" Oct 31 12:57:49.818538 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:49.818422 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td26m\" (UniqueName: \"kubernetes.io/projected/5780a4a9-4b92-4663-8211-d831e6cfad35-kube-api-access-td26m\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs\" (UID: \"5780a4a9-4b92-4663-8211-d831e6cfad35\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" Oct 31 12:57:49.818538 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:49.818430 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5780a4a9-4b92-4663-8211-d831e6cfad35-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs\" (UID: \"5780a4a9-4b92-4663-8211-d831e6cfad35\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" Oct 31 12:57:49.818721 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:49.818550 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5780a4a9-4b92-4663-8211-d831e6cfad35-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs\" (UID: \"5780a4a9-4b92-4663-8211-d831e6cfad35\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" Oct 31 12:57:49.818811 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:49.818790 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5780a4a9-4b92-4663-8211-d831e6cfad35-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs\" (UID: \"5780a4a9-4b92-4663-8211-d831e6cfad35\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" Oct 31 12:57:49.819029 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:49.819001 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5780a4a9-4b92-4663-8211-d831e6cfad35-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs\" (UID: \"5780a4a9-4b92-4663-8211-d831e6cfad35\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" Oct 31 12:57:49.819379 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:49.819353 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5780a4a9-4b92-4663-8211-d831e6cfad35-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs\" (UID: \"5780a4a9-4b92-4663-8211-d831e6cfad35\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" Oct 31 12:57:49.820829 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:49.820802 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5780a4a9-4b92-4663-8211-d831e6cfad35-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs\" (UID: \"5780a4a9-4b92-4663-8211-d831e6cfad35\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" Oct 31 12:57:49.820937 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:49.820921 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5780a4a9-4b92-4663-8211-d831e6cfad35-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs\" (UID: \"5780a4a9-4b92-4663-8211-d831e6cfad35\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" Oct 31 12:57:49.830965 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:49.830939 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5780a4a9-4b92-4663-8211-d831e6cfad35-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs\" (UID: \"5780a4a9-4b92-4663-8211-d831e6cfad35\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" Oct 31 12:57:49.831123 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:49.831104 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td26m\" (UniqueName: \"kubernetes.io/projected/5780a4a9-4b92-4663-8211-d831e6cfad35-kube-api-access-td26m\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs\" (UID: \"5780a4a9-4b92-4663-8211-d831e6cfad35\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" Oct 31 12:57:49.940058 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:49.940026 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" Oct 31 12:57:50.022200 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:50.020910 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" event={"ID":"530ebf58-0d39-42be-879f-3c16bdf11d58","Type":"ContainerStarted","Data":"93bfd3842572f8a090f70df8e792f2756cd560efbc47a45b955c8090efdba7fb"} Oct 31 12:57:50.255886 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:50.255817 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" podStartSLOduration=6.255800496 podStartE2EDuration="6.255800496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:57:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:57:50.046971907 +0000 UTC m=+1520.754558203" watchObservedRunningTime="2026-10-31 12:57:50.255800496 +0000 UTC m=+1520.963386791" Oct 31 12:57:50.256731 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:50.256711 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs"] Oct 31 12:57:50.259045 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:57:50.259005 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5780a4a9_4b92_4663_8211_d831e6cfad35.slice/crio-5b38f39845cf8870c99116b099167825b5b9465519f41cc6901f607fc89cb3eb WatchSource:0}: Error finding container 5b38f39845cf8870c99116b099167825b5b9465519f41cc6901f607fc89cb3eb: Status 404 returned error can't find the container with id 5b38f39845cf8870c99116b099167825b5b9465519f41cc6901f607fc89cb3eb Oct 31 12:57:50.261837 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:50.261804 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:57:50.261941 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:50.261891 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:57:50.262001 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:50.261937 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:57:50.834963 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:50.834929 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" Oct 31 12:57:50.836752 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:50.836724 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 12:57:50.836901 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:50.836781 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" podUID="530ebf58-0d39-42be-879f-3c16bdf11d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 12:57:51.029490 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:51.029454 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" event={"ID":"5780a4a9-4b92-4663-8211-d831e6cfad35","Type":"ContainerStarted","Data":"c9b9d5baac20d34656141c2b23e95d059ddda8e5bbe706f8e10ebe8617e260b9"} Oct 31 12:57:51.029700 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:51.029513 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" event={"ID":"5780a4a9-4b92-4663-8211-d831e6cfad35","Type":"ContainerStarted","Data":"5b38f39845cf8870c99116b099167825b5b9465519f41cc6901f607fc89cb3eb"} Oct 31 12:57:51.054734 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:51.054681 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" podStartSLOduration=2.054662734 podStartE2EDuration="2.054662734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:57:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:57:51.05256293 +0000 UTC m=+1521.760149225" watchObservedRunningTime="2026-10-31 12:57:51.054662734 +0000 UTC m=+1521.762249026" Oct 31 12:57:51.840416 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:51.840006 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 12:57:51.840416 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:51.840057 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" podUID="530ebf58-0d39-42be-879f-3c16bdf11d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 12:57:51.941452 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:51.941417 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" Oct 31 12:57:51.942661 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:51.942470 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 12:57:51.942661 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:51.942556 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" podUID="5780a4a9-4b92-4663-8211-d831e6cfad35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 12:57:52.836394 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:52.833672 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 12:57:52.836394 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:52.833728 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" podUID="530ebf58-0d39-42be-879f-3c16bdf11d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 12:57:52.940896 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:52.940861 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 12:57:52.941373 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:52.940914 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" podUID="5780a4a9-4b92-4663-8211-d831e6cfad35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 12:57:52.969011 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:52.968984 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73503b-56954d7b47-p8brk" Oct 31 12:57:53.833796 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:53.833753 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 12:57:53.833974 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:53.833818 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" podUID="530ebf58-0d39-42be-879f-3c16bdf11d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 12:57:53.941286 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:53.941249 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 12:57:53.941707 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:53.941298 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" podUID="5780a4a9-4b92-4663-8211-d831e6cfad35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 12:57:54.838910 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:54.838193 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 12:57:54.838910 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:54.838274 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" podUID="530ebf58-0d39-42be-879f-3c16bdf11d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 12:57:54.943883 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:54.943824 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 12:57:54.944389 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:54.943917 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" podUID="5780a4a9-4b92-4663-8211-d831e6cfad35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 12:57:55.833580 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:55.833529 5693 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 12:57:55.833835 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:55.833592 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" podUID="530ebf58-0d39-42be-879f-3c16bdf11d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 12:57:55.940741 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:55.940698 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 12:57:55.940940 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:55.940780 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" podUID="5780a4a9-4b92-4663-8211-d831e6cfad35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 12:57:56.941402 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:56.940903 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 12:57:56.941402 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:56.940970 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" podUID="5780a4a9-4b92-4663-8211-d831e6cfad35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 12:57:57.039275 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:57.039241 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" Oct 31 12:57:57.039766 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:57.039731 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" Oct 31 12:57:57.040626 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:57.040603 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" Oct 31 12:57:58.300321 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:58.300287 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" Oct 31 12:57:58.300810 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:58.300694 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" Oct 31 12:57:58.301763 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:57:58.301739 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" Oct 31 12:58:18.691790 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:58:18.691715 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:58:19.619167 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:19.619134 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s"] Oct 31 12:58:19.623278 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:19.623258 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" Oct 31 12:58:19.644327 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:19.644290 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s"] Oct 31 12:58:19.805049 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:19.805016 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c723db-b336-48a6-a52b-df5174b4ab3d-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s\" (UID: \"99c723db-b336-48a6-a52b-df5174b4ab3d\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" Oct 31 12:58:19.805049 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:19.805052 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99c723db-b336-48a6-a52b-df5174b4ab3d-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s\" (UID: \"99c723db-b336-48a6-a52b-df5174b4ab3d\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" Oct 31 12:58:19.805702 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:19.805190 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99c723db-b336-48a6-a52b-df5174b4ab3d-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s\" (UID: \"99c723db-b336-48a6-a52b-df5174b4ab3d\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" Oct 31 12:58:19.805702 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:19.805224 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c723db-b336-48a6-a52b-df5174b4ab3d-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s\" (UID: \"99c723db-b336-48a6-a52b-df5174b4ab3d\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" Oct 31 12:58:19.805702 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:19.805243 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99c723db-b336-48a6-a52b-df5174b4ab3d-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s\" (UID: \"99c723db-b336-48a6-a52b-df5174b4ab3d\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" Oct 31 12:58:19.805702 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:19.805263 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7jpc\" (UniqueName: \"kubernetes.io/projected/99c723db-b336-48a6-a52b-df5174b4ab3d-kube-api-access-z7jpc\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s\" (UID: \"99c723db-b336-48a6-a52b-df5174b4ab3d\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" Oct 31 12:58:19.805702 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:19.805396 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99c723db-b336-48a6-a52b-df5174b4ab3d-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s\" (UID: \"99c723db-b336-48a6-a52b-df5174b4ab3d\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" Oct 31 12:58:19.805702 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:19.805443 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99c723db-b336-48a6-a52b-df5174b4ab3d-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s\" (UID: \"99c723db-b336-48a6-a52b-df5174b4ab3d\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" Oct 31 12:58:19.805702 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:19.805529 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99c723db-b336-48a6-a52b-df5174b4ab3d-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s\" (UID: \"99c723db-b336-48a6-a52b-df5174b4ab3d\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" Oct 31 12:58:19.906395 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:19.906362 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c723db-b336-48a6-a52b-df5174b4ab3d-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s\" (UID: \"99c723db-b336-48a6-a52b-df5174b4ab3d\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" Oct 31 12:58:19.906568 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:19.906407 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99c723db-b336-48a6-a52b-df5174b4ab3d-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s\" (UID: \"99c723db-b336-48a6-a52b-df5174b4ab3d\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" Oct 31 12:58:19.906568 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:19.906490 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99c723db-b336-48a6-a52b-df5174b4ab3d-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s\" (UID: \"99c723db-b336-48a6-a52b-df5174b4ab3d\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" Oct 31 12:58:19.906568 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:19.906514 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c723db-b336-48a6-a52b-df5174b4ab3d-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s\" (UID: \"99c723db-b336-48a6-a52b-df5174b4ab3d\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" Oct 31 12:58:19.906568 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:19.906541 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99c723db-b336-48a6-a52b-df5174b4ab3d-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s\" (UID: \"99c723db-b336-48a6-a52b-df5174b4ab3d\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" Oct 31 12:58:19.906568 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:19.906564 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7jpc\" (UniqueName: \"kubernetes.io/projected/99c723db-b336-48a6-a52b-df5174b4ab3d-kube-api-access-z7jpc\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s\" (UID: \"99c723db-b336-48a6-a52b-df5174b4ab3d\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" Oct 31 12:58:19.906887 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:19.906678 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99c723db-b336-48a6-a52b-df5174b4ab3d-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s\" (UID: \"99c723db-b336-48a6-a52b-df5174b4ab3d\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" Oct 31 12:58:19.906887 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:19.906726 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99c723db-b336-48a6-a52b-df5174b4ab3d-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s\" (UID: \"99c723db-b336-48a6-a52b-df5174b4ab3d\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" Oct 31 12:58:19.906887 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:19.906814 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99c723db-b336-48a6-a52b-df5174b4ab3d-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s\" (UID: \"99c723db-b336-48a6-a52b-df5174b4ab3d\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" Oct 31 12:58:19.907045 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:19.906911 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c723db-b336-48a6-a52b-df5174b4ab3d-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s\" (UID: \"99c723db-b336-48a6-a52b-df5174b4ab3d\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" Oct 31 12:58:19.907045 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:19.906945 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c723db-b336-48a6-a52b-df5174b4ab3d-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s\" (UID: \"99c723db-b336-48a6-a52b-df5174b4ab3d\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" Oct 31 12:58:19.907158 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:19.907037 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99c723db-b336-48a6-a52b-df5174b4ab3d-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s\" (UID: \"99c723db-b336-48a6-a52b-df5174b4ab3d\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" Oct 31 12:58:19.907303 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:19.907258 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99c723db-b336-48a6-a52b-df5174b4ab3d-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s\" (UID: \"99c723db-b336-48a6-a52b-df5174b4ab3d\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" Oct 31 12:58:19.907731 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:19.907708 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99c723db-b336-48a6-a52b-df5174b4ab3d-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s\" (UID: \"99c723db-b336-48a6-a52b-df5174b4ab3d\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" Oct 31 12:58:19.909419 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:19.909393 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99c723db-b336-48a6-a52b-df5174b4ab3d-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s\" (UID: \"99c723db-b336-48a6-a52b-df5174b4ab3d\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" Oct 31 12:58:19.909554 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:19.909537 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99c723db-b336-48a6-a52b-df5174b4ab3d-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s\" (UID: \"99c723db-b336-48a6-a52b-df5174b4ab3d\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" Oct 31 12:58:19.929174 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:19.929134 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99c723db-b336-48a6-a52b-df5174b4ab3d-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s\" (UID: \"99c723db-b336-48a6-a52b-df5174b4ab3d\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" Oct 31 12:58:19.929350 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:19.929334 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7jpc\" (UniqueName: \"kubernetes.io/projected/99c723db-b336-48a6-a52b-df5174b4ab3d-kube-api-access-z7jpc\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s\" (UID: \"99c723db-b336-48a6-a52b-df5174b4ab3d\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" Oct 31 12:58:19.939864 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:19.939839 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" Oct 31 12:58:20.234359 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:20.234330 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s"] Oct 31 12:58:20.236512 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:58:20.236487 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99c723db_b336_48a6_a52b_df5174b4ab3d.slice/crio-a9f8ad8229848efc7cc98ee3092ab432a60cac1b5d383c6cdc1604b11548df6f WatchSource:0}: Error finding container a9f8ad8229848efc7cc98ee3092ab432a60cac1b5d383c6cdc1604b11548df6f: Status 404 returned error can't find the container with id a9f8ad8229848efc7cc98ee3092ab432a60cac1b5d383c6cdc1604b11548df6f Oct 31 12:58:20.239088 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:20.239050 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:58:20.239194 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:20.239121 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:58:20.239194 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:20.239149 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:58:20.321892 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:20.321377 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" event={"ID":"99c723db-b336-48a6-a52b-df5174b4ab3d","Type":"ContainerStarted","Data":"f9d68877ba05896005c9b4fba27ef2270e2203345979dcc1614586007df04487"} Oct 31 12:58:20.321892 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:20.321439 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" event={"ID":"99c723db-b336-48a6-a52b-df5174b4ab3d","Type":"ContainerStarted","Data":"a9f8ad8229848efc7cc98ee3092ab432a60cac1b5d383c6cdc1604b11548df6f"} Oct 31 12:58:20.363452 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:20.363396 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" podStartSLOduration=1.363382399 podStartE2EDuration="1.363382399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:58:20.359771572 +0000 UTC m=+1551.067357892" watchObservedRunningTime="2026-10-31 12:58:20.363382399 +0000 UTC m=+1551.070968694" Oct 31 12:58:20.940011 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:20.939981 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" Oct 31 12:58:21.941775 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:21.941742 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 31 12:58:21.942245 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:21.941792 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" podUID="99c723db-b336-48a6-a52b-df5174b4ab3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 31 12:58:22.940935 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:22.940898 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 31 12:58:22.941110 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:22.940946 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" podUID="99c723db-b336-48a6-a52b-df5174b4ab3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 31 12:58:23.940523 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:23.940476 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 31 12:58:23.940990 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:23.940547 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" podUID="99c723db-b336-48a6-a52b-df5174b4ab3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 31 12:58:24.942083 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:24.942041 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 31 12:58:24.942593 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:24.942106 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" podUID="99c723db-b336-48a6-a52b-df5174b4ab3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 31 12:58:25.513274 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:25.513237 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-702419-f5484d7cd-4xm5c"] Oct 31 12:58:25.513581 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:25.513498 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-702419-f5484d7cd-4xm5c" podUID="32faa6d4-5f7a-449c-973d-819d5aa6e1a6" containerName="maas-api" containerID="cri-o://f1f773e94d2c4d2477ed05408b2b756f1edec796f90ebaf53beae6f15ce8cf15" gracePeriod=30 Oct 31 12:58:25.948852 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:25.947555 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 31 12:58:25.948852 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:25.947631 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" podUID="99c723db-b336-48a6-a52b-df5174b4ab3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 31 12:58:26.030795 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:26.030761 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-702419-f5484d7cd-4xm5c" Oct 31 12:58:26.051009 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:26.050979 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32faa6d4-5f7a-449c-973d-819d5aa6e1a6-kube-api-access-792r2\" (UniqueName: \"kubernetes.io/projected/32faa6d4-5f7a-449c-973d-819d5aa6e1a6-kube-api-access-792r2\") pod \"32faa6d4-5f7a-449c-973d-819d5aa6e1a6\" (UID: \"32faa6d4-5f7a-449c-973d-819d5aa6e1a6\") " Oct 31 12:58:26.051170 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:26.051099 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32faa6d4-5f7a-449c-973d-819d5aa6e1a6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32faa6d4-5f7a-449c-973d-819d5aa6e1a6-maas-api-tls\") pod \"32faa6d4-5f7a-449c-973d-819d5aa6e1a6\" (UID: \"32faa6d4-5f7a-449c-973d-819d5aa6e1a6\") " Oct 31 12:58:26.051239 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:26.051194 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32faa6d4-5f7a-449c-973d-819d5aa6e1a6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/32faa6d4-5f7a-449c-973d-819d5aa6e1a6-metrics-certs\") pod \"32faa6d4-5f7a-449c-973d-819d5aa6e1a6\" (UID: \"32faa6d4-5f7a-449c-973d-819d5aa6e1a6\") " Oct 31 12:58:26.051301 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:26.051243 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32faa6d4-5f7a-449c-973d-819d5aa6e1a6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32faa6d4-5f7a-449c-973d-819d5aa6e1a6-openshift-service-ca\") pod \"32faa6d4-5f7a-449c-973d-819d5aa6e1a6\" (UID: \"32faa6d4-5f7a-449c-973d-819d5aa6e1a6\") " Oct 31 12:58:26.051706 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:26.051681 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32faa6d4-5f7a-449c-973d-819d5aa6e1a6-openshift-service-ca" pod "32faa6d4-5f7a-449c-973d-819d5aa6e1a6" (UID: "32faa6d4-5f7a-449c-973d-819d5aa6e1a6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:58:26.051912 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:26.051894 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32faa6d4-5f7a-449c-973d-819d5aa6e1a6-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:26.054112 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:26.054071 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32faa6d4-5f7a-449c-973d-819d5aa6e1a6-metrics-certs" pod "32faa6d4-5f7a-449c-973d-819d5aa6e1a6" (UID: "32faa6d4-5f7a-449c-973d-819d5aa6e1a6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:58:26.054484 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:26.054461 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32faa6d4-5f7a-449c-973d-819d5aa6e1a6-kube-api-access-792r2" pod "32faa6d4-5f7a-449c-973d-819d5aa6e1a6" (UID: "32faa6d4-5f7a-449c-973d-819d5aa6e1a6"). InnerVolumeSpecName "kube-api-access-792r2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:58:26.054620 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:26.054601 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32faa6d4-5f7a-449c-973d-819d5aa6e1a6-maas-api-tls" pod "32faa6d4-5f7a-449c-973d-819d5aa6e1a6" (UID: "32faa6d4-5f7a-449c-973d-819d5aa6e1a6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:58:26.153661 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:26.153606 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-792r2\" (UniqueName: \"kubernetes.io/projected/32faa6d4-5f7a-449c-973d-819d5aa6e1a6-kube-api-access-792r2\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:26.153661 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:26.153633 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32faa6d4-5f7a-449c-973d-819d5aa6e1a6-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:26.153862 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:26.153673 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32faa6d4-5f7a-449c-973d-819d5aa6e1a6-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:26.378500 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:26.378414 5693 generic.go:356] "Generic (PLEG): container finished" podID="32faa6d4-5f7a-449c-973d-819d5aa6e1a6" containerID="f1f773e94d2c4d2477ed05408b2b756f1edec796f90ebaf53beae6f15ce8cf15" exitCode=0 Oct 31 12:58:26.378500 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:26.378469 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-702419-f5484d7cd-4xm5c" Oct 31 12:58:26.378500 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:26.378477 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-702419-f5484d7cd-4xm5c" event={"ID":"32faa6d4-5f7a-449c-973d-819d5aa6e1a6","Type":"ContainerDied","Data":"f1f773e94d2c4d2477ed05408b2b756f1edec796f90ebaf53beae6f15ce8cf15"} Oct 31 12:58:26.378801 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:26.378518 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-702419-f5484d7cd-4xm5c" event={"ID":"32faa6d4-5f7a-449c-973d-819d5aa6e1a6","Type":"ContainerDied","Data":"b1504661d1e126eafee53f6a8c14df4cb345562d998051af614cf5b882993d03"} Oct 31 12:58:26.378801 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:26.378538 5693 scope.go:122] "RemoveContainer" containerID="f1f773e94d2c4d2477ed05408b2b756f1edec796f90ebaf53beae6f15ce8cf15" Oct 31 12:58:26.389732 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:26.389711 5693 scope.go:122] "RemoveContainer" containerID="f1f773e94d2c4d2477ed05408b2b756f1edec796f90ebaf53beae6f15ce8cf15" Oct 31 12:58:26.390045 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:58:26.390025 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1f773e94d2c4d2477ed05408b2b756f1edec796f90ebaf53beae6f15ce8cf15\": container with ID starting with f1f773e94d2c4d2477ed05408b2b756f1edec796f90ebaf53beae6f15ce8cf15 not found: ID does not exist" containerID="f1f773e94d2c4d2477ed05408b2b756f1edec796f90ebaf53beae6f15ce8cf15" Oct 31 12:58:26.390113 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:26.390052 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1f773e94d2c4d2477ed05408b2b756f1edec796f90ebaf53beae6f15ce8cf15"} err="failed to get container status \"f1f773e94d2c4d2477ed05408b2b756f1edec796f90ebaf53beae6f15ce8cf15\": rpc error: code = NotFound desc = could not find container \"f1f773e94d2c4d2477ed05408b2b756f1edec796f90ebaf53beae6f15ce8cf15\": container with ID starting with f1f773e94d2c4d2477ed05408b2b756f1edec796f90ebaf53beae6f15ce8cf15 not found: ID does not exist" Oct 31 12:58:26.407291 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:26.407263 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-702419-f5484d7cd-4xm5c"] Oct 31 12:58:26.410332 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:26.410303 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-702419-f5484d7cd-4xm5c"] Oct 31 12:58:26.941212 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:26.941169 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 31 12:58:26.941416 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:26.941242 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" podUID="99c723db-b336-48a6-a52b-df5174b4ab3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 31 12:58:27.408054 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:27.408022 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32faa6d4-5f7a-449c-973d-819d5aa6e1a6" path="/var/lib/kubelet/pods/32faa6d4-5f7a-449c-973d-819d5aa6e1a6/volumes" Oct 31 12:58:27.946220 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:27.946188 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" Oct 31 12:58:27.947759 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:27.947738 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" Oct 31 12:58:27.948741 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:27.948721 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" Oct 31 12:58:31.073891 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:31.073861 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb"] Oct 31 12:58:31.074564 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:31.074542 5693 cpu_manager.go:414] "containerMap: removing container" podUID="32faa6d4-5f7a-449c-973d-819d5aa6e1a6" containerName="maas-api" Oct 31 12:58:31.074680 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:31.074565 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32faa6d4-5f7a-449c-973d-819d5aa6e1a6" containerName="maas-api" Oct 31 12:58:31.074751 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:31.074715 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="32faa6d4-5f7a-449c-973d-819d5aa6e1a6" containerName="maas-api" Oct 31 12:58:31.079920 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:31.078550 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" Oct 31 12:58:31.097214 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:31.097186 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-15c8ff-openshift-default-dockercfg-k8259\"" Oct 31 12:58:31.098860 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:31.098833 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb"] Oct 31 12:58:31.182237 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:31.182197 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dba5662-b598-4803-a569-eb3dd0ff025d-istio-podinfo\") pod \"e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb\" (UID: \"0dba5662-b598-4803-a569-eb3dd0ff025d\") " pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" Oct 31 12:58:31.182403 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:31.182250 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dba5662-b598-4803-a569-eb3dd0ff025d-istiod-ca-cert\") pod \"e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb\" (UID: \"0dba5662-b598-4803-a569-eb3dd0ff025d\") " pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" Oct 31 12:58:31.182403 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:31.182372 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dba5662-b598-4803-a569-eb3dd0ff025d-workload-certs\") pod \"e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb\" (UID: \"0dba5662-b598-4803-a569-eb3dd0ff025d\") " pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" Oct 31 12:58:31.182474 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:31.182407 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dba5662-b598-4803-a569-eb3dd0ff025d-workload-socket\") pod \"e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb\" (UID: \"0dba5662-b598-4803-a569-eb3dd0ff025d\") " pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" Oct 31 12:58:31.182508 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:31.182484 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n59vc\" (UniqueName: \"kubernetes.io/projected/0dba5662-b598-4803-a569-eb3dd0ff025d-kube-api-access-n59vc\") pod \"e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb\" (UID: \"0dba5662-b598-4803-a569-eb3dd0ff025d\") " pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" Oct 31 12:58:31.182550 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:31.182524 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dba5662-b598-4803-a569-eb3dd0ff025d-istio-token\") pod \"e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb\" (UID: \"0dba5662-b598-4803-a569-eb3dd0ff025d\") " pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" Oct 31 12:58:31.182618 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:31.182600 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dba5662-b598-4803-a569-eb3dd0ff025d-credential-socket\") pod \"e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb\" (UID: \"0dba5662-b598-4803-a569-eb3dd0ff025d\") " pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" Oct 31 12:58:31.182697 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:31.182684 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dba5662-b598-4803-a569-eb3dd0ff025d-istio-envoy\") pod \"e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb\" (UID: \"0dba5662-b598-4803-a569-eb3dd0ff025d\") " pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" Oct 31 12:58:31.182808 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:31.182795 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dba5662-b598-4803-a569-eb3dd0ff025d-istio-data\") pod \"e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb\" (UID: \"0dba5662-b598-4803-a569-eb3dd0ff025d\") " pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" Oct 31 12:58:31.294675 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:31.292519 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dba5662-b598-4803-a569-eb3dd0ff025d-workload-certs\") pod \"e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb\" (UID: \"0dba5662-b598-4803-a569-eb3dd0ff025d\") " pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" Oct 31 12:58:31.294675 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:31.292584 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dba5662-b598-4803-a569-eb3dd0ff025d-workload-socket\") pod \"e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb\" (UID: \"0dba5662-b598-4803-a569-eb3dd0ff025d\") " pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" Oct 31 12:58:31.294675 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:31.292683 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n59vc\" (UniqueName: \"kubernetes.io/projected/0dba5662-b598-4803-a569-eb3dd0ff025d-kube-api-access-n59vc\") pod \"e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb\" (UID: \"0dba5662-b598-4803-a569-eb3dd0ff025d\") " pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" Oct 31 12:58:31.294675 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:31.292730 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dba5662-b598-4803-a569-eb3dd0ff025d-istio-token\") pod \"e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb\" (UID: \"0dba5662-b598-4803-a569-eb3dd0ff025d\") " pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" Oct 31 12:58:31.294675 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:31.292800 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dba5662-b598-4803-a569-eb3dd0ff025d-credential-socket\") pod \"e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb\" (UID: \"0dba5662-b598-4803-a569-eb3dd0ff025d\") " pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" Oct 31 12:58:31.294675 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:31.292856 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dba5662-b598-4803-a569-eb3dd0ff025d-istio-envoy\") pod \"e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb\" (UID: \"0dba5662-b598-4803-a569-eb3dd0ff025d\") " pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" Oct 31 12:58:31.294675 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:31.292982 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dba5662-b598-4803-a569-eb3dd0ff025d-istio-data\") pod \"e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb\" (UID: \"0dba5662-b598-4803-a569-eb3dd0ff025d\") " pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" Oct 31 12:58:31.294675 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:31.293017 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dba5662-b598-4803-a569-eb3dd0ff025d-istio-podinfo\") pod \"e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb\" (UID: \"0dba5662-b598-4803-a569-eb3dd0ff025d\") " pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" Oct 31 12:58:31.294675 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:31.293065 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dba5662-b598-4803-a569-eb3dd0ff025d-istiod-ca-cert\") pod \"e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb\" (UID: \"0dba5662-b598-4803-a569-eb3dd0ff025d\") " pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" Oct 31 12:58:31.294675 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:31.293967 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dba5662-b598-4803-a569-eb3dd0ff025d-istiod-ca-cert\") pod \"e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb\" (UID: \"0dba5662-b598-4803-a569-eb3dd0ff025d\") " pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" Oct 31 12:58:31.294675 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:31.294246 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dba5662-b598-4803-a569-eb3dd0ff025d-workload-certs\") pod \"e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb\" (UID: \"0dba5662-b598-4803-a569-eb3dd0ff025d\") " pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" Oct 31 12:58:31.294675 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:31.294473 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dba5662-b598-4803-a569-eb3dd0ff025d-workload-socket\") pod \"e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb\" (UID: \"0dba5662-b598-4803-a569-eb3dd0ff025d\") " pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" Oct 31 12:58:31.302170 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:31.302124 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dba5662-b598-4803-a569-eb3dd0ff025d-credential-socket\") pod \"e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb\" (UID: \"0dba5662-b598-4803-a569-eb3dd0ff025d\") " pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" Oct 31 12:58:31.302758 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:31.302739 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dba5662-b598-4803-a569-eb3dd0ff025d-istio-data\") pod \"e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb\" (UID: \"0dba5662-b598-4803-a569-eb3dd0ff025d\") " pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" Oct 31 12:58:31.324806 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:31.323377 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n59vc\" (UniqueName: \"kubernetes.io/projected/0dba5662-b598-4803-a569-eb3dd0ff025d-kube-api-access-n59vc\") pod \"e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb\" (UID: \"0dba5662-b598-4803-a569-eb3dd0ff025d\") " pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" Oct 31 12:58:31.324806 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:31.324152 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dba5662-b598-4803-a569-eb3dd0ff025d-istio-podinfo\") pod \"e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb\" (UID: \"0dba5662-b598-4803-a569-eb3dd0ff025d\") " pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" Oct 31 12:58:31.324806 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:31.324483 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dba5662-b598-4803-a569-eb3dd0ff025d-istio-envoy\") pod \"e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb\" (UID: \"0dba5662-b598-4803-a569-eb3dd0ff025d\") " pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" Oct 31 12:58:31.327479 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:31.327452 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dba5662-b598-4803-a569-eb3dd0ff025d-istio-token\") pod \"e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb\" (UID: \"0dba5662-b598-4803-a569-eb3dd0ff025d\") " pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" Oct 31 12:58:31.406683 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:31.406620 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" Oct 31 12:58:31.698686 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:31.698631 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb"] Oct 31 12:58:31.699533 ip-10-0-131-157 kubenswrapper[5693]: W1031 12:58:31.699507 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dba5662_b598_4803_a569_eb3dd0ff025d.slice/crio-63845c8a0b55c4a0c1beaa85401746f1f98c907216df2f2b734e501a5b31cf7a WatchSource:0}: Error finding container 63845c8a0b55c4a0c1beaa85401746f1f98c907216df2f2b734e501a5b31cf7a: Status 404 returned error can't find the container with id 63845c8a0b55c4a0c1beaa85401746f1f98c907216df2f2b734e501a5b31cf7a Oct 31 12:58:31.701964 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:31.701920 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:58:31.702069 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:31.701999 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:58:31.702069 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:31.702048 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:58:32.420976 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:32.420946 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" event={"ID":"0dba5662-b598-4803-a569-eb3dd0ff025d","Type":"ContainerStarted","Data":"1b46af497bbe115b85c657708a6b7c867a77ac4c1889a87b11fd1a41eb8a50ac"} Oct 31 12:58:32.421378 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:32.420993 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" event={"ID":"0dba5662-b598-4803-a569-eb3dd0ff025d","Type":"ContainerStarted","Data":"63845c8a0b55c4a0c1beaa85401746f1f98c907216df2f2b734e501a5b31cf7a"} Oct 31 12:58:32.450373 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:32.450306 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" podStartSLOduration=1.4502881300000001 podStartE2EDuration="1.45028813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:58:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:58:32.446733825 +0000 UTC m=+1563.154320121" watchObservedRunningTime="2026-10-31 12:58:32.45028813 +0000 UTC m=+1563.157874425" Oct 31 12:58:33.408960 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:33.408927 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" Oct 31 12:58:33.412907 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:33.412883 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" Oct 31 12:58:33.427242 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:33.427218 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" Oct 31 12:58:33.428120 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:33.428101 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" Oct 31 12:58:36.862668 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:36.862110 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-73503b-56954d7b47-p8brk"] Oct 31 12:58:41.862762 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:41.862720 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73503b-56954d7b47-p8brk" podUID="895988e2-2481-4504-99b1-dbc84d806479" containerName="payload-pre-processing" containerID="cri-o://a6e336407318b61738a958119decf4f261e2ac3f05679fd33f4282f89c5902a5" gracePeriod=25 Oct 31 12:58:42.234726 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:42.234698 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73503b-56954d7b47-p8brk" Oct 31 12:58:42.338402 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:42.338358 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/895988e2-2481-4504-99b1-dbc84d806479-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/895988e2-2481-4504-99b1-dbc84d806479-plugins-config-volume\") pod \"895988e2-2481-4504-99b1-dbc84d806479\" (UID: \"895988e2-2481-4504-99b1-dbc84d806479\") " Oct 31 12:58:42.338602 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:42.338438 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/895988e2-2481-4504-99b1-dbc84d806479-kube-api-access-qzqkp\" (UniqueName: \"kubernetes.io/projected/895988e2-2481-4504-99b1-dbc84d806479-kube-api-access-qzqkp\") pod \"895988e2-2481-4504-99b1-dbc84d806479\" (UID: \"895988e2-2481-4504-99b1-dbc84d806479\") " Oct 31 12:58:42.338791 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:42.338767 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/895988e2-2481-4504-99b1-dbc84d806479-plugins-config-volume" pod "895988e2-2481-4504-99b1-dbc84d806479" (UID: "895988e2-2481-4504-99b1-dbc84d806479"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:58:42.339044 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:42.339028 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/895988e2-2481-4504-99b1-dbc84d806479-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:42.340848 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:42.340821 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/895988e2-2481-4504-99b1-dbc84d806479-kube-api-access-qzqkp" pod "895988e2-2481-4504-99b1-dbc84d806479" (UID: "895988e2-2481-4504-99b1-dbc84d806479"). InnerVolumeSpecName "kube-api-access-qzqkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:58:42.440080 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:42.440047 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzqkp\" (UniqueName: \"kubernetes.io/projected/895988e2-2481-4504-99b1-dbc84d806479-kube-api-access-qzqkp\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:42.531577 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:42.531475 5693 generic.go:356] "Generic (PLEG): container finished" podID="895988e2-2481-4504-99b1-dbc84d806479" containerID="a6e336407318b61738a958119decf4f261e2ac3f05679fd33f4282f89c5902a5" exitCode=0 Oct 31 12:58:42.531577 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:42.531523 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73503b-56954d7b47-p8brk" Oct 31 12:58:42.531577 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:42.531551 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73503b-56954d7b47-p8brk" event={"ID":"895988e2-2481-4504-99b1-dbc84d806479","Type":"ContainerDied","Data":"a6e336407318b61738a958119decf4f261e2ac3f05679fd33f4282f89c5902a5"} Oct 31 12:58:42.531905 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:42.531602 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73503b-56954d7b47-p8brk" event={"ID":"895988e2-2481-4504-99b1-dbc84d806479","Type":"ContainerDied","Data":"e02a486b20a3209840f07fb3d9c13625650e987189ac64859fa1a87f0ff89184"} Oct 31 12:58:42.531905 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:42.531625 5693 scope.go:122] "RemoveContainer" containerID="a6e336407318b61738a958119decf4f261e2ac3f05679fd33f4282f89c5902a5" Oct 31 12:58:42.544298 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:42.544269 5693 scope.go:122] "RemoveContainer" containerID="a6e336407318b61738a958119decf4f261e2ac3f05679fd33f4282f89c5902a5" Oct 31 12:58:42.544566 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:58:42.544542 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6e336407318b61738a958119decf4f261e2ac3f05679fd33f4282f89c5902a5\": container with ID starting with a6e336407318b61738a958119decf4f261e2ac3f05679fd33f4282f89c5902a5 not found: ID does not exist" containerID="a6e336407318b61738a958119decf4f261e2ac3f05679fd33f4282f89c5902a5" Oct 31 12:58:42.544630 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:42.544579 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6e336407318b61738a958119decf4f261e2ac3f05679fd33f4282f89c5902a5"} err="failed to get container status \"a6e336407318b61738a958119decf4f261e2ac3f05679fd33f4282f89c5902a5\": rpc error: code = NotFound desc = could not find container \"a6e336407318b61738a958119decf4f261e2ac3f05679fd33f4282f89c5902a5\": container with ID starting with a6e336407318b61738a958119decf4f261e2ac3f05679fd33f4282f89c5902a5 not found: ID does not exist" Oct 31 12:58:42.558515 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:42.558492 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-73503b-56954d7b47-p8brk"] Oct 31 12:58:42.565084 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:42.565055 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-73503b-56954d7b47-p8brk"] Oct 31 12:58:43.415515 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:58:43.415473 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="895988e2-2481-4504-99b1-dbc84d806479" path="/var/lib/kubelet/pods/895988e2-2481-4504-99b1-dbc84d806479/volumes" Oct 31 12:58:48.688671 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:58:48.688614 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:59:10.780144 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:10.780069 5693 generic.go:356] "Generic (PLEG): container finished" podID="0dba5662-b598-4803-a569-eb3dd0ff025d" containerID="1b46af497bbe115b85c657708a6b7c867a77ac4c1889a87b11fd1a41eb8a50ac" exitCode=0 Oct 31 12:59:10.780723 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:10.780252 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" event={"ID":"0dba5662-b598-4803-a569-eb3dd0ff025d","Type":"ContainerDied","Data":"1b46af497bbe115b85c657708a6b7c867a77ac4c1889a87b11fd1a41eb8a50ac"} Oct 31 12:59:10.780817 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:10.780768 5693 scope.go:122] "RemoveContainer" containerID="1b46af497bbe115b85c657708a6b7c867a77ac4c1889a87b11fd1a41eb8a50ac" Oct 31 12:59:10.784490 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:10.784437 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:59:10.784616 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:10.784548 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:59:10.784616 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:10.784596 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 12:59:11.826114 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:11.826079 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" event={"ID":"0dba5662-b598-4803-a569-eb3dd0ff025d","Type":"ContainerStarted","Data":"eff0e098057d4da209e336a214ebbb212f75687b4f6819979456de739005a5b2"} Oct 31 12:59:12.407399 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:12.407362 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" Oct 31 12:59:12.407658 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:12.407616 5693 patch_prober.go:29] interesting pod/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 31 12:59:12.407754 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:12.407676 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" podUID="0dba5662-b598-4803-a569-eb3dd0ff025d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 31 12:59:13.408460 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:13.408342 5693 patch_prober.go:29] interesting pod/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 31 12:59:13.408460 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:13.408403 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" podUID="0dba5662-b598-4803-a569-eb3dd0ff025d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 31 12:59:14.407578 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:14.407541 5693 patch_prober.go:29] interesting pod/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 31 12:59:14.407862 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:14.407593 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" podUID="0dba5662-b598-4803-a569-eb3dd0ff025d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 31 12:59:15.410095 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:15.410050 5693 patch_prober.go:29] interesting pod/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 31 12:59:15.410607 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:15.410119 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" podUID="0dba5662-b598-4803-a569-eb3dd0ff025d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 31 12:59:16.412663 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:16.411688 5693 patch_prober.go:29] interesting pod/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 31 12:59:16.412663 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:16.411768 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" podUID="0dba5662-b598-4803-a569-eb3dd0ff025d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 31 12:59:17.412154 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:17.412125 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" Oct 31 12:59:17.412507 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:17.412487 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" Oct 31 12:59:17.413270 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:17.413250 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" Oct 31 12:59:18.691873 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:59:18.691833 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 12:59:35.484327 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:35.484294 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb"] Oct 31 12:59:35.484871 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:35.484584 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" podUID="0dba5662-b598-4803-a569-eb3dd0ff025d" containerName="istio-proxy" containerID="cri-o://eff0e098057d4da209e336a214ebbb212f75687b4f6819979456de739005a5b2" gracePeriod=30 Oct 31 12:59:40.790140 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:40.790108 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" Oct 31 12:59:40.902098 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:40.902066 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0dba5662-b598-4803-a569-eb3dd0ff025d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dba5662-b598-4803-a569-eb3dd0ff025d-istio-podinfo\") pod \"0dba5662-b598-4803-a569-eb3dd0ff025d\" (UID: \"0dba5662-b598-4803-a569-eb3dd0ff025d\") " Oct 31 12:59:40.902274 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:40.902113 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dba5662-b598-4803-a569-eb3dd0ff025d-kube-api-access-n59vc\" (UniqueName: \"kubernetes.io/projected/0dba5662-b598-4803-a569-eb3dd0ff025d-kube-api-access-n59vc\") pod \"0dba5662-b598-4803-a569-eb3dd0ff025d\" (UID: \"0dba5662-b598-4803-a569-eb3dd0ff025d\") " Oct 31 12:59:40.902274 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:40.902246 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dba5662-b598-4803-a569-eb3dd0ff025d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dba5662-b598-4803-a569-eb3dd0ff025d-workload-certs\") pod \"0dba5662-b598-4803-a569-eb3dd0ff025d\" (UID: \"0dba5662-b598-4803-a569-eb3dd0ff025d\") " Oct 31 12:59:40.902366 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:40.902327 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dba5662-b598-4803-a569-eb3dd0ff025d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dba5662-b598-4803-a569-eb3dd0ff025d-istio-data\") pod \"0dba5662-b598-4803-a569-eb3dd0ff025d\" (UID: \"0dba5662-b598-4803-a569-eb3dd0ff025d\") " Oct 31 12:59:40.902442 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:40.902421 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dba5662-b598-4803-a569-eb3dd0ff025d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dba5662-b598-4803-a569-eb3dd0ff025d-istiod-ca-cert\") pod \"0dba5662-b598-4803-a569-eb3dd0ff025d\" (UID: \"0dba5662-b598-4803-a569-eb3dd0ff025d\") " Oct 31 12:59:40.902518 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:40.902460 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dba5662-b598-4803-a569-eb3dd0ff025d-istio-token\" (UniqueName: \"kubernetes.io/projected/0dba5662-b598-4803-a569-eb3dd0ff025d-istio-token\") pod \"0dba5662-b598-4803-a569-eb3dd0ff025d\" (UID: \"0dba5662-b598-4803-a569-eb3dd0ff025d\") " Oct 31 12:59:40.902518 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:40.902474 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dba5662-b598-4803-a569-eb3dd0ff025d-workload-certs" pod "0dba5662-b598-4803-a569-eb3dd0ff025d" (UID: "0dba5662-b598-4803-a569-eb3dd0ff025d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:40.902627 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:40.902592 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dba5662-b598-4803-a569-eb3dd0ff025d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dba5662-b598-4803-a569-eb3dd0ff025d-credential-socket\") pod \"0dba5662-b598-4803-a569-eb3dd0ff025d\" (UID: \"0dba5662-b598-4803-a569-eb3dd0ff025d\") " Oct 31 12:59:40.902627 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:40.902592 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dba5662-b598-4803-a569-eb3dd0ff025d-istio-data" pod "0dba5662-b598-4803-a569-eb3dd0ff025d" (UID: "0dba5662-b598-4803-a569-eb3dd0ff025d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:40.902762 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:40.902699 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dba5662-b598-4803-a569-eb3dd0ff025d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dba5662-b598-4803-a569-eb3dd0ff025d-workload-socket\") pod \"0dba5662-b598-4803-a569-eb3dd0ff025d\" (UID: \"0dba5662-b598-4803-a569-eb3dd0ff025d\") " Oct 31 12:59:40.902762 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:40.902719 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dba5662-b598-4803-a569-eb3dd0ff025d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dba5662-b598-4803-a569-eb3dd0ff025d-istio-envoy\") pod \"0dba5662-b598-4803-a569-eb3dd0ff025d\" (UID: \"0dba5662-b598-4803-a569-eb3dd0ff025d\") " Oct 31 12:59:40.902876 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:40.902854 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dba5662-b598-4803-a569-eb3dd0ff025d-istiod-ca-cert" pod "0dba5662-b598-4803-a569-eb3dd0ff025d" (UID: "0dba5662-b598-4803-a569-eb3dd0ff025d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:59:40.902935 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:40.902855 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dba5662-b598-4803-a569-eb3dd0ff025d-credential-socket" pod "0dba5662-b598-4803-a569-eb3dd0ff025d" (UID: "0dba5662-b598-4803-a569-eb3dd0ff025d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:40.902989 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:40.902950 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dba5662-b598-4803-a569-eb3dd0ff025d-workload-socket" pod "0dba5662-b598-4803-a569-eb3dd0ff025d" (UID: "0dba5662-b598-4803-a569-eb3dd0ff025d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:40.903300 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:40.903283 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dba5662-b598-4803-a569-eb3dd0ff025d-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:40.903373 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:40.903306 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dba5662-b598-4803-a569-eb3dd0ff025d-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:40.903373 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:40.903320 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dba5662-b598-4803-a569-eb3dd0ff025d-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:40.903373 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:40.903333 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dba5662-b598-4803-a569-eb3dd0ff025d-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:40.903373 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:40.903347 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dba5662-b598-4803-a569-eb3dd0ff025d-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:40.905446 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:40.905418 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dba5662-b598-4803-a569-eb3dd0ff025d-istio-envoy" pod "0dba5662-b598-4803-a569-eb3dd0ff025d" (UID: "0dba5662-b598-4803-a569-eb3dd0ff025d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:40.905575 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:40.905448 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dba5662-b598-4803-a569-eb3dd0ff025d-kube-api-access-n59vc" pod "0dba5662-b598-4803-a569-eb3dd0ff025d" (UID: "0dba5662-b598-4803-a569-eb3dd0ff025d"). InnerVolumeSpecName "kube-api-access-n59vc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:40.905706 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:40.905690 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0dba5662-b598-4803-a569-eb3dd0ff025d-istio-podinfo" pod "0dba5662-b598-4803-a569-eb3dd0ff025d" (UID: "0dba5662-b598-4803-a569-eb3dd0ff025d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:59:40.905774 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:40.905725 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dba5662-b598-4803-a569-eb3dd0ff025d-istio-token" pod "0dba5662-b598-4803-a569-eb3dd0ff025d" (UID: "0dba5662-b598-4803-a569-eb3dd0ff025d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:41.005206 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:41.005134 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dba5662-b598-4803-a569-eb3dd0ff025d-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:41.005206 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:41.005161 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n59vc\" (UniqueName: \"kubernetes.io/projected/0dba5662-b598-4803-a569-eb3dd0ff025d-kube-api-access-n59vc\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:41.005206 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:41.005173 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dba5662-b598-4803-a569-eb3dd0ff025d-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:41.005206 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:41.005182 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dba5662-b598-4803-a569-eb3dd0ff025d-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:41.100534 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:41.100504 5693 generic.go:356] "Generic (PLEG): container finished" podID="0dba5662-b598-4803-a569-eb3dd0ff025d" containerID="eff0e098057d4da209e336a214ebbb212f75687b4f6819979456de739005a5b2" exitCode=0 Oct 31 12:59:41.100750 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:41.100554 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" Oct 31 12:59:41.100750 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:41.100596 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" event={"ID":"0dba5662-b598-4803-a569-eb3dd0ff025d","Type":"ContainerDied","Data":"eff0e098057d4da209e336a214ebbb212f75687b4f6819979456de739005a5b2"} Oct 31 12:59:41.100750 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:41.100666 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb" event={"ID":"0dba5662-b598-4803-a569-eb3dd0ff025d","Type":"ContainerDied","Data":"63845c8a0b55c4a0c1beaa85401746f1f98c907216df2f2b734e501a5b31cf7a"} Oct 31 12:59:41.100750 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:41.100689 5693 scope.go:122] "RemoveContainer" containerID="eff0e098057d4da209e336a214ebbb212f75687b4f6819979456de739005a5b2" Oct 31 12:59:41.120026 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:41.119996 5693 scope.go:122] "RemoveContainer" containerID="1b46af497bbe115b85c657708a6b7c867a77ac4c1889a87b11fd1a41eb8a50ac" Oct 31 12:59:41.130123 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:41.130103 5693 scope.go:122] "RemoveContainer" containerID="eff0e098057d4da209e336a214ebbb212f75687b4f6819979456de739005a5b2" Oct 31 12:59:41.130426 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:59:41.130405 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eff0e098057d4da209e336a214ebbb212f75687b4f6819979456de739005a5b2\": container with ID starting with eff0e098057d4da209e336a214ebbb212f75687b4f6819979456de739005a5b2 not found: ID does not exist" containerID="eff0e098057d4da209e336a214ebbb212f75687b4f6819979456de739005a5b2" Oct 31 12:59:41.130508 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:41.130435 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eff0e098057d4da209e336a214ebbb212f75687b4f6819979456de739005a5b2"} err="failed to get container status \"eff0e098057d4da209e336a214ebbb212f75687b4f6819979456de739005a5b2\": rpc error: code = NotFound desc = could not find container \"eff0e098057d4da209e336a214ebbb212f75687b4f6819979456de739005a5b2\": container with ID starting with eff0e098057d4da209e336a214ebbb212f75687b4f6819979456de739005a5b2 not found: ID does not exist" Oct 31 12:59:41.130508 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:41.130451 5693 scope.go:122] "RemoveContainer" containerID="1b46af497bbe115b85c657708a6b7c867a77ac4c1889a87b11fd1a41eb8a50ac" Oct 31 12:59:41.130752 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:59:41.130726 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b46af497bbe115b85c657708a6b7c867a77ac4c1889a87b11fd1a41eb8a50ac\": container with ID starting with 1b46af497bbe115b85c657708a6b7c867a77ac4c1889a87b11fd1a41eb8a50ac not found: ID does not exist" containerID="1b46af497bbe115b85c657708a6b7c867a77ac4c1889a87b11fd1a41eb8a50ac" Oct 31 12:59:41.130858 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:41.130758 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b46af497bbe115b85c657708a6b7c867a77ac4c1889a87b11fd1a41eb8a50ac"} err="failed to get container status \"1b46af497bbe115b85c657708a6b7c867a77ac4c1889a87b11fd1a41eb8a50ac\": rpc error: code = NotFound desc = could not find container \"1b46af497bbe115b85c657708a6b7c867a77ac4c1889a87b11fd1a41eb8a50ac\": container with ID starting with 1b46af497bbe115b85c657708a6b7c867a77ac4c1889a87b11fd1a41eb8a50ac not found: ID does not exist" Oct 31 12:59:41.132732 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:41.132714 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb"] Oct 31 12:59:41.138428 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:41.138406 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-15c8ff-openshift-default-78986dcbc-6vqfb"] Oct 31 12:59:41.408517 ip-10-0-131-157 kubenswrapper[5693]: I1031 12:59:41.408486 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dba5662-b598-4803-a569-eb3dd0ff025d" path="/var/lib/kubelet/pods/0dba5662-b598-4803-a569-eb3dd0ff025d/volumes" Oct 31 12:59:48.687433 ip-10-0-131-157 kubenswrapper[5693]: E1031 12:59:48.687396 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:00:00.166120 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:00.166087 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890860-pvk6z"] Oct 31 13:00:00.166674 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:00.166630 5693 cpu_manager.go:414] "containerMap: removing container" podUID="0dba5662-b598-4803-a569-eb3dd0ff025d" containerName="istio-proxy" Oct 31 13:00:00.166674 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:00.166657 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dba5662-b598-4803-a569-eb3dd0ff025d" containerName="istio-proxy" Oct 31 13:00:00.166674 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:00.166674 5693 cpu_manager.go:414] "containerMap: removing container" podUID="895988e2-2481-4504-99b1-dbc84d806479" containerName="payload-pre-processing" Oct 31 13:00:00.166793 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:00.166681 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="895988e2-2481-4504-99b1-dbc84d806479" containerName="payload-pre-processing" Oct 31 13:00:00.166793 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:00.166783 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dba5662-b598-4803-a569-eb3dd0ff025d" containerName="istio-proxy" Oct 31 13:00:00.166793 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:00.166791 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="895988e2-2481-4504-99b1-dbc84d806479" containerName="payload-pre-processing" Oct 31 13:00:00.166885 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:00.166797 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dba5662-b598-4803-a569-eb3dd0ff025d" containerName="istio-proxy" Oct 31 13:00:00.170181 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:00.170160 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890860-pvk6z" Oct 31 13:00:00.193759 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:00.193732 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890860-pvk6z"] Oct 31 13:00:00.195757 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:00.195729 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:00:00.212881 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:00.212852 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890860-zq9vx"] Oct 31 13:00:00.213761 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:00.213737 5693 cpu_manager.go:414] "containerMap: removing container" podUID="0dba5662-b598-4803-a569-eb3dd0ff025d" containerName="istio-proxy" Oct 31 13:00:00.213761 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:00.213759 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dba5662-b598-4803-a569-eb3dd0ff025d" containerName="istio-proxy" Oct 31 13:00:00.217798 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:00.217776 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890860-zq9vx" Oct 31 13:00:00.230314 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:00.228654 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890860-zq9vx"] Oct 31 13:00:00.354422 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:00.354381 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4a060ad-6376-4030-9a91-b5ff30e561d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890860-zq9vx\" (UID: \"d4a060ad-6376-4030-9a91-b5ff30e561d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890860-zq9vx" Oct 31 13:00:00.354607 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:00.354474 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5669ae43-7d6d-47df-a7e9-db2a0a000f88-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890860-pvk6z\" (UID: \"5669ae43-7d6d-47df-a7e9-db2a0a000f88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890860-pvk6z" Oct 31 13:00:00.354607 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:00.354501 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcvhk\" (UniqueName: \"kubernetes.io/projected/5669ae43-7d6d-47df-a7e9-db2a0a000f88-kube-api-access-dcvhk\") pod \"maas-api-key-cleanup-29890860-pvk6z\" (UID: \"5669ae43-7d6d-47df-a7e9-db2a0a000f88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890860-pvk6z" Oct 31 13:00:00.354731 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:00.354660 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9m78\" (UniqueName: \"kubernetes.io/projected/d4a060ad-6376-4030-9a91-b5ff30e561d8-kube-api-access-l9m78\") pod \"maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890860-zq9vx\" (UID: \"d4a060ad-6376-4030-9a91-b5ff30e561d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890860-zq9vx" Oct 31 13:00:00.455927 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:00.455830 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4a060ad-6376-4030-9a91-b5ff30e561d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890860-zq9vx\" (UID: \"d4a060ad-6376-4030-9a91-b5ff30e561d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890860-zq9vx" Oct 31 13:00:00.455927 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:00.455881 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5669ae43-7d6d-47df-a7e9-db2a0a000f88-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890860-pvk6z\" (UID: \"5669ae43-7d6d-47df-a7e9-db2a0a000f88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890860-pvk6z" Oct 31 13:00:00.455927 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:00.455913 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcvhk\" (UniqueName: \"kubernetes.io/projected/5669ae43-7d6d-47df-a7e9-db2a0a000f88-kube-api-access-dcvhk\") pod \"maas-api-key-cleanup-29890860-pvk6z\" (UID: \"5669ae43-7d6d-47df-a7e9-db2a0a000f88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890860-pvk6z" Oct 31 13:00:00.456211 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:00.456061 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9m78\" (UniqueName: \"kubernetes.io/projected/d4a060ad-6376-4030-9a91-b5ff30e561d8-kube-api-access-l9m78\") pod \"maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890860-zq9vx\" (UID: \"d4a060ad-6376-4030-9a91-b5ff30e561d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890860-zq9vx" Oct 31 13:00:00.456555 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:00.456530 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4a060ad-6376-4030-9a91-b5ff30e561d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890860-zq9vx\" (UID: \"d4a060ad-6376-4030-9a91-b5ff30e561d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890860-zq9vx" Oct 31 13:00:00.456695 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:00.456591 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5669ae43-7d6d-47df-a7e9-db2a0a000f88-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890860-pvk6z\" (UID: \"5669ae43-7d6d-47df-a7e9-db2a0a000f88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890860-pvk6z" Oct 31 13:00:00.468434 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:00.468403 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcvhk\" (UniqueName: \"kubernetes.io/projected/5669ae43-7d6d-47df-a7e9-db2a0a000f88-kube-api-access-dcvhk\") pod \"maas-api-key-cleanup-29890860-pvk6z\" (UID: \"5669ae43-7d6d-47df-a7e9-db2a0a000f88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890860-pvk6z" Oct 31 13:00:00.468434 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:00.468412 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9m78\" (UniqueName: \"kubernetes.io/projected/d4a060ad-6376-4030-9a91-b5ff30e561d8-kube-api-access-l9m78\") pod \"maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890860-zq9vx\" (UID: \"d4a060ad-6376-4030-9a91-b5ff30e561d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890860-zq9vx" Oct 31 13:00:00.483291 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:00.483269 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890860-pvk6z" Oct 31 13:00:00.542528 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:00.542241 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890860-zq9vx" Oct 31 13:00:00.771737 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:00.765824 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890860-pvk6z"] Oct 31 13:00:00.924882 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:00.924857 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890860-zq9vx"] Oct 31 13:00:00.925761 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:00:00.925739 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4a060ad_6376_4030_9a91_b5ff30e561d8.slice/crio-d260ea9406f669913a30661c376b95a803dc71f7dc54896b489df1f5627858d9 WatchSource:0}: Error finding container d260ea9406f669913a30661c376b95a803dc71f7dc54896b489df1f5627858d9: Status 404 returned error can't find the container with id d260ea9406f669913a30661c376b95a803dc71f7dc54896b489df1f5627858d9 Oct 31 13:00:01.252490 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:01.252456 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890860-zq9vx" event={"ID":"d4a060ad-6376-4030-9a91-b5ff30e561d8","Type":"ContainerStarted","Data":"d260ea9406f669913a30661c376b95a803dc71f7dc54896b489df1f5627858d9"} Oct 31 13:00:01.253719 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:01.253695 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890860-pvk6z" event={"ID":"5669ae43-7d6d-47df-a7e9-db2a0a000f88","Type":"ContainerStarted","Data":"73fd18960bfd5cb97a8e46bc4f346d59839d9d05c15c4ddc1a3ecbad38538554"} Oct 31 13:00:04.304759 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:04.304711 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3128e5-85c845bff5-tw7sl"] Oct 31 13:00:04.310856 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:04.310829 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3128e5-85c845bff5-tw7sl" Oct 31 13:00:04.367039 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:04.365746 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnnz4\" (UniqueName: \"kubernetes.io/projected/f5662f90-e280-43aa-8f47-4957c0c14431-kube-api-access-fnnz4\") pod \"payload-pre-processing-e2e-api-a-3128e5-85c845bff5-tw7sl\" (UID: \"f5662f90-e280-43aa-8f47-4957c0c14431\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3128e5-85c845bff5-tw7sl" Oct 31 13:00:04.367039 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:04.365980 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5662f90-e280-43aa-8f47-4957c0c14431-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3128e5-85c845bff5-tw7sl\" (UID: \"f5662f90-e280-43aa-8f47-4957c0c14431\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3128e5-85c845bff5-tw7sl" Oct 31 13:00:04.369315 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:04.367854 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-3128e5\"" Oct 31 13:00:04.369315 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:04.369060 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3128e5-85c845bff5-tw7sl"] Oct 31 13:00:04.477672 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:04.475575 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5662f90-e280-43aa-8f47-4957c0c14431-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3128e5-85c845bff5-tw7sl\" (UID: \"f5662f90-e280-43aa-8f47-4957c0c14431\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3128e5-85c845bff5-tw7sl" Oct 31 13:00:04.477672 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:04.475739 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnnz4\" (UniqueName: \"kubernetes.io/projected/f5662f90-e280-43aa-8f47-4957c0c14431-kube-api-access-fnnz4\") pod \"payload-pre-processing-e2e-api-a-3128e5-85c845bff5-tw7sl\" (UID: \"f5662f90-e280-43aa-8f47-4957c0c14431\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3128e5-85c845bff5-tw7sl" Oct 31 13:00:04.496534 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:04.481883 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5662f90-e280-43aa-8f47-4957c0c14431-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3128e5-85c845bff5-tw7sl\" (UID: \"f5662f90-e280-43aa-8f47-4957c0c14431\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3128e5-85c845bff5-tw7sl" Oct 31 13:00:04.541661 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:04.537792 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnnz4\" (UniqueName: \"kubernetes.io/projected/f5662f90-e280-43aa-8f47-4957c0c14431-kube-api-access-fnnz4\") pod \"payload-pre-processing-e2e-api-a-3128e5-85c845bff5-tw7sl\" (UID: \"f5662f90-e280-43aa-8f47-4957c0c14431\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3128e5-85c845bff5-tw7sl" Oct 31 13:00:04.649939 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:04.649900 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3128e5-85c845bff5-tw7sl" Oct 31 13:00:05.015998 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:05.015964 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3128e5-85c845bff5-tw7sl"] Oct 31 13:00:05.017343 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:00:05.017310 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5662f90_e280_43aa_8f47_4957c0c14431.slice/crio-29ad09f7d95e20d953a18738b1678cbad024203cece879da92ed82d22553dabe WatchSource:0}: Error finding container 29ad09f7d95e20d953a18738b1678cbad024203cece879da92ed82d22553dabe: Status 404 returned error can't find the container with id 29ad09f7d95e20d953a18738b1678cbad024203cece879da92ed82d22553dabe Oct 31 13:00:05.290563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:05.290479 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3128e5-85c845bff5-tw7sl" event={"ID":"f5662f90-e280-43aa-8f47-4957c0c14431","Type":"ContainerStarted","Data":"56ec4611c7a72b7f1ef6cb17d77a88d03cfa6620c00fff21606a3f4bf9d0d582"} Oct 31 13:00:05.290563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:05.290540 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3128e5-85c845bff5-tw7sl" event={"ID":"f5662f90-e280-43aa-8f47-4957c0c14431","Type":"ContainerStarted","Data":"29ad09f7d95e20d953a18738b1678cbad024203cece879da92ed82d22553dabe"} Oct 31 13:00:05.290806 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:05.290738 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3128e5-85c845bff5-tw7sl" Oct 31 13:00:06.299703 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:06.299659 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890860-zq9vx" event={"ID":"d4a060ad-6376-4030-9a91-b5ff30e561d8","Type":"ContainerStarted","Data":"f04e3034bce7738df089b95509d51be56d5235a17ce64652c70252003db2975c"} Oct 31 13:00:06.319303 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:06.319251 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890860-zq9vx" podStartSLOduration=1.393030346 podStartE2EDuration="6.319233967s" totalImagesPullingTime="4.926203621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:06.318960685 +0000 UTC m=+1657.026546982" watchObservedRunningTime="2026-10-31 13:00:06.319233967 +0000 UTC m=+1657.026820264" Oct 31 13:00:06.321031 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:06.320994 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3128e5-85c845bff5-tw7sl" podStartSLOduration=2.320983018 podStartE2EDuration="2.320983018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:05.3193992 +0000 UTC m=+1656.026985495" watchObservedRunningTime="2026-10-31 13:00:06.320983018 +0000 UTC m=+1657.028569312" Oct 31 13:00:07.309997 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:07.309955 5693 generic.go:356] "Generic (PLEG): container finished" podID="5669ae43-7d6d-47df-a7e9-db2a0a000f88" containerID="57b4eeadde3249d96a56547d050de4c8f8c26e60f91fc789d56fcab0f3728a1a" exitCode=60 Oct 31 13:00:07.310458 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:07.310061 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890860-pvk6z" event={"ID":"5669ae43-7d6d-47df-a7e9-db2a0a000f88","Type":"ContainerDied","Data":"57b4eeadde3249d96a56547d050de4c8f8c26e60f91fc789d56fcab0f3728a1a"} Oct 31 13:00:07.310458 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:07.310367 5693 scope.go:122] "RemoveContainer" containerID="57b4eeadde3249d96a56547d050de4c8f8c26e60f91fc789d56fcab0f3728a1a" Oct 31 13:00:07.316006 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:07.315964 5693 generic.go:356] "Generic (PLEG): container finished" podID="d4a060ad-6376-4030-9a91-b5ff30e561d8" containerID="f04e3034bce7738df089b95509d51be56d5235a17ce64652c70252003db2975c" exitCode=60 Oct 31 13:00:07.316147 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:07.316013 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890860-zq9vx" event={"ID":"d4a060ad-6376-4030-9a91-b5ff30e561d8","Type":"ContainerDied","Data":"f04e3034bce7738df089b95509d51be56d5235a17ce64652c70252003db2975c"} Oct 31 13:00:07.316410 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:07.316395 5693 scope.go:122] "RemoveContainer" containerID="f04e3034bce7738df089b95509d51be56d5235a17ce64652c70252003db2975c" Oct 31 13:00:08.383199 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:08.380244 5693 generic.go:356] "Generic (PLEG): container finished" podID="5669ae43-7d6d-47df-a7e9-db2a0a000f88" containerID="300c7328755baa543dc5d331aee3c1b56146a15459dc4596cc98c05fda7e11c5" exitCode=60 Oct 31 13:00:08.383199 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:08.380570 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890860-pvk6z" event={"ID":"5669ae43-7d6d-47df-a7e9-db2a0a000f88","Type":"ContainerDied","Data":"300c7328755baa543dc5d331aee3c1b56146a15459dc4596cc98c05fda7e11c5"} Oct 31 13:00:08.383199 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:08.380709 5693 scope.go:122] "RemoveContainer" containerID="57b4eeadde3249d96a56547d050de4c8f8c26e60f91fc789d56fcab0f3728a1a" Oct 31 13:00:08.383199 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:08.381589 5693 scope.go:122] "RemoveContainer" containerID="300c7328755baa543dc5d331aee3c1b56146a15459dc4596cc98c05fda7e11c5" Oct 31 13:00:08.383199 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:00:08.382153 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890860-pvk6z_odh-ai-gateway-infra(5669ae43-7d6d-47df-a7e9-db2a0a000f88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890860-pvk6z" podUID="5669ae43-7d6d-47df-a7e9-db2a0a000f88" Oct 31 13:00:08.387458 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:08.387433 5693 generic.go:356] "Generic (PLEG): container finished" podID="d4a060ad-6376-4030-9a91-b5ff30e561d8" containerID="a0737bd0f8ef01ce104eca4cf3b729e09b82705df038ef945fb19a30cf2fed96" exitCode=60 Oct 31 13:00:08.387613 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:08.387500 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890860-zq9vx" event={"ID":"d4a060ad-6376-4030-9a91-b5ff30e561d8","Type":"ContainerDied","Data":"a0737bd0f8ef01ce104eca4cf3b729e09b82705df038ef945fb19a30cf2fed96"} Oct 31 13:00:08.388331 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:08.388299 5693 scope.go:122] "RemoveContainer" containerID="a0737bd0f8ef01ce104eca4cf3b729e09b82705df038ef945fb19a30cf2fed96" Oct 31 13:00:08.388946 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:00:08.388919 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890860-zq9vx_odh-ai-gateway-infra(d4a060ad-6376-4030-9a91-b5ff30e561d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890860-zq9vx" podUID="d4a060ad-6376-4030-9a91-b5ff30e561d8" Oct 31 13:00:08.422457 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:08.420587 5693 scope.go:122] "RemoveContainer" containerID="f04e3034bce7738df089b95509d51be56d5235a17ce64652c70252003db2975c" Oct 31 13:00:08.495357 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:08.495314 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-7bddf9d89f-jznzk"] Oct 31 13:00:08.501523 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:08.501496 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-cc9d58849-mgh7t"] Oct 31 13:00:08.501718 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:08.501678 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-7bddf9d89f-jznzk" Oct 31 13:00:08.507806 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:08.507083 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-cc9d58849-mgh7t" Oct 31 13:00:08.511351 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:08.511327 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-bc9ae8\"" Oct 31 13:00:08.538709 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:08.511961 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-bc9ae8-dockercfg-qtxht\"" Oct 31 13:00:08.538709 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:08.515699 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-7bddf9d89f-jznzk"] Oct 31 13:00:08.538709 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:08.528836 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-cc9d58849-mgh7t"] Oct 31 13:00:08.550873 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:08.550551 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87125609-1ebd-43c1-bdb2-d389c4756467-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bc9ae8-7bddf9d89f-jznzk\" (UID: \"87125609-1ebd-43c1-bdb2-d389c4756467\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-7bddf9d89f-jznzk" Oct 31 13:00:08.550873 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:08.550750 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cef1f7a7-aeae-4841-b5f0-8b54c3e9d768-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bc9ae8-cc9d58849-mgh7t\" (UID: \"cef1f7a7-aeae-4841-b5f0-8b54c3e9d768\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-cc9d58849-mgh7t" Oct 31 13:00:08.550873 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:08.550777 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlm29\" (UniqueName: \"kubernetes.io/projected/cef1f7a7-aeae-4841-b5f0-8b54c3e9d768-kube-api-access-dlm29\") pod \"payload-processing-e2e-ipp-a-bc9ae8-cc9d58849-mgh7t\" (UID: \"cef1f7a7-aeae-4841-b5f0-8b54c3e9d768\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-cc9d58849-mgh7t" Oct 31 13:00:08.550873 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:08.550807 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-646wr\" (UniqueName: \"kubernetes.io/projected/87125609-1ebd-43c1-bdb2-d389c4756467-kube-api-access-646wr\") pod \"payload-pre-processing-e2e-ipp-a-bc9ae8-7bddf9d89f-jznzk\" (UID: \"87125609-1ebd-43c1-bdb2-d389c4756467\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-7bddf9d89f-jznzk" Oct 31 13:00:08.652987 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:08.652908 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87125609-1ebd-43c1-bdb2-d389c4756467-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bc9ae8-7bddf9d89f-jznzk\" (UID: \"87125609-1ebd-43c1-bdb2-d389c4756467\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-7bddf9d89f-jznzk" Oct 31 13:00:08.655087 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:08.654839 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87125609-1ebd-43c1-bdb2-d389c4756467-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bc9ae8-7bddf9d89f-jznzk\" (UID: \"87125609-1ebd-43c1-bdb2-d389c4756467\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-7bddf9d89f-jznzk" Oct 31 13:00:08.688672 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:08.676906 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cef1f7a7-aeae-4841-b5f0-8b54c3e9d768-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bc9ae8-cc9d58849-mgh7t\" (UID: \"cef1f7a7-aeae-4841-b5f0-8b54c3e9d768\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-cc9d58849-mgh7t" Oct 31 13:00:08.688672 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:08.676952 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlm29\" (UniqueName: \"kubernetes.io/projected/cef1f7a7-aeae-4841-b5f0-8b54c3e9d768-kube-api-access-dlm29\") pod \"payload-processing-e2e-ipp-a-bc9ae8-cc9d58849-mgh7t\" (UID: \"cef1f7a7-aeae-4841-b5f0-8b54c3e9d768\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-cc9d58849-mgh7t" Oct 31 13:00:08.688672 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:08.676986 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-646wr\" (UniqueName: \"kubernetes.io/projected/87125609-1ebd-43c1-bdb2-d389c4756467-kube-api-access-646wr\") pod \"payload-pre-processing-e2e-ipp-a-bc9ae8-7bddf9d89f-jznzk\" (UID: \"87125609-1ebd-43c1-bdb2-d389c4756467\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-7bddf9d89f-jznzk" Oct 31 13:00:08.688672 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:08.678330 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cef1f7a7-aeae-4841-b5f0-8b54c3e9d768-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bc9ae8-cc9d58849-mgh7t\" (UID: \"cef1f7a7-aeae-4841-b5f0-8b54c3e9d768\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-cc9d58849-mgh7t" Oct 31 13:00:08.709028 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:08.700936 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-646wr\" (UniqueName: \"kubernetes.io/projected/87125609-1ebd-43c1-bdb2-d389c4756467-kube-api-access-646wr\") pod \"payload-pre-processing-e2e-ipp-a-bc9ae8-7bddf9d89f-jznzk\" (UID: \"87125609-1ebd-43c1-bdb2-d389c4756467\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-7bddf9d89f-jznzk" Oct 31 13:00:08.746701 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:08.740423 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlm29\" (UniqueName: \"kubernetes.io/projected/cef1f7a7-aeae-4841-b5f0-8b54c3e9d768-kube-api-access-dlm29\") pod \"payload-processing-e2e-ipp-a-bc9ae8-cc9d58849-mgh7t\" (UID: \"cef1f7a7-aeae-4841-b5f0-8b54c3e9d768\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-cc9d58849-mgh7t" Oct 31 13:00:08.847764 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:08.847729 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-7bddf9d89f-jznzk" Oct 31 13:00:08.869765 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:08.864928 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-cc9d58849-mgh7t" Oct 31 13:00:09.253621 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:09.253592 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-7bddf9d89f-jznzk"] Oct 31 13:00:09.254613 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:00:09.254586 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87125609_1ebd_43c1_bdb2_d389c4756467.slice/crio-76bbbb766d9e673efd511ccc83f040b40e327a2a40009bba5903849173e43032 WatchSource:0}: Error finding container 76bbbb766d9e673efd511ccc83f040b40e327a2a40009bba5903849173e43032: Status 404 returned error can't find the container with id 76bbbb766d9e673efd511ccc83f040b40e327a2a40009bba5903849173e43032 Oct 31 13:00:09.270212 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:09.270184 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-cc9d58849-mgh7t"] Oct 31 13:00:09.280975 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:00:09.280944 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcef1f7a7_aeae_4841_b5f0_8b54c3e9d768.slice/crio-adb0262a4df3643b402780e46984750b2499d4f098e767276f354cc1d8d0b439 WatchSource:0}: Error finding container adb0262a4df3643b402780e46984750b2499d4f098e767276f354cc1d8d0b439: Status 404 returned error can't find the container with id adb0262a4df3643b402780e46984750b2499d4f098e767276f354cc1d8d0b439 Oct 31 13:00:09.436660 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:09.433193 5693 scope.go:122] "RemoveContainer" containerID="a0737bd0f8ef01ce104eca4cf3b729e09b82705df038ef945fb19a30cf2fed96" Oct 31 13:00:09.445675 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:09.444685 5693 scope.go:122] "RemoveContainer" containerID="300c7328755baa543dc5d331aee3c1b56146a15459dc4596cc98c05fda7e11c5" Oct 31 13:00:09.445675 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:00:09.445022 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890860-pvk6z_odh-ai-gateway-infra(5669ae43-7d6d-47df-a7e9-db2a0a000f88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890860-pvk6z" podUID="5669ae43-7d6d-47df-a7e9-db2a0a000f88" Oct 31 13:00:09.445999 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:00:09.445967 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890860-zq9vx_odh-ai-gateway-infra(d4a060ad-6376-4030-9a91-b5ff30e561d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890860-zq9vx" podUID="d4a060ad-6376-4030-9a91-b5ff30e561d8" Oct 31 13:00:09.456796 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:09.456633 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-cc9d58849-mgh7t" event={"ID":"cef1f7a7-aeae-4841-b5f0-8b54c3e9d768","Type":"ContainerStarted","Data":"adb0262a4df3643b402780e46984750b2499d4f098e767276f354cc1d8d0b439"} Oct 31 13:00:09.456796 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:09.456713 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-7bddf9d89f-jznzk" event={"ID":"87125609-1ebd-43c1-bdb2-d389c4756467","Type":"ContainerStarted","Data":"76bbbb766d9e673efd511ccc83f040b40e327a2a40009bba5903849173e43032"} Oct 31 13:00:10.459913 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:10.459879 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-cc9d58849-mgh7t" event={"ID":"cef1f7a7-aeae-4841-b5f0-8b54c3e9d768","Type":"ContainerStarted","Data":"8a85c3a187df437081c168cbd08322a51805aac447ba7b4ac289bb7703458aeb"} Oct 31 13:00:10.460477 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:10.459970 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-cc9d58849-mgh7t" Oct 31 13:00:10.461565 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:10.461541 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-7bddf9d89f-jznzk" event={"ID":"87125609-1ebd-43c1-bdb2-d389c4756467","Type":"ContainerStarted","Data":"2b4d60a0a76668c8d437cdc45cf41975b540349d62c395e12b3e5ebfce5871d6"} Oct 31 13:00:10.461717 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:10.461581 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-7bddf9d89f-jznzk" Oct 31 13:00:10.533668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:10.510485 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-cc9d58849-mgh7t" podStartSLOduration=2.5104629320000003 podStartE2EDuration="2.510462932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:10.484621847 +0000 UTC m=+1661.192208156" watchObservedRunningTime="2026-10-31 13:00:10.510462932 +0000 UTC m=+1661.218049228" Oct 31 13:00:10.547694 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:10.536205 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-7bddf9d89f-jznzk" podStartSLOduration=2.536186957 podStartE2EDuration="2.536186957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:10.509069836 +0000 UTC m=+1661.216656133" watchObservedRunningTime="2026-10-31 13:00:10.536186957 +0000 UTC m=+1661.243773253" Oct 31 13:00:10.967668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:10.962933 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3128e5-59cc88c949-r42bb"] Oct 31 13:00:10.988666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:10.985878 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3128e5-59cc88c949-r42bb"] Oct 31 13:00:10.988666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:10.986005 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3128e5-59cc88c949-r42bb" Oct 31 13:00:11.029864 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:11.029834 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-3128e5-serving-cert\"" Oct 31 13:00:11.114767 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:11.114726 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06baf8a7-b936-4a35-8bf6-a919e152bd62-openshift-service-ca\") pod \"maas-api-e2e-api-a-3128e5-59cc88c949-r42bb\" (UID: \"06baf8a7-b936-4a35-8bf6-a919e152bd62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3128e5-59cc88c949-r42bb" Oct 31 13:00:11.115036 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:11.115014 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06baf8a7-b936-4a35-8bf6-a919e152bd62-maas-api-tls\") pod \"maas-api-e2e-api-a-3128e5-59cc88c949-r42bb\" (UID: \"06baf8a7-b936-4a35-8bf6-a919e152bd62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3128e5-59cc88c949-r42bb" Oct 31 13:00:11.115360 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:11.115339 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkcbm\" (UniqueName: \"kubernetes.io/projected/06baf8a7-b936-4a35-8bf6-a919e152bd62-kube-api-access-rkcbm\") pod \"maas-api-e2e-api-a-3128e5-59cc88c949-r42bb\" (UID: \"06baf8a7-b936-4a35-8bf6-a919e152bd62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3128e5-59cc88c949-r42bb" Oct 31 13:00:11.115592 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:11.115573 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06baf8a7-b936-4a35-8bf6-a919e152bd62-metrics-certs\") pod \"maas-api-e2e-api-a-3128e5-59cc88c949-r42bb\" (UID: \"06baf8a7-b936-4a35-8bf6-a919e152bd62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3128e5-59cc88c949-r42bb" Oct 31 13:00:11.251716 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:11.250365 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkcbm\" (UniqueName: \"kubernetes.io/projected/06baf8a7-b936-4a35-8bf6-a919e152bd62-kube-api-access-rkcbm\") pod \"maas-api-e2e-api-a-3128e5-59cc88c949-r42bb\" (UID: \"06baf8a7-b936-4a35-8bf6-a919e152bd62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3128e5-59cc88c949-r42bb" Oct 31 13:00:11.251716 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:11.250500 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06baf8a7-b936-4a35-8bf6-a919e152bd62-metrics-certs\") pod \"maas-api-e2e-api-a-3128e5-59cc88c949-r42bb\" (UID: \"06baf8a7-b936-4a35-8bf6-a919e152bd62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3128e5-59cc88c949-r42bb" Oct 31 13:00:11.251716 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:11.250589 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06baf8a7-b936-4a35-8bf6-a919e152bd62-openshift-service-ca\") pod \"maas-api-e2e-api-a-3128e5-59cc88c949-r42bb\" (UID: \"06baf8a7-b936-4a35-8bf6-a919e152bd62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3128e5-59cc88c949-r42bb" Oct 31 13:00:11.251716 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:11.250629 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06baf8a7-b936-4a35-8bf6-a919e152bd62-maas-api-tls\") pod \"maas-api-e2e-api-a-3128e5-59cc88c949-r42bb\" (UID: \"06baf8a7-b936-4a35-8bf6-a919e152bd62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3128e5-59cc88c949-r42bb" Oct 31 13:00:11.283024 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:11.274494 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06baf8a7-b936-4a35-8bf6-a919e152bd62-openshift-service-ca\") pod \"maas-api-e2e-api-a-3128e5-59cc88c949-r42bb\" (UID: \"06baf8a7-b936-4a35-8bf6-a919e152bd62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3128e5-59cc88c949-r42bb" Oct 31 13:00:11.283024 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:11.275049 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06baf8a7-b936-4a35-8bf6-a919e152bd62-maas-api-tls\") pod \"maas-api-e2e-api-a-3128e5-59cc88c949-r42bb\" (UID: \"06baf8a7-b936-4a35-8bf6-a919e152bd62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3128e5-59cc88c949-r42bb" Oct 31 13:00:11.309813 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:11.308472 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06baf8a7-b936-4a35-8bf6-a919e152bd62-metrics-certs\") pod \"maas-api-e2e-api-a-3128e5-59cc88c949-r42bb\" (UID: \"06baf8a7-b936-4a35-8bf6-a919e152bd62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3128e5-59cc88c949-r42bb" Oct 31 13:00:11.320069 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:11.319963 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkcbm\" (UniqueName: \"kubernetes.io/projected/06baf8a7-b936-4a35-8bf6-a919e152bd62-kube-api-access-rkcbm\") pod \"maas-api-e2e-api-a-3128e5-59cc88c949-r42bb\" (UID: \"06baf8a7-b936-4a35-8bf6-a919e152bd62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3128e5-59cc88c949-r42bb" Oct 31 13:00:11.419216 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:11.419145 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3128e5-59cc88c949-r42bb" Oct 31 13:00:11.753360 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:00:11.753309 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06baf8a7_b936_4a35_8bf6_a919e152bd62.slice/crio-3e0288356ba76389970c0f9407c285f34638c932bea9fb428d6ce9ed381b7300 WatchSource:0}: Error finding container 3e0288356ba76389970c0f9407c285f34638c932bea9fb428d6ce9ed381b7300: Status 404 returned error can't find the container with id 3e0288356ba76389970c0f9407c285f34638c932bea9fb428d6ce9ed381b7300 Oct 31 13:00:11.753844 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:11.753524 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3128e5-59cc88c949-r42bb"] Oct 31 13:00:12.476514 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:12.476465 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3128e5-59cc88c949-r42bb" event={"ID":"06baf8a7-b936-4a35-8bf6-a919e152bd62","Type":"ContainerStarted","Data":"cb36292757ebd355685ce0ccf2ffd4d62252a5629b949b520172042b58ac0246"} Oct 31 13:00:12.476725 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:12.476541 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3128e5-59cc88c949-r42bb" event={"ID":"06baf8a7-b936-4a35-8bf6-a919e152bd62","Type":"ContainerStarted","Data":"3e0288356ba76389970c0f9407c285f34638c932bea9fb428d6ce9ed381b7300"} Oct 31 13:00:12.476725 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:12.476572 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3128e5-59cc88c949-r42bb" Oct 31 13:00:12.501690 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:12.501526 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3128e5-59cc88c949-r42bb" podStartSLOduration=2.023782761 podStartE2EDuration="2.501507377s" totalImagesPullingTime="477.724616ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:12.501129613 +0000 UTC m=+1663.208715908" watchObservedRunningTime="2026-10-31 13:00:12.501507377 +0000 UTC m=+1663.209093684" Oct 31 13:00:16.333663 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:16.326876 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3128e5-85c845bff5-tw7sl" Oct 31 13:00:18.488541 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:18.488513 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3128e5-59cc88c949-r42bb" Oct 31 13:00:18.694840 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:00:18.694806 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:00:19.405136 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:19.405110 5693 scope.go:122] "RemoveContainer" containerID="300c7328755baa543dc5d331aee3c1b56146a15459dc4596cc98c05fda7e11c5" Oct 31 13:00:20.400825 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:20.400799 5693 scope.go:122] "RemoveContainer" containerID="a0737bd0f8ef01ce104eca4cf3b729e09b82705df038ef945fb19a30cf2fed96" Oct 31 13:00:20.528197 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:20.528161 5693 generic.go:356] "Generic (PLEG): container finished" podID="5669ae43-7d6d-47df-a7e9-db2a0a000f88" containerID="72aa455276aff0984201c503b7039c33bcc03e507a5e751205aabd5ad6b6ed5f" exitCode=60 Oct 31 13:00:20.528395 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:20.528307 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890860-pvk6z" event={"ID":"5669ae43-7d6d-47df-a7e9-db2a0a000f88","Type":"ContainerDied","Data":"72aa455276aff0984201c503b7039c33bcc03e507a5e751205aabd5ad6b6ed5f"} Oct 31 13:00:20.528395 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:20.528364 5693 scope.go:122] "RemoveContainer" containerID="300c7328755baa543dc5d331aee3c1b56146a15459dc4596cc98c05fda7e11c5" Oct 31 13:00:20.528948 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:20.528925 5693 scope.go:122] "RemoveContainer" containerID="72aa455276aff0984201c503b7039c33bcc03e507a5e751205aabd5ad6b6ed5f" Oct 31 13:00:20.529471 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:00:20.529446 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890860-pvk6z_odh-ai-gateway-infra(5669ae43-7d6d-47df-a7e9-db2a0a000f88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890860-pvk6z" podUID="5669ae43-7d6d-47df-a7e9-db2a0a000f88" Oct 31 13:00:21.473095 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:21.473065 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-7bddf9d89f-jznzk" Oct 31 13:00:21.473547 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:21.473345 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-cc9d58849-mgh7t" Oct 31 13:00:21.544431 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:21.544353 5693 generic.go:356] "Generic (PLEG): container finished" podID="d4a060ad-6376-4030-9a91-b5ff30e561d8" containerID="8ec32c94d78954ac640763235c44f4d11ac92c45206055b07b556336166d9300" exitCode=60 Oct 31 13:00:21.544569 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:21.544468 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890860-zq9vx" event={"ID":"d4a060ad-6376-4030-9a91-b5ff30e561d8","Type":"ContainerDied","Data":"8ec32c94d78954ac640763235c44f4d11ac92c45206055b07b556336166d9300"} Oct 31 13:00:21.544614 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:21.544566 5693 scope.go:122] "RemoveContainer" containerID="a0737bd0f8ef01ce104eca4cf3b729e09b82705df038ef945fb19a30cf2fed96" Oct 31 13:00:21.545094 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:21.545077 5693 scope.go:122] "RemoveContainer" containerID="8ec32c94d78954ac640763235c44f4d11ac92c45206055b07b556336166d9300" Oct 31 13:00:21.545418 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:00:21.545400 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890860-zq9vx_odh-ai-gateway-infra(d4a060ad-6376-4030-9a91-b5ff30e561d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890860-zq9vx" podUID="d4a060ad-6376-4030-9a91-b5ff30e561d8" Oct 31 13:00:21.568118 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:21.568095 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890860-pvk6z"] Oct 31 13:00:21.736234 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:21.736206 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890860-pvk6z" Oct 31 13:00:21.912144 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:21.912087 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5669ae43-7d6d-47df-a7e9-db2a0a000f88-kube-api-access-dcvhk\" (UniqueName: \"kubernetes.io/projected/5669ae43-7d6d-47df-a7e9-db2a0a000f88-kube-api-access-dcvhk\") pod \"5669ae43-7d6d-47df-a7e9-db2a0a000f88\" (UID: \"5669ae43-7d6d-47df-a7e9-db2a0a000f88\") " Oct 31 13:00:21.912381 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:21.912231 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5669ae43-7d6d-47df-a7e9-db2a0a000f88-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5669ae43-7d6d-47df-a7e9-db2a0a000f88-maas-api-service-ca\") pod \"5669ae43-7d6d-47df-a7e9-db2a0a000f88\" (UID: \"5669ae43-7d6d-47df-a7e9-db2a0a000f88\") " Oct 31 13:00:21.912877 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:21.912846 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5669ae43-7d6d-47df-a7e9-db2a0a000f88-maas-api-service-ca" pod "5669ae43-7d6d-47df-a7e9-db2a0a000f88" (UID: "5669ae43-7d6d-47df-a7e9-db2a0a000f88"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:00:21.913039 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:21.913001 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5669ae43-7d6d-47df-a7e9-db2a0a000f88-maas-api-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:21.914908 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:21.914874 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5669ae43-7d6d-47df-a7e9-db2a0a000f88-kube-api-access-dcvhk" pod "5669ae43-7d6d-47df-a7e9-db2a0a000f88" (UID: "5669ae43-7d6d-47df-a7e9-db2a0a000f88"). InnerVolumeSpecName "kube-api-access-dcvhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:00:22.014358 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:22.014318 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcvhk\" (UniqueName: \"kubernetes.io/projected/5669ae43-7d6d-47df-a7e9-db2a0a000f88-kube-api-access-dcvhk\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:22.593665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:22.587538 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890860-zq9vx"] Oct 31 13:00:22.615669 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:22.610689 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890860-pvk6z" event={"ID":"5669ae43-7d6d-47df-a7e9-db2a0a000f88","Type":"ContainerDied","Data":"73fd18960bfd5cb97a8e46bc4f346d59839d9d05c15c4ddc1a3ecbad38538554"} Oct 31 13:00:22.615669 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:22.610761 5693 scope.go:122] "RemoveContainer" containerID="72aa455276aff0984201c503b7039c33bcc03e507a5e751205aabd5ad6b6ed5f" Oct 31 13:00:22.615669 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:22.610863 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890860-pvk6z" Oct 31 13:00:22.664156 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:22.663676 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890860-pvk6z"] Oct 31 13:00:22.670446 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:22.669373 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890860-pvk6z"] Oct 31 13:00:22.818423 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:22.818393 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890860-zq9vx" Oct 31 13:00:22.847970 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:22.841891 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4a060ad-6376-4030-9a91-b5ff30e561d8-kube-api-access-l9m78\" (UniqueName: \"kubernetes.io/projected/d4a060ad-6376-4030-9a91-b5ff30e561d8-kube-api-access-l9m78\") pod \"d4a060ad-6376-4030-9a91-b5ff30e561d8\" (UID: \"d4a060ad-6376-4030-9a91-b5ff30e561d8\") " Oct 31 13:00:22.847970 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:22.842046 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4a060ad-6376-4030-9a91-b5ff30e561d8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4a060ad-6376-4030-9a91-b5ff30e561d8-maas-api-service-ca\") pod \"d4a060ad-6376-4030-9a91-b5ff30e561d8\" (UID: \"d4a060ad-6376-4030-9a91-b5ff30e561d8\") " Oct 31 13:00:22.847970 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:22.844436 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4a060ad-6376-4030-9a91-b5ff30e561d8-maas-api-service-ca" pod "d4a060ad-6376-4030-9a91-b5ff30e561d8" (UID: "d4a060ad-6376-4030-9a91-b5ff30e561d8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:00:22.860189 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:22.860156 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4a060ad-6376-4030-9a91-b5ff30e561d8-kube-api-access-l9m78" pod "d4a060ad-6376-4030-9a91-b5ff30e561d8" (UID: "d4a060ad-6376-4030-9a91-b5ff30e561d8"). InnerVolumeSpecName "kube-api-access-l9m78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:00:22.946558 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:22.946519 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4a060ad-6376-4030-9a91-b5ff30e561d8-maas-api-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:22.946558 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:22.946560 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9m78\" (UniqueName: \"kubernetes.io/projected/d4a060ad-6376-4030-9a91-b5ff30e561d8-kube-api-access-l9m78\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:23.424289 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:23.424241 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5669ae43-7d6d-47df-a7e9-db2a0a000f88" path="/var/lib/kubelet/pods/5669ae43-7d6d-47df-a7e9-db2a0a000f88/volumes" Oct 31 13:00:23.617908 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:23.617882 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890860-zq9vx" Oct 31 13:00:23.617908 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:23.617892 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890860-zq9vx" event={"ID":"d4a060ad-6376-4030-9a91-b5ff30e561d8","Type":"ContainerDied","Data":"d260ea9406f669913a30661c376b95a803dc71f7dc54896b489df1f5627858d9"} Oct 31 13:00:23.618407 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:23.617951 5693 scope.go:122] "RemoveContainer" containerID="8ec32c94d78954ac640763235c44f4d11ac92c45206055b07b556336166d9300" Oct 31 13:00:23.635969 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:23.635937 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890860-zq9vx"] Oct 31 13:00:23.639890 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:23.639868 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890860-zq9vx"] Oct 31 13:00:24.085045 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.085012 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s"] Oct 31 13:00:24.086057 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.086032 5693 cpu_manager.go:414] "containerMap: removing container" podUID="5669ae43-7d6d-47df-a7e9-db2a0a000f88" containerName="cleanup" Oct 31 13:00:24.086057 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.086056 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5669ae43-7d6d-47df-a7e9-db2a0a000f88" containerName="cleanup" Oct 31 13:00:24.086275 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.086072 5693 cpu_manager.go:414] "containerMap: removing container" podUID="d4a060ad-6376-4030-9a91-b5ff30e561d8" containerName="cleanup" Oct 31 13:00:24.086275 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.086079 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4a060ad-6376-4030-9a91-b5ff30e561d8" containerName="cleanup" Oct 31 13:00:24.086275 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.086093 5693 cpu_manager.go:414] "containerMap: removing container" podUID="d4a060ad-6376-4030-9a91-b5ff30e561d8" containerName="cleanup" Oct 31 13:00:24.086275 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.086101 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4a060ad-6376-4030-9a91-b5ff30e561d8" containerName="cleanup" Oct 31 13:00:24.086275 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.086112 5693 cpu_manager.go:414] "containerMap: removing container" podUID="d4a060ad-6376-4030-9a91-b5ff30e561d8" containerName="cleanup" Oct 31 13:00:24.086275 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.086119 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4a060ad-6376-4030-9a91-b5ff30e561d8" containerName="cleanup" Oct 31 13:00:24.086275 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.086133 5693 cpu_manager.go:414] "containerMap: removing container" podUID="5669ae43-7d6d-47df-a7e9-db2a0a000f88" containerName="cleanup" Oct 31 13:00:24.086275 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.086141 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5669ae43-7d6d-47df-a7e9-db2a0a000f88" containerName="cleanup" Oct 31 13:00:24.086275 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.086157 5693 cpu_manager.go:414] "containerMap: removing container" podUID="5669ae43-7d6d-47df-a7e9-db2a0a000f88" containerName="cleanup" Oct 31 13:00:24.086275 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.086163 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5669ae43-7d6d-47df-a7e9-db2a0a000f88" containerName="cleanup" Oct 31 13:00:24.086889 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.086361 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="5669ae43-7d6d-47df-a7e9-db2a0a000f88" containerName="cleanup" Oct 31 13:00:24.086889 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.086374 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4a060ad-6376-4030-9a91-b5ff30e561d8" containerName="cleanup" Oct 31 13:00:24.086889 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.086383 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4a060ad-6376-4030-9a91-b5ff30e561d8" containerName="cleanup" Oct 31 13:00:24.086889 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.086393 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="5669ae43-7d6d-47df-a7e9-db2a0a000f88" containerName="cleanup" Oct 31 13:00:24.086889 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.086403 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="5669ae43-7d6d-47df-a7e9-db2a0a000f88" containerName="cleanup" Oct 31 13:00:24.091951 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.091925 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" Oct 31 13:00:24.112125 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.112101 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s"] Oct 31 13:00:24.133456 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.133424 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-56a83a-openshift-default-dockercfg-s7c8k\"" Oct 31 13:00:24.162972 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.162932 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8b99947-a9a4-4b81-83ee-6cafebb48503-istio-data\") pod \"e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s\" (UID: \"b8b99947-a9a4-4b81-83ee-6cafebb48503\") " pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" Oct 31 13:00:24.163104 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.163005 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8b99947-a9a4-4b81-83ee-6cafebb48503-istio-token\") pod \"e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s\" (UID: \"b8b99947-a9a4-4b81-83ee-6cafebb48503\") " pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" Oct 31 13:00:24.163185 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.163154 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8b99947-a9a4-4b81-83ee-6cafebb48503-istiod-ca-cert\") pod \"e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s\" (UID: \"b8b99947-a9a4-4b81-83ee-6cafebb48503\") " pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" Oct 31 13:00:24.163280 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.163261 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8b99947-a9a4-4b81-83ee-6cafebb48503-workload-certs\") pod \"e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s\" (UID: \"b8b99947-a9a4-4b81-83ee-6cafebb48503\") " pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" Oct 31 13:00:24.163403 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.163379 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8b99947-a9a4-4b81-83ee-6cafebb48503-istio-envoy\") pod \"e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s\" (UID: \"b8b99947-a9a4-4b81-83ee-6cafebb48503\") " pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" Oct 31 13:00:24.163467 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.163424 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krppg\" (UniqueName: \"kubernetes.io/projected/b8b99947-a9a4-4b81-83ee-6cafebb48503-kube-api-access-krppg\") pod \"e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s\" (UID: \"b8b99947-a9a4-4b81-83ee-6cafebb48503\") " pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" Oct 31 13:00:24.163573 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.163555 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8b99947-a9a4-4b81-83ee-6cafebb48503-istio-podinfo\") pod \"e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s\" (UID: \"b8b99947-a9a4-4b81-83ee-6cafebb48503\") " pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" Oct 31 13:00:24.163633 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.163601 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b99947-a9a4-4b81-83ee-6cafebb48503-workload-socket\") pod \"e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s\" (UID: \"b8b99947-a9a4-4b81-83ee-6cafebb48503\") " pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" Oct 31 13:00:24.163708 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.163689 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b99947-a9a4-4b81-83ee-6cafebb48503-credential-socket\") pod \"e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s\" (UID: \"b8b99947-a9a4-4b81-83ee-6cafebb48503\") " pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" Oct 31 13:00:24.281669 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.273630 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8b99947-a9a4-4b81-83ee-6cafebb48503-istiod-ca-cert\") pod \"e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s\" (UID: \"b8b99947-a9a4-4b81-83ee-6cafebb48503\") " pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" Oct 31 13:00:24.281669 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.273746 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8b99947-a9a4-4b81-83ee-6cafebb48503-workload-certs\") pod \"e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s\" (UID: \"b8b99947-a9a4-4b81-83ee-6cafebb48503\") " pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" Oct 31 13:00:24.281669 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.273826 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8b99947-a9a4-4b81-83ee-6cafebb48503-istio-envoy\") pod \"e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s\" (UID: \"b8b99947-a9a4-4b81-83ee-6cafebb48503\") " pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" Oct 31 13:00:24.281669 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.273854 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krppg\" (UniqueName: \"kubernetes.io/projected/b8b99947-a9a4-4b81-83ee-6cafebb48503-kube-api-access-krppg\") pod \"e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s\" (UID: \"b8b99947-a9a4-4b81-83ee-6cafebb48503\") " pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" Oct 31 13:00:24.281669 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.273929 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8b99947-a9a4-4b81-83ee-6cafebb48503-istio-podinfo\") pod \"e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s\" (UID: \"b8b99947-a9a4-4b81-83ee-6cafebb48503\") " pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" Oct 31 13:00:24.281669 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.273968 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b99947-a9a4-4b81-83ee-6cafebb48503-workload-socket\") pod \"e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s\" (UID: \"b8b99947-a9a4-4b81-83ee-6cafebb48503\") " pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" Oct 31 13:00:24.281669 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.274003 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b99947-a9a4-4b81-83ee-6cafebb48503-credential-socket\") pod \"e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s\" (UID: \"b8b99947-a9a4-4b81-83ee-6cafebb48503\") " pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" Oct 31 13:00:24.281669 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.274078 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8b99947-a9a4-4b81-83ee-6cafebb48503-istio-data\") pod \"e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s\" (UID: \"b8b99947-a9a4-4b81-83ee-6cafebb48503\") " pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" Oct 31 13:00:24.281669 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.274144 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8b99947-a9a4-4b81-83ee-6cafebb48503-istio-token\") pod \"e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s\" (UID: \"b8b99947-a9a4-4b81-83ee-6cafebb48503\") " pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" Oct 31 13:00:24.285664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.285487 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8b99947-a9a4-4b81-83ee-6cafebb48503-istiod-ca-cert\") pod \"e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s\" (UID: \"b8b99947-a9a4-4b81-83ee-6cafebb48503\") " pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" Oct 31 13:00:24.296662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.285899 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8b99947-a9a4-4b81-83ee-6cafebb48503-workload-certs\") pod \"e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s\" (UID: \"b8b99947-a9a4-4b81-83ee-6cafebb48503\") " pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" Oct 31 13:00:24.296662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.292975 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b99947-a9a4-4b81-83ee-6cafebb48503-workload-socket\") pod \"e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s\" (UID: \"b8b99947-a9a4-4b81-83ee-6cafebb48503\") " pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" Oct 31 13:00:24.296662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.293215 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8b99947-a9a4-4b81-83ee-6cafebb48503-istio-data\") pod \"e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s\" (UID: \"b8b99947-a9a4-4b81-83ee-6cafebb48503\") " pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" Oct 31 13:00:24.296662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.293487 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b99947-a9a4-4b81-83ee-6cafebb48503-credential-socket\") pod \"e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s\" (UID: \"b8b99947-a9a4-4b81-83ee-6cafebb48503\") " pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" Oct 31 13:00:24.343752 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.329411 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krppg\" (UniqueName: \"kubernetes.io/projected/b8b99947-a9a4-4b81-83ee-6cafebb48503-kube-api-access-krppg\") pod \"e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s\" (UID: \"b8b99947-a9a4-4b81-83ee-6cafebb48503\") " pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" Oct 31 13:00:24.343752 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.337458 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8b99947-a9a4-4b81-83ee-6cafebb48503-istio-envoy\") pod \"e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s\" (UID: \"b8b99947-a9a4-4b81-83ee-6cafebb48503\") " pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" Oct 31 13:00:24.352055 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.350852 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8b99947-a9a4-4b81-83ee-6cafebb48503-istio-podinfo\") pod \"e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s\" (UID: \"b8b99947-a9a4-4b81-83ee-6cafebb48503\") " pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" Oct 31 13:00:24.375476 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.375218 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8b99947-a9a4-4b81-83ee-6cafebb48503-istio-token\") pod \"e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s\" (UID: \"b8b99947-a9a4-4b81-83ee-6cafebb48503\") " pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" Oct 31 13:00:24.408149 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.408116 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" Oct 31 13:00:24.707915 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.707888 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s"] Oct 31 13:00:24.709053 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:00:24.709029 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b99947_a9a4_4b81_83ee_6cafebb48503.slice/crio-7df62ab0e032848bf110bfa969fcffd762923d4eb95fc3654345afa92226afd5 WatchSource:0}: Error finding container 7df62ab0e032848bf110bfa969fcffd762923d4eb95fc3654345afa92226afd5: Status 404 returned error can't find the container with id 7df62ab0e032848bf110bfa969fcffd762923d4eb95fc3654345afa92226afd5 Oct 31 13:00:24.711817 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.711784 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:00:24.711893 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.711879 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:00:24.711937 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:24.711926 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:00:25.409341 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:25.409309 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4a060ad-6376-4030-9a91-b5ff30e561d8" path="/var/lib/kubelet/pods/d4a060ad-6376-4030-9a91-b5ff30e561d8/volumes" Oct 31 13:00:25.646665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:25.645740 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" event={"ID":"b8b99947-a9a4-4b81-83ee-6cafebb48503","Type":"ContainerStarted","Data":"206d0f7ab203dda912103ebe173a851f305b79c00aed7959dfa1da8b48f095a0"} Oct 31 13:00:25.646665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:25.645810 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" event={"ID":"b8b99947-a9a4-4b81-83ee-6cafebb48503","Type":"ContainerStarted","Data":"7df62ab0e032848bf110bfa969fcffd762923d4eb95fc3654345afa92226afd5"} Oct 31 13:00:26.409092 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:26.409061 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" Oct 31 13:00:26.414627 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:26.414601 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" Oct 31 13:00:26.445427 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:26.445106 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" podStartSLOduration=2.445093616 podStartE2EDuration="2.445093616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:25.691566652 +0000 UTC m=+1676.399152972" watchObservedRunningTime="2026-10-31 13:00:26.445093616 +0000 UTC m=+1677.152679911" Oct 31 13:00:26.651916 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:26.651882 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" Oct 31 13:00:26.653014 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:26.652995 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" Oct 31 13:00:27.062962 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:27.062932 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-cc9d58849-mgh7t"] Oct 31 13:00:27.078105 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:27.078075 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-7bddf9d89f-jznzk"] Oct 31 13:00:32.063667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:32.063599 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-cc9d58849-mgh7t" podUID="cef1f7a7-aeae-4841-b5f0-8b54c3e9d768" containerName="payload-processing" containerID="cri-o://8a85c3a187df437081c168cbd08322a51805aac447ba7b4ac289bb7703458aeb" gracePeriod=25 Oct 31 13:00:32.079162 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:32.079099 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-7bddf9d89f-jznzk" podUID="87125609-1ebd-43c1-bdb2-d389c4756467" containerName="payload-pre-processing" containerID="cri-o://2b4d60a0a76668c8d437cdc45cf41975b540349d62c395e12b3e5ebfce5871d6" gracePeriod=25 Oct 31 13:00:32.406938 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:32.406908 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-cc9d58849-mgh7t" Oct 31 13:00:32.461231 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:32.461207 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-7bddf9d89f-jznzk" Oct 31 13:00:32.461600 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:32.461582 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cef1f7a7-aeae-4841-b5f0-8b54c3e9d768-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cef1f7a7-aeae-4841-b5f0-8b54c3e9d768-plugins-config-volume\") pod \"cef1f7a7-aeae-4841-b5f0-8b54c3e9d768\" (UID: \"cef1f7a7-aeae-4841-b5f0-8b54c3e9d768\") " Oct 31 13:00:32.461723 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:32.461673 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cef1f7a7-aeae-4841-b5f0-8b54c3e9d768-kube-api-access-dlm29\" (UniqueName: \"kubernetes.io/projected/cef1f7a7-aeae-4841-b5f0-8b54c3e9d768-kube-api-access-dlm29\") pod \"cef1f7a7-aeae-4841-b5f0-8b54c3e9d768\" (UID: \"cef1f7a7-aeae-4841-b5f0-8b54c3e9d768\") " Oct 31 13:00:32.461984 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:32.461949 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cef1f7a7-aeae-4841-b5f0-8b54c3e9d768-plugins-config-volume" pod "cef1f7a7-aeae-4841-b5f0-8b54c3e9d768" (UID: "cef1f7a7-aeae-4841-b5f0-8b54c3e9d768"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:00:32.462333 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:32.462315 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cef1f7a7-aeae-4841-b5f0-8b54c3e9d768-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:32.464205 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:32.464183 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cef1f7a7-aeae-4841-b5f0-8b54c3e9d768-kube-api-access-dlm29" pod "cef1f7a7-aeae-4841-b5f0-8b54c3e9d768" (UID: "cef1f7a7-aeae-4841-b5f0-8b54c3e9d768"). InnerVolumeSpecName "kube-api-access-dlm29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:00:32.563439 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:32.563407 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87125609-1ebd-43c1-bdb2-d389c4756467-kube-api-access-646wr\" (UniqueName: \"kubernetes.io/projected/87125609-1ebd-43c1-bdb2-d389c4756467-kube-api-access-646wr\") pod \"87125609-1ebd-43c1-bdb2-d389c4756467\" (UID: \"87125609-1ebd-43c1-bdb2-d389c4756467\") " Oct 31 13:00:32.563660 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:32.563531 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87125609-1ebd-43c1-bdb2-d389c4756467-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87125609-1ebd-43c1-bdb2-d389c4756467-plugins-config-volume\") pod \"87125609-1ebd-43c1-bdb2-d389c4756467\" (UID: \"87125609-1ebd-43c1-bdb2-d389c4756467\") " Oct 31 13:00:32.563946 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:32.563920 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87125609-1ebd-43c1-bdb2-d389c4756467-plugins-config-volume" pod "87125609-1ebd-43c1-bdb2-d389c4756467" (UID: "87125609-1ebd-43c1-bdb2-d389c4756467"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:00:32.564231 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:32.564211 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlm29\" (UniqueName: \"kubernetes.io/projected/cef1f7a7-aeae-4841-b5f0-8b54c3e9d768-kube-api-access-dlm29\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:32.564322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:32.564233 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87125609-1ebd-43c1-bdb2-d389c4756467-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:32.566217 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:32.566192 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87125609-1ebd-43c1-bdb2-d389c4756467-kube-api-access-646wr" pod "87125609-1ebd-43c1-bdb2-d389c4756467" (UID: "87125609-1ebd-43c1-bdb2-d389c4756467"). InnerVolumeSpecName "kube-api-access-646wr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:00:32.665410 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:32.665381 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-646wr\" (UniqueName: \"kubernetes.io/projected/87125609-1ebd-43c1-bdb2-d389c4756467-kube-api-access-646wr\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:32.707315 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:32.707284 5693 generic.go:356] "Generic (PLEG): container finished" podID="cef1f7a7-aeae-4841-b5f0-8b54c3e9d768" containerID="8a85c3a187df437081c168cbd08322a51805aac447ba7b4ac289bb7703458aeb" exitCode=0 Oct 31 13:00:32.707503 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:32.707331 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-cc9d58849-mgh7t" Oct 31 13:00:32.707503 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:32.707361 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-cc9d58849-mgh7t" event={"ID":"cef1f7a7-aeae-4841-b5f0-8b54c3e9d768","Type":"ContainerDied","Data":"8a85c3a187df437081c168cbd08322a51805aac447ba7b4ac289bb7703458aeb"} Oct 31 13:00:32.707503 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:32.707411 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-cc9d58849-mgh7t" event={"ID":"cef1f7a7-aeae-4841-b5f0-8b54c3e9d768","Type":"ContainerDied","Data":"adb0262a4df3643b402780e46984750b2499d4f098e767276f354cc1d8d0b439"} Oct 31 13:00:32.707503 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:32.707444 5693 scope.go:122] "RemoveContainer" containerID="8a85c3a187df437081c168cbd08322a51805aac447ba7b4ac289bb7703458aeb" Oct 31 13:00:32.709100 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:32.709075 5693 generic.go:356] "Generic (PLEG): container finished" podID="87125609-1ebd-43c1-bdb2-d389c4756467" containerID="2b4d60a0a76668c8d437cdc45cf41975b540349d62c395e12b3e5ebfce5871d6" exitCode=0 Oct 31 13:00:32.709373 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:32.709122 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-7bddf9d89f-jznzk" Oct 31 13:00:32.709373 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:32.709204 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-7bddf9d89f-jznzk" event={"ID":"87125609-1ebd-43c1-bdb2-d389c4756467","Type":"ContainerDied","Data":"2b4d60a0a76668c8d437cdc45cf41975b540349d62c395e12b3e5ebfce5871d6"} Oct 31 13:00:32.709373 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:32.709233 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-7bddf9d89f-jznzk" event={"ID":"87125609-1ebd-43c1-bdb2-d389c4756467","Type":"ContainerDied","Data":"76bbbb766d9e673efd511ccc83f040b40e327a2a40009bba5903849173e43032"} Oct 31 13:00:32.719906 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:32.719889 5693 scope.go:122] "RemoveContainer" containerID="8a85c3a187df437081c168cbd08322a51805aac447ba7b4ac289bb7703458aeb" Oct 31 13:00:32.720225 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:00:32.720194 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a85c3a187df437081c168cbd08322a51805aac447ba7b4ac289bb7703458aeb\": container with ID starting with 8a85c3a187df437081c168cbd08322a51805aac447ba7b4ac289bb7703458aeb not found: ID does not exist" containerID="8a85c3a187df437081c168cbd08322a51805aac447ba7b4ac289bb7703458aeb" Oct 31 13:00:32.720353 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:32.720234 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a85c3a187df437081c168cbd08322a51805aac447ba7b4ac289bb7703458aeb"} err="failed to get container status \"8a85c3a187df437081c168cbd08322a51805aac447ba7b4ac289bb7703458aeb\": rpc error: code = NotFound desc = could not find container \"8a85c3a187df437081c168cbd08322a51805aac447ba7b4ac289bb7703458aeb\": container with ID starting with 8a85c3a187df437081c168cbd08322a51805aac447ba7b4ac289bb7703458aeb not found: ID does not exist" Oct 31 13:00:32.720353 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:32.720257 5693 scope.go:122] "RemoveContainer" containerID="2b4d60a0a76668c8d437cdc45cf41975b540349d62c395e12b3e5ebfce5871d6" Oct 31 13:00:32.732051 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:32.732032 5693 scope.go:122] "RemoveContainer" containerID="2b4d60a0a76668c8d437cdc45cf41975b540349d62c395e12b3e5ebfce5871d6" Oct 31 13:00:32.732349 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:00:32.732332 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b4d60a0a76668c8d437cdc45cf41975b540349d62c395e12b3e5ebfce5871d6\": container with ID starting with 2b4d60a0a76668c8d437cdc45cf41975b540349d62c395e12b3e5ebfce5871d6 not found: ID does not exist" containerID="2b4d60a0a76668c8d437cdc45cf41975b540349d62c395e12b3e5ebfce5871d6" Oct 31 13:00:32.732392 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:32.732357 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b4d60a0a76668c8d437cdc45cf41975b540349d62c395e12b3e5ebfce5871d6"} err="failed to get container status \"2b4d60a0a76668c8d437cdc45cf41975b540349d62c395e12b3e5ebfce5871d6\": rpc error: code = NotFound desc = could not find container \"2b4d60a0a76668c8d437cdc45cf41975b540349d62c395e12b3e5ebfce5871d6\": container with ID starting with 2b4d60a0a76668c8d437cdc45cf41975b540349d62c395e12b3e5ebfce5871d6 not found: ID does not exist" Oct 31 13:00:32.741147 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:32.741120 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-cc9d58849-mgh7t"] Oct 31 13:00:32.745508 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:32.745489 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bc9ae8-cc9d58849-mgh7t"] Oct 31 13:00:32.756141 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:32.756116 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-7bddf9d89f-jznzk"] Oct 31 13:00:32.760037 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:32.760005 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-7bddf9d89f-jznzk"] Oct 31 13:00:33.408997 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:33.408963 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87125609-1ebd-43c1-bdb2-d389c4756467" path="/var/lib/kubelet/pods/87125609-1ebd-43c1-bdb2-d389c4756467/volumes" Oct 31 13:00:33.409396 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:33.409379 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cef1f7a7-aeae-4841-b5f0-8b54c3e9d768" path="/var/lib/kubelet/pods/cef1f7a7-aeae-4841-b5f0-8b54c3e9d768/volumes" Oct 31 13:00:34.624161 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:34.624126 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-64f86ffbff-5fcnb"] Oct 31 13:00:34.624775 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:34.624759 5693 cpu_manager.go:414] "containerMap: removing container" podUID="87125609-1ebd-43c1-bdb2-d389c4756467" containerName="payload-pre-processing" Oct 31 13:00:34.624851 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:34.624779 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87125609-1ebd-43c1-bdb2-d389c4756467" containerName="payload-pre-processing" Oct 31 13:00:34.624851 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:34.624811 5693 cpu_manager.go:414] "containerMap: removing container" podUID="cef1f7a7-aeae-4841-b5f0-8b54c3e9d768" containerName="payload-processing" Oct 31 13:00:34.624851 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:34.624816 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cef1f7a7-aeae-4841-b5f0-8b54c3e9d768" containerName="payload-processing" Oct 31 13:00:34.624949 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:34.624916 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="87125609-1ebd-43c1-bdb2-d389c4756467" containerName="payload-pre-processing" Oct 31 13:00:34.624949 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:34.624924 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4a060ad-6376-4030-9a91-b5ff30e561d8" containerName="cleanup" Oct 31 13:00:34.624949 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:34.624939 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="cef1f7a7-aeae-4841-b5f0-8b54c3e9d768" containerName="payload-processing" Oct 31 13:00:34.628189 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:34.628170 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-64f86ffbff-5fcnb" Oct 31 13:00:34.635476 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:34.635454 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-bc9ae8\"" Oct 31 13:00:34.635991 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:34.635967 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-bc9ae8-dockercfg-gw9sc\"" Oct 31 13:00:34.641013 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:34.640985 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-64f86ffbff-5fcnb"] Oct 31 13:00:34.710760 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:34.710713 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/820722a4-2f7a-45b0-911a-fc744c660284-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bc9ae8-64f86ffbff-5fcnb\" (UID: \"820722a4-2f7a-45b0-911a-fc744c660284\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-64f86ffbff-5fcnb" Oct 31 13:00:34.711000 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:34.710974 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6z9p\" (UniqueName: \"kubernetes.io/projected/820722a4-2f7a-45b0-911a-fc744c660284-kube-api-access-f6z9p\") pod \"payload-pre-processing-e2e-ipp-a-bc9ae8-64f86ffbff-5fcnb\" (UID: \"820722a4-2f7a-45b0-911a-fc744c660284\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-64f86ffbff-5fcnb" Oct 31 13:00:34.812354 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:34.812312 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/820722a4-2f7a-45b0-911a-fc744c660284-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bc9ae8-64f86ffbff-5fcnb\" (UID: \"820722a4-2f7a-45b0-911a-fc744c660284\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-64f86ffbff-5fcnb" Oct 31 13:00:34.821682 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:34.816980 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6z9p\" (UniqueName: \"kubernetes.io/projected/820722a4-2f7a-45b0-911a-fc744c660284-kube-api-access-f6z9p\") pod \"payload-pre-processing-e2e-ipp-a-bc9ae8-64f86ffbff-5fcnb\" (UID: \"820722a4-2f7a-45b0-911a-fc744c660284\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-64f86ffbff-5fcnb" Oct 31 13:00:34.821682 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:34.819827 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/820722a4-2f7a-45b0-911a-fc744c660284-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bc9ae8-64f86ffbff-5fcnb\" (UID: \"820722a4-2f7a-45b0-911a-fc744c660284\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-64f86ffbff-5fcnb" Oct 31 13:00:34.847664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:34.836685 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6z9p\" (UniqueName: \"kubernetes.io/projected/820722a4-2f7a-45b0-911a-fc744c660284-kube-api-access-f6z9p\") pod \"payload-pre-processing-e2e-ipp-a-bc9ae8-64f86ffbff-5fcnb\" (UID: \"820722a4-2f7a-45b0-911a-fc744c660284\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-64f86ffbff-5fcnb" Oct 31 13:00:34.943452 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:34.943420 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-64f86ffbff-5fcnb" Oct 31 13:00:35.232404 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:35.232380 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-64f86ffbff-5fcnb"] Oct 31 13:00:35.234093 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:00:35.234063 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod820722a4_2f7a_45b0_911a_fc744c660284.slice/crio-f75de74786817626a858a0f65ee7e9fba114ab551be41c5114e11c664087e16a WatchSource:0}: Error finding container f75de74786817626a858a0f65ee7e9fba114ab551be41c5114e11c664087e16a: Status 404 returned error can't find the container with id f75de74786817626a858a0f65ee7e9fba114ab551be41c5114e11c664087e16a Oct 31 13:00:35.531448 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:35.531359 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg"] Oct 31 13:00:35.537592 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:35.537569 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" Oct 31 13:00:35.560261 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:35.560240 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-adfd11-openshift-default-dockercfg-9j7dr\"" Oct 31 13:00:35.571255 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:35.571224 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg"] Oct 31 13:00:35.631549 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:35.631494 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-istio-token\") pod \"e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg\" (UID: \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\") " pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" Oct 31 13:00:35.632253 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:35.632223 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-istio-envoy\") pod \"e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg\" (UID: \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\") " pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" Oct 31 13:00:35.632401 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:35.632383 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmv5d\" (UniqueName: \"kubernetes.io/projected/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-kube-api-access-hmv5d\") pod \"e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg\" (UID: \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\") " pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" Oct 31 13:00:35.632515 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:35.632501 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-credential-socket\") pod \"e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg\" (UID: \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\") " pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" Oct 31 13:00:35.632708 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:35.632684 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-istiod-ca-cert\") pod \"e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg\" (UID: \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\") " pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" Oct 31 13:00:35.632961 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:35.632938 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-istio-podinfo\") pod \"e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg\" (UID: \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\") " pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" Oct 31 13:00:35.633158 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:35.633140 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-workload-socket\") pod \"e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg\" (UID: \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\") " pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" Oct 31 13:00:35.633271 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:35.633257 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-istio-data\") pod \"e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg\" (UID: \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\") " pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" Oct 31 13:00:35.633393 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:35.633378 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-workload-certs\") pod \"e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg\" (UID: \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\") " pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" Oct 31 13:00:35.743113 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:35.743079 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmv5d\" (UniqueName: \"kubernetes.io/projected/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-kube-api-access-hmv5d\") pod \"e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg\" (UID: \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\") " pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" Oct 31 13:00:35.743717 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:35.743682 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-credential-socket\") pod \"e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg\" (UID: \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\") " pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" Oct 31 13:00:35.744341 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:35.744324 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-istiod-ca-cert\") pod \"e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg\" (UID: \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\") " pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" Oct 31 13:00:35.758938 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:35.746836 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-istio-podinfo\") pod \"e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg\" (UID: \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\") " pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" Oct 31 13:00:35.758938 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:35.746945 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-workload-socket\") pod \"e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg\" (UID: \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\") " pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" Oct 31 13:00:35.758938 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:35.746969 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-istio-data\") pod \"e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg\" (UID: \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\") " pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" Oct 31 13:00:35.758938 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:35.747009 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-workload-certs\") pod \"e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg\" (UID: \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\") " pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" Oct 31 13:00:35.758938 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:35.747074 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-istio-token\") pod \"e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg\" (UID: \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\") " pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" Oct 31 13:00:35.758938 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:35.747176 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-istio-envoy\") pod \"e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg\" (UID: \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\") " pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" Oct 31 13:00:35.758938 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:35.747815 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-workload-socket\") pod \"e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg\" (UID: \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\") " pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" Oct 31 13:00:35.758938 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:35.744223 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-credential-socket\") pod \"e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg\" (UID: \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\") " pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" Oct 31 13:00:35.758938 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:35.750361 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-istio-envoy\") pod \"e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg\" (UID: \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\") " pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" Oct 31 13:00:35.758938 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:35.750763 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-workload-certs\") pod \"e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg\" (UID: \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\") " pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" Oct 31 13:00:35.758938 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:35.745102 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-istiod-ca-cert\") pod \"e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg\" (UID: \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\") " pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" Oct 31 13:00:35.758938 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:35.750882 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-istio-data\") pod \"e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg\" (UID: \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\") " pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" Oct 31 13:00:35.771898 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:35.771166 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-istio-token\") pod \"e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg\" (UID: \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\") " pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" Oct 31 13:00:35.775330 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:35.774083 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-istio-podinfo\") pod \"e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg\" (UID: \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\") " pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" Oct 31 13:00:35.793162 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:35.787175 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmv5d\" (UniqueName: \"kubernetes.io/projected/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-kube-api-access-hmv5d\") pod \"e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg\" (UID: \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\") " pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" Oct 31 13:00:35.801394 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:35.801322 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-64f86ffbff-5fcnb" event={"ID":"820722a4-2f7a-45b0-911a-fc744c660284","Type":"ContainerStarted","Data":"f75de74786817626a858a0f65ee7e9fba114ab551be41c5114e11c664087e16a"} Oct 31 13:00:35.854030 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:35.853991 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" Oct 31 13:00:36.156767 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:36.156691 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg"] Oct 31 13:00:36.158830 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:00:36.158800 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c8d2a80_a147_44f0_b49b_c3bd5b29fef0.slice/crio-8e4dc2bc5f22c2ded4d803bcab6f8d36fd5e75950264574d325e1ecac23acdd5 WatchSource:0}: Error finding container 8e4dc2bc5f22c2ded4d803bcab6f8d36fd5e75950264574d325e1ecac23acdd5: Status 404 returned error can't find the container with id 8e4dc2bc5f22c2ded4d803bcab6f8d36fd5e75950264574d325e1ecac23acdd5 Oct 31 13:00:36.162279 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:36.162236 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:00:36.162387 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:36.162329 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:00:36.162387 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:36.162374 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:00:36.811881 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:36.811847 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" event={"ID":"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0","Type":"ContainerStarted","Data":"04d076bc5e9ccd0ffcf480ba0b8e33c104bfc85033cf970007f7c761f5205dfb"} Oct 31 13:00:36.812289 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:36.811935 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" event={"ID":"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0","Type":"ContainerStarted","Data":"8e4dc2bc5f22c2ded4d803bcab6f8d36fd5e75950264574d325e1ecac23acdd5"} Oct 31 13:00:36.855143 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:36.855114 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" Oct 31 13:00:36.915784 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:36.915720 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" podStartSLOduration=1.915701957 podStartE2EDuration="1.915701957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:36.912071334 +0000 UTC m=+1687.619657632" watchObservedRunningTime="2026-10-31 13:00:36.915701957 +0000 UTC m=+1687.623288277" Oct 31 13:00:37.693478 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:37.693450 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j"] Oct 31 13:00:37.700281 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:37.700260 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j" Oct 31 13:00:37.713428 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:37.713406 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-56a83a-serving-cert\"" Oct 31 13:00:37.714633 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:37.714608 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j"] Oct 31 13:00:37.794836 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:37.794794 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddfdedb8-f94c-4b8b-a717-42b9b8b3642c-metrics-certs\") pod \"maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j\" (UID: \"ddfdedb8-f94c-4b8b-a717-42b9b8b3642c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j" Oct 31 13:00:37.794957 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:37.794910 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddfdedb8-f94c-4b8b-a717-42b9b8b3642c-openshift-service-ca\") pod \"maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j\" (UID: \"ddfdedb8-f94c-4b8b-a717-42b9b8b3642c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j" Oct 31 13:00:37.795109 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:37.795097 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhs7j\" (UniqueName: \"kubernetes.io/projected/ddfdedb8-f94c-4b8b-a717-42b9b8b3642c-kube-api-access-mhs7j\") pod \"maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j\" (UID: \"ddfdedb8-f94c-4b8b-a717-42b9b8b3642c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j" Oct 31 13:00:37.795147 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:37.795135 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddfdedb8-f94c-4b8b-a717-42b9b8b3642c-maas-api-tls\") pod \"maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j\" (UID: \"ddfdedb8-f94c-4b8b-a717-42b9b8b3642c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j" Oct 31 13:00:37.822715 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:37.822682 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-64f86ffbff-5fcnb" event={"ID":"820722a4-2f7a-45b0-911a-fc744c660284","Type":"ContainerStarted","Data":"acdfe5595168d409baca1b8e25529285338071072f041698521c460c928e8f01"} Oct 31 13:00:37.823069 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:37.822838 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-64f86ffbff-5fcnb" Oct 31 13:00:37.850944 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:37.850862 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-64f86ffbff-5fcnb" podStartSLOduration=1.402720388 podStartE2EDuration="3.850838372s" totalImagesPullingTime="2.448117984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:37.842333239 +0000 UTC m=+1688.549919560" watchObservedRunningTime="2026-10-31 13:00:37.850838372 +0000 UTC m=+1688.558424669" Oct 31 13:00:37.876748 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:37.876718 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" Oct 31 13:00:37.912074 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:37.912025 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhs7j\" (UniqueName: \"kubernetes.io/projected/ddfdedb8-f94c-4b8b-a717-42b9b8b3642c-kube-api-access-mhs7j\") pod \"maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j\" (UID: \"ddfdedb8-f94c-4b8b-a717-42b9b8b3642c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j" Oct 31 13:00:37.912368 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:37.912347 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddfdedb8-f94c-4b8b-a717-42b9b8b3642c-maas-api-tls\") pod \"maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j\" (UID: \"ddfdedb8-f94c-4b8b-a717-42b9b8b3642c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j" Oct 31 13:00:37.912555 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:37.912537 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddfdedb8-f94c-4b8b-a717-42b9b8b3642c-metrics-certs\") pod \"maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j\" (UID: \"ddfdedb8-f94c-4b8b-a717-42b9b8b3642c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j" Oct 31 13:00:37.913468 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:37.912811 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddfdedb8-f94c-4b8b-a717-42b9b8b3642c-openshift-service-ca\") pod \"maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j\" (UID: \"ddfdedb8-f94c-4b8b-a717-42b9b8b3642c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j" Oct 31 13:00:37.915260 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:37.915207 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddfdedb8-f94c-4b8b-a717-42b9b8b3642c-openshift-service-ca\") pod \"maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j\" (UID: \"ddfdedb8-f94c-4b8b-a717-42b9b8b3642c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j" Oct 31 13:00:37.918562 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:37.918508 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddfdedb8-f94c-4b8b-a717-42b9b8b3642c-maas-api-tls\") pod \"maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j\" (UID: \"ddfdedb8-f94c-4b8b-a717-42b9b8b3642c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j" Oct 31 13:00:37.921202 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:37.921169 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddfdedb8-f94c-4b8b-a717-42b9b8b3642c-metrics-certs\") pod \"maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j\" (UID: \"ddfdedb8-f94c-4b8b-a717-42b9b8b3642c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j" Oct 31 13:00:37.928710 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:37.928262 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhs7j\" (UniqueName: \"kubernetes.io/projected/ddfdedb8-f94c-4b8b-a717-42b9b8b3642c-kube-api-access-mhs7j\") pod \"maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j\" (UID: \"ddfdedb8-f94c-4b8b-a717-42b9b8b3642c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j" Oct 31 13:00:38.054250 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:38.053891 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j" Oct 31 13:00:38.409253 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:38.409079 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j"] Oct 31 13:00:38.413067 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:00:38.413027 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddfdedb8_f94c_4b8b_a717_42b9b8b3642c.slice/crio-fa113e41f73eb249ea981e79540da2153193290174678cb4c3270a76a6e056c5 WatchSource:0}: Error finding container fa113e41f73eb249ea981e79540da2153193290174678cb4c3270a76a6e056c5: Status 404 returned error can't find the container with id fa113e41f73eb249ea981e79540da2153193290174678cb4c3270a76a6e056c5 Oct 31 13:00:38.831571 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:38.831488 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j" event={"ID":"ddfdedb8-f94c-4b8b-a717-42b9b8b3642c","Type":"ContainerStarted","Data":"fa113e41f73eb249ea981e79540da2153193290174678cb4c3270a76a6e056c5"} Oct 31 13:00:38.832052 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:38.831761 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" Oct 31 13:00:38.833072 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:38.833050 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" Oct 31 13:00:39.842707 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:39.842664 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j" event={"ID":"ddfdedb8-f94c-4b8b-a717-42b9b8b3642c","Type":"ContainerStarted","Data":"f240f8d2ac8eb471a2064dc890a34460cecd5a15bda5f437e23b3fa139eba818"} Oct 31 13:00:39.843223 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:39.843047 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j" Oct 31 13:00:39.879220 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:39.879165 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j" podStartSLOduration=2.439583875 podStartE2EDuration="2.879151837s" totalImagesPullingTime="439.567962ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:39.878956169 +0000 UTC m=+1690.586542460" watchObservedRunningTime="2026-10-31 13:00:39.879151837 +0000 UTC m=+1690.586738132" Oct 31 13:00:44.348291 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:44.348261 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42"] Oct 31 13:00:44.353857 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:44.353833 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" Oct 31 13:00:44.363282 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:44.363116 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-3128e5-openshift-default-dockercfg-chgx7\"" Oct 31 13:00:44.390813 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:44.390773 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42"] Oct 31 13:00:44.495986 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:44.481136 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e427b40-6255-497b-a5a3-7aa06c339de9-istio-podinfo\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42\" (UID: \"9e427b40-6255-497b-a5a3-7aa06c339de9\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" Oct 31 13:00:44.495986 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:44.481196 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e427b40-6255-497b-a5a3-7aa06c339de9-workload-socket\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42\" (UID: \"9e427b40-6255-497b-a5a3-7aa06c339de9\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" Oct 31 13:00:44.495986 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:44.481226 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e427b40-6255-497b-a5a3-7aa06c339de9-istio-envoy\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42\" (UID: \"9e427b40-6255-497b-a5a3-7aa06c339de9\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" Oct 31 13:00:44.495986 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:44.481275 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e427b40-6255-497b-a5a3-7aa06c339de9-credential-socket\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42\" (UID: \"9e427b40-6255-497b-a5a3-7aa06c339de9\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" Oct 31 13:00:44.495986 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:44.481300 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e427b40-6255-497b-a5a3-7aa06c339de9-istio-data\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42\" (UID: \"9e427b40-6255-497b-a5a3-7aa06c339de9\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" Oct 31 13:00:44.495986 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:44.481329 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e427b40-6255-497b-a5a3-7aa06c339de9-istio-token\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42\" (UID: \"9e427b40-6255-497b-a5a3-7aa06c339de9\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" Oct 31 13:00:44.495986 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:44.481388 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e427b40-6255-497b-a5a3-7aa06c339de9-workload-certs\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42\" (UID: \"9e427b40-6255-497b-a5a3-7aa06c339de9\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" Oct 31 13:00:44.495986 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:44.481459 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e427b40-6255-497b-a5a3-7aa06c339de9-istiod-ca-cert\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42\" (UID: \"9e427b40-6255-497b-a5a3-7aa06c339de9\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" Oct 31 13:00:44.495986 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:44.481512 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6bkq\" (UniqueName: \"kubernetes.io/projected/9e427b40-6255-497b-a5a3-7aa06c339de9-kube-api-access-q6bkq\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42\" (UID: \"9e427b40-6255-497b-a5a3-7aa06c339de9\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" Oct 31 13:00:44.583543 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:44.583506 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6bkq\" (UniqueName: \"kubernetes.io/projected/9e427b40-6255-497b-a5a3-7aa06c339de9-kube-api-access-q6bkq\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42\" (UID: \"9e427b40-6255-497b-a5a3-7aa06c339de9\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" Oct 31 13:00:44.583790 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:44.583724 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e427b40-6255-497b-a5a3-7aa06c339de9-istio-podinfo\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42\" (UID: \"9e427b40-6255-497b-a5a3-7aa06c339de9\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" Oct 31 13:00:44.583790 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:44.583765 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e427b40-6255-497b-a5a3-7aa06c339de9-workload-socket\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42\" (UID: \"9e427b40-6255-497b-a5a3-7aa06c339de9\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" Oct 31 13:00:44.583933 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:44.583803 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e427b40-6255-497b-a5a3-7aa06c339de9-istio-envoy\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42\" (UID: \"9e427b40-6255-497b-a5a3-7aa06c339de9\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" Oct 31 13:00:44.583933 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:44.583848 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e427b40-6255-497b-a5a3-7aa06c339de9-credential-socket\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42\" (UID: \"9e427b40-6255-497b-a5a3-7aa06c339de9\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" Oct 31 13:00:44.584045 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:44.583958 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e427b40-6255-497b-a5a3-7aa06c339de9-istio-data\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42\" (UID: \"9e427b40-6255-497b-a5a3-7aa06c339de9\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" Oct 31 13:00:44.584045 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:44.584012 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e427b40-6255-497b-a5a3-7aa06c339de9-istio-token\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42\" (UID: \"9e427b40-6255-497b-a5a3-7aa06c339de9\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" Oct 31 13:00:44.584145 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:44.584075 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e427b40-6255-497b-a5a3-7aa06c339de9-workload-certs\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42\" (UID: \"9e427b40-6255-497b-a5a3-7aa06c339de9\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" Oct 31 13:00:44.584201 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:44.584149 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e427b40-6255-497b-a5a3-7aa06c339de9-istiod-ca-cert\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42\" (UID: \"9e427b40-6255-497b-a5a3-7aa06c339de9\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" Oct 31 13:00:44.584249 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:44.584212 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e427b40-6255-497b-a5a3-7aa06c339de9-workload-socket\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42\" (UID: \"9e427b40-6255-497b-a5a3-7aa06c339de9\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" Oct 31 13:00:44.584406 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:44.584386 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e427b40-6255-497b-a5a3-7aa06c339de9-istio-data\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42\" (UID: \"9e427b40-6255-497b-a5a3-7aa06c339de9\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" Oct 31 13:00:44.584515 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:44.584491 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e427b40-6255-497b-a5a3-7aa06c339de9-credential-socket\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42\" (UID: \"9e427b40-6255-497b-a5a3-7aa06c339de9\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" Oct 31 13:00:44.584660 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:44.584618 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e427b40-6255-497b-a5a3-7aa06c339de9-workload-certs\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42\" (UID: \"9e427b40-6255-497b-a5a3-7aa06c339de9\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" Oct 31 13:00:44.585231 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:44.585202 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e427b40-6255-497b-a5a3-7aa06c339de9-istiod-ca-cert\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42\" (UID: \"9e427b40-6255-497b-a5a3-7aa06c339de9\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" Oct 31 13:00:44.586450 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:44.586433 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e427b40-6255-497b-a5a3-7aa06c339de9-istio-envoy\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42\" (UID: \"9e427b40-6255-497b-a5a3-7aa06c339de9\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" Oct 31 13:00:44.586727 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:44.586697 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e427b40-6255-497b-a5a3-7aa06c339de9-istio-podinfo\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42\" (UID: \"9e427b40-6255-497b-a5a3-7aa06c339de9\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" Oct 31 13:00:44.600226 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:44.600155 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6bkq\" (UniqueName: \"kubernetes.io/projected/9e427b40-6255-497b-a5a3-7aa06c339de9-kube-api-access-q6bkq\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42\" (UID: \"9e427b40-6255-497b-a5a3-7aa06c339de9\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" Oct 31 13:00:44.600950 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:44.600923 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e427b40-6255-497b-a5a3-7aa06c339de9-istio-token\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42\" (UID: \"9e427b40-6255-497b-a5a3-7aa06c339de9\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" Oct 31 13:00:44.669073 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:44.669044 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" Oct 31 13:00:44.976880 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:44.976851 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42"] Oct 31 13:00:44.977106 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:00:44.977082 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e427b40_6255_497b_a5a3_7aa06c339de9.slice/crio-116f6c7fa019c0b40c7befa2b84fb529eff6f716b19ec663bdecc57c47825d28 WatchSource:0}: Error finding container 116f6c7fa019c0b40c7befa2b84fb529eff6f716b19ec663bdecc57c47825d28: Status 404 returned error can't find the container with id 116f6c7fa019c0b40c7befa2b84fb529eff6f716b19ec663bdecc57c47825d28 Oct 31 13:00:45.006666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:44.981320 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:00:45.006666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:44.981411 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:00:45.006666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:44.981459 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:00:45.855174 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:45.855145 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j" Oct 31 13:00:45.917159 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:45.917107 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" event={"ID":"9e427b40-6255-497b-a5a3-7aa06c339de9","Type":"ContainerStarted","Data":"be5ae300e5daa207779b536bf6a633c60764381f859208b2e40944e5e0bbf6ac"} Oct 31 13:00:45.917332 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:45.917176 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" event={"ID":"9e427b40-6255-497b-a5a3-7aa06c339de9","Type":"ContainerStarted","Data":"116f6c7fa019c0b40c7befa2b84fb529eff6f716b19ec663bdecc57c47825d28"} Oct 31 13:00:45.944333 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:45.944271 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" podStartSLOduration=1.944252787 podStartE2EDuration="1.944252787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:45.942564553 +0000 UTC m=+1696.650150860" watchObservedRunningTime="2026-10-31 13:00:45.944252787 +0000 UTC m=+1696.651839084" Oct 31 13:00:46.669869 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:46.669831 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" Oct 31 13:00:46.671606 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:46.671576 5693 patch_prober.go:29] interesting pod/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 31 13:00:46.671746 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:46.671665 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" podUID="9e427b40-6255-497b-a5a3-7aa06c339de9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 31 13:00:47.670259 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:47.670219 5693 patch_prober.go:29] interesting pod/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 31 13:00:47.670680 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:47.670289 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" podUID="9e427b40-6255-497b-a5a3-7aa06c339de9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 31 13:00:48.671564 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:48.670305 5693 patch_prober.go:29] interesting pod/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 31 13:00:48.671564 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:48.670396 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" podUID="9e427b40-6255-497b-a5a3-7aa06c339de9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 31 13:00:48.771465 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:00:48.771425 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:00:48.888665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:48.869062 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-64f86ffbff-5fcnb" Oct 31 13:00:49.670650 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:49.670600 5693 patch_prober.go:29] interesting pod/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 31 13:00:49.670830 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:49.670678 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" podUID="9e427b40-6255-497b-a5a3-7aa06c339de9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 31 13:00:50.680667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:50.672851 5693 patch_prober.go:29] interesting pod/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 31 13:00:50.680667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:50.672914 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" podUID="9e427b40-6255-497b-a5a3-7aa06c339de9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 31 13:00:50.947899 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:50.944765 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m"] Oct 31 13:00:50.956612 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:50.956359 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m"] Oct 31 13:00:50.956612 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:50.956478 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m" Oct 31 13:00:50.970301 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:50.970273 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-adfd11-serving-cert\"" Oct 31 13:00:51.001670 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:51.001604 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74g45\" (UniqueName: \"kubernetes.io/projected/6a1ee165-b894-429a-b703-d99a2839af4f-kube-api-access-74g45\") pod \"maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m\" (UID: \"6a1ee165-b894-429a-b703-d99a2839af4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m" Oct 31 13:00:51.002080 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:51.002032 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a1ee165-b894-429a-b703-d99a2839af4f-metrics-certs\") pod \"maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m\" (UID: \"6a1ee165-b894-429a-b703-d99a2839af4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m" Oct 31 13:00:51.002450 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:51.002423 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a1ee165-b894-429a-b703-d99a2839af4f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m\" (UID: \"6a1ee165-b894-429a-b703-d99a2839af4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m" Oct 31 13:00:51.002744 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:51.002725 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a1ee165-b894-429a-b703-d99a2839af4f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m\" (UID: \"6a1ee165-b894-429a-b703-d99a2839af4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m" Oct 31 13:00:51.104711 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:51.104233 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a1ee165-b894-429a-b703-d99a2839af4f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m\" (UID: \"6a1ee165-b894-429a-b703-d99a2839af4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m" Oct 31 13:00:51.104711 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:51.104420 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74g45\" (UniqueName: \"kubernetes.io/projected/6a1ee165-b894-429a-b703-d99a2839af4f-kube-api-access-74g45\") pod \"maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m\" (UID: \"6a1ee165-b894-429a-b703-d99a2839af4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m" Oct 31 13:00:51.104711 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:51.104453 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a1ee165-b894-429a-b703-d99a2839af4f-metrics-certs\") pod \"maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m\" (UID: \"6a1ee165-b894-429a-b703-d99a2839af4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m" Oct 31 13:00:51.104711 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:51.104680 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a1ee165-b894-429a-b703-d99a2839af4f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m\" (UID: \"6a1ee165-b894-429a-b703-d99a2839af4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m" Oct 31 13:00:51.119665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:51.115706 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a1ee165-b894-429a-b703-d99a2839af4f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m\" (UID: \"6a1ee165-b894-429a-b703-d99a2839af4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m" Oct 31 13:00:51.160668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:51.137959 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a1ee165-b894-429a-b703-d99a2839af4f-metrics-certs\") pod \"maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m\" (UID: \"6a1ee165-b894-429a-b703-d99a2839af4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m" Oct 31 13:00:51.160668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:51.156612 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a1ee165-b894-429a-b703-d99a2839af4f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m\" (UID: \"6a1ee165-b894-429a-b703-d99a2839af4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m" Oct 31 13:00:51.160668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:51.157393 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74g45\" (UniqueName: \"kubernetes.io/projected/6a1ee165-b894-429a-b703-d99a2839af4f-kube-api-access-74g45\") pod \"maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m\" (UID: \"6a1ee165-b894-429a-b703-d99a2839af4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m" Oct 31 13:00:51.282714 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:51.282296 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m" Oct 31 13:00:51.675397 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:00:51.675358 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a1ee165_b894_429a_b703_d99a2839af4f.slice/crio-2159e177bef853bc6f8161d4297d8c93d47d3c6d6969535fd040d017fd16d46c WatchSource:0}: Error finding container 2159e177bef853bc6f8161d4297d8c93d47d3c6d6969535fd040d017fd16d46c: Status 404 returned error can't find the container with id 2159e177bef853bc6f8161d4297d8c93d47d3c6d6969535fd040d017fd16d46c Oct 31 13:00:51.675610 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:51.675587 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m"] Oct 31 13:00:51.892664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:51.885998 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" Oct 31 13:00:51.892664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:51.889721 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" Oct 31 13:00:51.937766 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:51.930211 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" Oct 31 13:00:52.145661 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:52.144755 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m" event={"ID":"6a1ee165-b894-429a-b703-d99a2839af4f","Type":"ContainerStarted","Data":"2159e177bef853bc6f8161d4297d8c93d47d3c6d6969535fd040d017fd16d46c"} Oct 31 13:00:53.151403 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:53.151367 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m" event={"ID":"6a1ee165-b894-429a-b703-d99a2839af4f","Type":"ContainerStarted","Data":"e688e6a208d6b3b99cf4c1a5fd59cf1fa1ef10c4f7e59576127a62b530d98195"} Oct 31 13:00:53.151896 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:53.151616 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m" Oct 31 13:00:57.693829 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:57.693792 5693 patch_prober.go:29] interesting pod/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:00:57.694338 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:57.693870 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" podUID="b8b99947-a9a4-4b81-83ee-6cafebb48503" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:00:59.163325 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:59.163295 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m" Oct 31 13:00:59.262517 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:00:59.262466 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m" podStartSLOduration=8.707510669 podStartE2EDuration="9.262450881s" totalImagesPullingTime="554.940212ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:53.200940042 +0000 UTC m=+1703.908526337" watchObservedRunningTime="2026-10-31 13:00:59.262450881 +0000 UTC m=+1709.970037221" Oct 31 13:01:00.211511 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:00.211480 5693 generic.go:356] "Generic (PLEG): container finished" podID="b8b99947-a9a4-4b81-83ee-6cafebb48503" containerID="206d0f7ab203dda912103ebe173a851f305b79c00aed7959dfa1da8b48f095a0" exitCode=0 Oct 31 13:01:00.211975 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:00.211552 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" event={"ID":"b8b99947-a9a4-4b81-83ee-6cafebb48503","Type":"ContainerDied","Data":"206d0f7ab203dda912103ebe173a851f305b79c00aed7959dfa1da8b48f095a0"} Oct 31 13:01:00.212041 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:00.212027 5693 scope.go:122] "RemoveContainer" containerID="206d0f7ab203dda912103ebe173a851f305b79c00aed7959dfa1da8b48f095a0" Oct 31 13:01:00.213611 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:00.213583 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:01:00.213690 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:00.213667 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:01:00.213729 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:00.213698 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:01:01.168023 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:01.167988 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-558f58b547-wwbxw"] Oct 31 13:01:01.171354 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:01.171336 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-558f58b547-wwbxw" Oct 31 13:01:01.220434 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:01.220401 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" event={"ID":"b8b99947-a9a4-4b81-83ee-6cafebb48503","Type":"ContainerStarted","Data":"a11b518544fc0212d8560689d0f954d11e490bf75acb914a4da5a45aa978583f"} Oct 31 13:01:01.237776 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:01.237748 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3c8c51\"" Oct 31 13:01:01.241682 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:01.241654 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-558f58b547-wwbxw"] Oct 31 13:01:01.253779 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:01.253746 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wjj7\" (UniqueName: \"kubernetes.io/projected/ce656873-15e8-4959-aa03-23bcd234a3f2-kube-api-access-5wjj7\") pod \"payload-pre-processing-e2e-ipp-b-3c8c51-558f58b547-wwbxw\" (UID: \"ce656873-15e8-4959-aa03-23bcd234a3f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-558f58b547-wwbxw" Oct 31 13:01:01.253983 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:01.253960 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce656873-15e8-4959-aa03-23bcd234a3f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3c8c51-558f58b547-wwbxw\" (UID: \"ce656873-15e8-4959-aa03-23bcd234a3f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-558f58b547-wwbxw" Oct 31 13:01:01.384198 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:01.384156 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce656873-15e8-4959-aa03-23bcd234a3f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3c8c51-558f58b547-wwbxw\" (UID: \"ce656873-15e8-4959-aa03-23bcd234a3f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-558f58b547-wwbxw" Oct 31 13:01:01.384756 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:01.384736 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wjj7\" (UniqueName: \"kubernetes.io/projected/ce656873-15e8-4959-aa03-23bcd234a3f2-kube-api-access-5wjj7\") pod \"payload-pre-processing-e2e-ipp-b-3c8c51-558f58b547-wwbxw\" (UID: \"ce656873-15e8-4959-aa03-23bcd234a3f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-558f58b547-wwbxw" Oct 31 13:01:01.410667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:01.386241 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce656873-15e8-4959-aa03-23bcd234a3f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3c8c51-558f58b547-wwbxw\" (UID: \"ce656873-15e8-4959-aa03-23bcd234a3f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-558f58b547-wwbxw" Oct 31 13:01:01.455967 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:01.452790 5693 patch_prober.go:29] interesting pod/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 13:01:01.456231 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:01.456201 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" podUID="b8b99947-a9a4-4b81-83ee-6cafebb48503" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 13:01:01.462670 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:01.462329 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wjj7\" (UniqueName: \"kubernetes.io/projected/ce656873-15e8-4959-aa03-23bcd234a3f2-kube-api-access-5wjj7\") pod \"payload-pre-processing-e2e-ipp-b-3c8c51-558f58b547-wwbxw\" (UID: \"ce656873-15e8-4959-aa03-23bcd234a3f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-558f58b547-wwbxw" Oct 31 13:01:01.531947 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:01.531896 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-558f58b547-wwbxw" Oct 31 13:01:01.592689 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:01.586450 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" Oct 31 13:01:02.016138 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:02.016068 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-558f58b547-wwbxw"] Oct 31 13:01:02.024119 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:01:02.024086 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce656873_15e8_4959_aa03_23bcd234a3f2.slice/crio-6d6fa0ba1055a80bcb5196d5f7736326331b6d9dde48cefc910094a5788955e3 WatchSource:0}: Error finding container 6d6fa0ba1055a80bcb5196d5f7736326331b6d9dde48cefc910094a5788955e3: Status 404 returned error can't find the container with id 6d6fa0ba1055a80bcb5196d5f7736326331b6d9dde48cefc910094a5788955e3 Oct 31 13:01:02.228468 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:02.228426 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-558f58b547-wwbxw" event={"ID":"ce656873-15e8-4959-aa03-23bcd234a3f2","Type":"ContainerStarted","Data":"e5d07bc84702b5c6bd7dc9cba966ae0167f45375f660e3407aee31f359750bcd"} Oct 31 13:01:02.228899 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:02.228493 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-558f58b547-wwbxw" event={"ID":"ce656873-15e8-4959-aa03-23bcd234a3f2","Type":"ContainerStarted","Data":"6d6fa0ba1055a80bcb5196d5f7736326331b6d9dde48cefc910094a5788955e3"} Oct 31 13:01:02.228899 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:02.228523 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-558f58b547-wwbxw" Oct 31 13:01:02.257474 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:02.257415 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-558f58b547-wwbxw" podStartSLOduration=1.257397605 podStartE2EDuration="1.257397605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:01:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:01:02.25715498 +0000 UTC m=+1712.964741289" watchObservedRunningTime="2026-10-31 13:01:02.257397605 +0000 UTC m=+1712.964983901" Oct 31 13:01:02.423669 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:02.419170 5693 patch_prober.go:29] interesting pod/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 13:01:02.423669 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:02.419229 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" podUID="b8b99947-a9a4-4b81-83ee-6cafebb48503" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 13:01:02.564962 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:02.564933 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs"] Oct 31 13:01:02.568315 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:02.568293 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs" Oct 31 13:01:02.602810 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:02.593968 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-dc2db4-serving-cert\"" Oct 31 13:01:02.603928 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:02.603896 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs"] Oct 31 13:01:02.671530 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:02.671495 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/659d976b-0eec-480e-ab29-c95814ab7e11-metrics-certs\") pod \"maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs\" (UID: \"659d976b-0eec-480e-ab29-c95814ab7e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs" Oct 31 13:01:02.671766 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:02.671631 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/659d976b-0eec-480e-ab29-c95814ab7e11-openshift-service-ca\") pod \"maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs\" (UID: \"659d976b-0eec-480e-ab29-c95814ab7e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs" Oct 31 13:01:02.671921 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:02.671905 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/659d976b-0eec-480e-ab29-c95814ab7e11-maas-api-tls\") pod \"maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs\" (UID: \"659d976b-0eec-480e-ab29-c95814ab7e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs" Oct 31 13:01:02.671982 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:02.671958 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgbst\" (UniqueName: \"kubernetes.io/projected/659d976b-0eec-480e-ab29-c95814ab7e11-kube-api-access-pgbst\") pod \"maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs\" (UID: \"659d976b-0eec-480e-ab29-c95814ab7e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs" Oct 31 13:01:02.773417 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:02.773268 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/659d976b-0eec-480e-ab29-c95814ab7e11-metrics-certs\") pod \"maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs\" (UID: \"659d976b-0eec-480e-ab29-c95814ab7e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs" Oct 31 13:01:02.773855 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:02.773440 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/659d976b-0eec-480e-ab29-c95814ab7e11-openshift-service-ca\") pod \"maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs\" (UID: \"659d976b-0eec-480e-ab29-c95814ab7e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs" Oct 31 13:01:02.773855 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:02.773659 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/659d976b-0eec-480e-ab29-c95814ab7e11-maas-api-tls\") pod \"maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs\" (UID: \"659d976b-0eec-480e-ab29-c95814ab7e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs" Oct 31 13:01:02.773855 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:02.773741 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgbst\" (UniqueName: \"kubernetes.io/projected/659d976b-0eec-480e-ab29-c95814ab7e11-kube-api-access-pgbst\") pod \"maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs\" (UID: \"659d976b-0eec-480e-ab29-c95814ab7e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs" Oct 31 13:01:02.774964 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:02.774834 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/659d976b-0eec-480e-ab29-c95814ab7e11-openshift-service-ca\") pod \"maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs\" (UID: \"659d976b-0eec-480e-ab29-c95814ab7e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs" Oct 31 13:01:02.818745 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:02.818707 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/659d976b-0eec-480e-ab29-c95814ab7e11-maas-api-tls\") pod \"maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs\" (UID: \"659d976b-0eec-480e-ab29-c95814ab7e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs" Oct 31 13:01:02.839284 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:02.839258 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgbst\" (UniqueName: \"kubernetes.io/projected/659d976b-0eec-480e-ab29-c95814ab7e11-kube-api-access-pgbst\") pod \"maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs\" (UID: \"659d976b-0eec-480e-ab29-c95814ab7e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs" Oct 31 13:01:02.850676 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:02.849120 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/659d976b-0eec-480e-ab29-c95814ab7e11-metrics-certs\") pod \"maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs\" (UID: \"659d976b-0eec-480e-ab29-c95814ab7e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs" Oct 31 13:01:02.909952 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:02.909922 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs" Oct 31 13:01:03.320427 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:03.320394 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs"] Oct 31 13:01:03.323863 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:01:03.323833 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod659d976b_0eec_480e_ab29_c95814ab7e11.slice/crio-6e319d78905c1265b58a07bbc6c058df4ea05e1ffbd2d27e948965ef3b5abf02 WatchSource:0}: Error finding container 6e319d78905c1265b58a07bbc6c058df4ea05e1ffbd2d27e948965ef3b5abf02: Status 404 returned error can't find the container with id 6e319d78905c1265b58a07bbc6c058df4ea05e1ffbd2d27e948965ef3b5abf02 Oct 31 13:01:03.417670 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:03.416370 5693 patch_prober.go:29] interesting pod/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 13:01:03.417670 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:03.416417 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" podUID="b8b99947-a9a4-4b81-83ee-6cafebb48503" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 13:01:04.300721 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:04.285627 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs" event={"ID":"659d976b-0eec-480e-ab29-c95814ab7e11","Type":"ContainerStarted","Data":"27e5dcbbe36f564fa3697d686d279fdbff9e0efeb24866354dcbfd5eaa1d28b7"} Oct 31 13:01:04.300721 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:04.285714 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs" event={"ID":"659d976b-0eec-480e-ab29-c95814ab7e11","Type":"ContainerStarted","Data":"6e319d78905c1265b58a07bbc6c058df4ea05e1ffbd2d27e948965ef3b5abf02"} Oct 31 13:01:04.300721 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:04.287092 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs" Oct 31 13:01:04.340634 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:04.338594 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs" podStartSLOduration=1.8061994540000001 podStartE2EDuration="2.338572193s" totalImagesPullingTime="532.372739ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:01:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:01:04.334850433 +0000 UTC m=+1715.042436741" watchObservedRunningTime="2026-10-31 13:01:04.338572193 +0000 UTC m=+1715.046158490" Oct 31 13:01:04.412299 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:04.411715 5693 patch_prober.go:29] interesting pod/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 13:01:04.412299 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:04.411800 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" podUID="b8b99947-a9a4-4b81-83ee-6cafebb48503" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 13:01:04.446690 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:04.445775 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3128e5-85c845bff5-tw7sl"] Oct 31 13:01:05.408481 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:05.408449 5693 patch_prober.go:29] interesting pod/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 13:01:05.408922 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:05.408508 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" podUID="b8b99947-a9a4-4b81-83ee-6cafebb48503" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 13:01:06.409332 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:06.409289 5693 patch_prober.go:29] interesting pod/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 13:01:06.409926 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:06.409343 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" podUID="b8b99947-a9a4-4b81-83ee-6cafebb48503" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 13:01:07.413234 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:07.413205 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" Oct 31 13:01:07.413765 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:07.413558 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" Oct 31 13:01:07.414242 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:07.414223 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" Oct 31 13:01:09.446772 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:09.446683 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3128e5-85c845bff5-tw7sl" podUID="f5662f90-e280-43aa-8f47-4957c0c14431" containerName="payload-pre-processing" containerID="cri-o://56ec4611c7a72b7f1ef6cb17d77a88d03cfa6620c00fff21606a3f4bf9d0d582" gracePeriod=25 Oct 31 13:01:09.768322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:09.766025 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3128e5-85c845bff5-tw7sl" Oct 31 13:01:09.879730 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:09.879693 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5662f90-e280-43aa-8f47-4957c0c14431-kube-api-access-fnnz4\" (UniqueName: \"kubernetes.io/projected/f5662f90-e280-43aa-8f47-4957c0c14431-kube-api-access-fnnz4\") pod \"f5662f90-e280-43aa-8f47-4957c0c14431\" (UID: \"f5662f90-e280-43aa-8f47-4957c0c14431\") " Oct 31 13:01:09.880020 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:09.880002 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5662f90-e280-43aa-8f47-4957c0c14431-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5662f90-e280-43aa-8f47-4957c0c14431-plugins-config-volume\") pod \"f5662f90-e280-43aa-8f47-4957c0c14431\" (UID: \"f5662f90-e280-43aa-8f47-4957c0c14431\") " Oct 31 13:01:09.880384 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:09.880357 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5662f90-e280-43aa-8f47-4957c0c14431-plugins-config-volume" pod "f5662f90-e280-43aa-8f47-4957c0c14431" (UID: "f5662f90-e280-43aa-8f47-4957c0c14431"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:01:09.880566 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:09.880549 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5662f90-e280-43aa-8f47-4957c0c14431-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:09.882009 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:09.881978 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5662f90-e280-43aa-8f47-4957c0c14431-kube-api-access-fnnz4" pod "f5662f90-e280-43aa-8f47-4957c0c14431" (UID: "f5662f90-e280-43aa-8f47-4957c0c14431"). InnerVolumeSpecName "kube-api-access-fnnz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:01:09.981923 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:09.981835 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnnz4\" (UniqueName: \"kubernetes.io/projected/f5662f90-e280-43aa-8f47-4957c0c14431-kube-api-access-fnnz4\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:10.356725 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:10.355246 5693 generic.go:356] "Generic (PLEG): container finished" podID="f5662f90-e280-43aa-8f47-4957c0c14431" containerID="56ec4611c7a72b7f1ef6cb17d77a88d03cfa6620c00fff21606a3f4bf9d0d582" exitCode=0 Oct 31 13:01:10.356725 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:10.355310 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3128e5-85c845bff5-tw7sl" Oct 31 13:01:10.356725 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:10.355332 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3128e5-85c845bff5-tw7sl" event={"ID":"f5662f90-e280-43aa-8f47-4957c0c14431","Type":"ContainerDied","Data":"56ec4611c7a72b7f1ef6cb17d77a88d03cfa6620c00fff21606a3f4bf9d0d582"} Oct 31 13:01:10.356725 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:10.355380 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3128e5-85c845bff5-tw7sl" event={"ID":"f5662f90-e280-43aa-8f47-4957c0c14431","Type":"ContainerDied","Data":"29ad09f7d95e20d953a18738b1678cbad024203cece879da92ed82d22553dabe"} Oct 31 13:01:10.356725 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:10.355411 5693 scope.go:122] "RemoveContainer" containerID="56ec4611c7a72b7f1ef6cb17d77a88d03cfa6620c00fff21606a3f4bf9d0d582" Oct 31 13:01:10.374450 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:10.374432 5693 scope.go:122] "RemoveContainer" containerID="56ec4611c7a72b7f1ef6cb17d77a88d03cfa6620c00fff21606a3f4bf9d0d582" Oct 31 13:01:10.374800 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:01:10.374775 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56ec4611c7a72b7f1ef6cb17d77a88d03cfa6620c00fff21606a3f4bf9d0d582\": container with ID starting with 56ec4611c7a72b7f1ef6cb17d77a88d03cfa6620c00fff21606a3f4bf9d0d582 not found: ID does not exist" containerID="56ec4611c7a72b7f1ef6cb17d77a88d03cfa6620c00fff21606a3f4bf9d0d582" Oct 31 13:01:10.374898 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:10.374812 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56ec4611c7a72b7f1ef6cb17d77a88d03cfa6620c00fff21606a3f4bf9d0d582"} err="failed to get container status \"56ec4611c7a72b7f1ef6cb17d77a88d03cfa6620c00fff21606a3f4bf9d0d582\": rpc error: code = NotFound desc = could not find container \"56ec4611c7a72b7f1ef6cb17d77a88d03cfa6620c00fff21606a3f4bf9d0d582\": container with ID starting with 56ec4611c7a72b7f1ef6cb17d77a88d03cfa6620c00fff21606a3f4bf9d0d582 not found: ID does not exist" Oct 31 13:01:10.389792 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:10.389755 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3128e5-85c845bff5-tw7sl"] Oct 31 13:01:10.408657 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:10.408603 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3128e5-85c845bff5-tw7sl"] Oct 31 13:01:10.932715 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:10.932683 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3128e5-59cc88c949-r42bb"] Oct 31 13:01:10.933183 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:10.932930 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3128e5-59cc88c949-r42bb" podUID="06baf8a7-b936-4a35-8bf6-a919e152bd62" containerName="maas-api" containerID="cri-o://cb36292757ebd355685ce0ccf2ffd4d62252a5629b949b520172042b58ac0246" gracePeriod=30 Oct 31 13:01:11.368446 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:11.367025 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs" Oct 31 13:01:11.372094 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:11.372063 5693 generic.go:356] "Generic (PLEG): container finished" podID="06baf8a7-b936-4a35-8bf6-a919e152bd62" containerID="cb36292757ebd355685ce0ccf2ffd4d62252a5629b949b520172042b58ac0246" exitCode=0 Oct 31 13:01:11.372094 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:11.372089 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3128e5-59cc88c949-r42bb" event={"ID":"06baf8a7-b936-4a35-8bf6-a919e152bd62","Type":"ContainerDied","Data":"cb36292757ebd355685ce0ccf2ffd4d62252a5629b949b520172042b58ac0246"} Oct 31 13:01:11.404049 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:11.404019 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3128e5-59cc88c949-r42bb" Oct 31 13:01:11.409848 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:11.409824 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5662f90-e280-43aa-8f47-4957c0c14431" path="/var/lib/kubelet/pods/f5662f90-e280-43aa-8f47-4957c0c14431/volumes" Oct 31 13:01:11.445417 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:11.445381 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06baf8a7-b936-4a35-8bf6-a919e152bd62-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06baf8a7-b936-4a35-8bf6-a919e152bd62-openshift-service-ca\") pod \"06baf8a7-b936-4a35-8bf6-a919e152bd62\" (UID: \"06baf8a7-b936-4a35-8bf6-a919e152bd62\") " Oct 31 13:01:11.445612 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:11.445547 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06baf8a7-b936-4a35-8bf6-a919e152bd62-kube-api-access-rkcbm\" (UniqueName: \"kubernetes.io/projected/06baf8a7-b936-4a35-8bf6-a919e152bd62-kube-api-access-rkcbm\") pod \"06baf8a7-b936-4a35-8bf6-a919e152bd62\" (UID: \"06baf8a7-b936-4a35-8bf6-a919e152bd62\") " Oct 31 13:01:11.445717 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:11.445624 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06baf8a7-b936-4a35-8bf6-a919e152bd62-metrics-certs\" (UniqueName: \"kubernetes.io/secret/06baf8a7-b936-4a35-8bf6-a919e152bd62-metrics-certs\") pod \"06baf8a7-b936-4a35-8bf6-a919e152bd62\" (UID: \"06baf8a7-b936-4a35-8bf6-a919e152bd62\") " Oct 31 13:01:11.446012 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:11.445963 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06baf8a7-b936-4a35-8bf6-a919e152bd62-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06baf8a7-b936-4a35-8bf6-a919e152bd62-maas-api-tls\") pod \"06baf8a7-b936-4a35-8bf6-a919e152bd62\" (UID: \"06baf8a7-b936-4a35-8bf6-a919e152bd62\") " Oct 31 13:01:11.446253 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:11.446230 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06baf8a7-b936-4a35-8bf6-a919e152bd62-openshift-service-ca" pod "06baf8a7-b936-4a35-8bf6-a919e152bd62" (UID: "06baf8a7-b936-4a35-8bf6-a919e152bd62"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:01:11.447091 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:11.447069 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06baf8a7-b936-4a35-8bf6-a919e152bd62-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:11.470621 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:11.470112 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06baf8a7-b936-4a35-8bf6-a919e152bd62-kube-api-access-rkcbm" pod "06baf8a7-b936-4a35-8bf6-a919e152bd62" (UID: "06baf8a7-b936-4a35-8bf6-a919e152bd62"). InnerVolumeSpecName "kube-api-access-rkcbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:01:11.470621 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:11.470142 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06baf8a7-b936-4a35-8bf6-a919e152bd62-metrics-certs" pod "06baf8a7-b936-4a35-8bf6-a919e152bd62" (UID: "06baf8a7-b936-4a35-8bf6-a919e152bd62"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:01:11.471964 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:11.471935 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06baf8a7-b936-4a35-8bf6-a919e152bd62-maas-api-tls" pod "06baf8a7-b936-4a35-8bf6-a919e152bd62" (UID: "06baf8a7-b936-4a35-8bf6-a919e152bd62"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:01:11.549187 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:11.549148 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06baf8a7-b936-4a35-8bf6-a919e152bd62-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:11.549187 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:11.549185 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkcbm\" (UniqueName: \"kubernetes.io/projected/06baf8a7-b936-4a35-8bf6-a919e152bd62-kube-api-access-rkcbm\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:11.549187 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:11.549195 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06baf8a7-b936-4a35-8bf6-a919e152bd62-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:12.380774 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:12.380745 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3128e5-59cc88c949-r42bb" Oct 31 13:01:12.381205 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:12.380745 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3128e5-59cc88c949-r42bb" event={"ID":"06baf8a7-b936-4a35-8bf6-a919e152bd62","Type":"ContainerDied","Data":"3e0288356ba76389970c0f9407c285f34638c932bea9fb428d6ce9ed381b7300"} Oct 31 13:01:12.381205 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:12.380883 5693 scope.go:122] "RemoveContainer" containerID="cb36292757ebd355685ce0ccf2ffd4d62252a5629b949b520172042b58ac0246" Oct 31 13:01:12.415479 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:12.415452 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3128e5-59cc88c949-r42bb"] Oct 31 13:01:12.421114 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:12.421095 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3128e5-59cc88c949-r42bb"] Oct 31 13:01:13.266752 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:13.265254 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-558f58b547-wwbxw" Oct 31 13:01:13.482086 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:13.482050 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06baf8a7-b936-4a35-8bf6-a919e152bd62" path="/var/lib/kubelet/pods/06baf8a7-b936-4a35-8bf6-a919e152bd62/volumes" Oct 31 13:01:13.943769 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:13.943735 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q"] Oct 31 13:01:13.944740 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:13.944703 5693 cpu_manager.go:414] "containerMap: removing container" podUID="f5662f90-e280-43aa-8f47-4957c0c14431" containerName="payload-pre-processing" Oct 31 13:01:13.944740 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:13.944727 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5662f90-e280-43aa-8f47-4957c0c14431" containerName="payload-pre-processing" Oct 31 13:01:13.944931 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:13.944785 5693 cpu_manager.go:414] "containerMap: removing container" podUID="06baf8a7-b936-4a35-8bf6-a919e152bd62" containerName="maas-api" Oct 31 13:01:13.944931 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:13.944794 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06baf8a7-b936-4a35-8bf6-a919e152bd62" containerName="maas-api" Oct 31 13:01:13.945036 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:13.944946 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="06baf8a7-b936-4a35-8bf6-a919e152bd62" containerName="maas-api" Oct 31 13:01:13.945036 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:13.944963 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5662f90-e280-43aa-8f47-4957c0c14431" containerName="payload-pre-processing" Oct 31 13:01:13.950712 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:13.950682 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q" Oct 31 13:01:13.982626 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:13.982596 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q"] Oct 31 13:01:14.004315 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:14.004283 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-3c8c51-serving-cert\"" Oct 31 13:01:14.121014 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:14.120968 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9-metrics-certs\") pod \"maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q\" (UID: \"0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q" Oct 31 13:01:14.121177 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:14.121086 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q\" (UID: \"0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q" Oct 31 13:01:14.121177 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:14.121138 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q\" (UID: \"0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q" Oct 31 13:01:14.121374 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:14.121356 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4t2g\" (UniqueName: \"kubernetes.io/projected/0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9-kube-api-access-q4t2g\") pod \"maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q\" (UID: \"0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q" Oct 31 13:01:14.223352 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:14.223253 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9-metrics-certs\") pod \"maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q\" (UID: \"0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q" Oct 31 13:01:14.223352 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:14.223321 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q\" (UID: \"0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q" Oct 31 13:01:14.223606 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:14.223360 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q\" (UID: \"0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q" Oct 31 13:01:14.223606 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:14.223510 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4t2g\" (UniqueName: \"kubernetes.io/projected/0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9-kube-api-access-q4t2g\") pod \"maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q\" (UID: \"0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q" Oct 31 13:01:14.224382 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:14.224353 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q\" (UID: \"0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q" Oct 31 13:01:14.226040 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:14.226019 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9-metrics-certs\") pod \"maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q\" (UID: \"0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q" Oct 31 13:01:14.226156 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:14.226047 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q\" (UID: \"0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q" Oct 31 13:01:14.238045 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:14.238012 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4t2g\" (UniqueName: \"kubernetes.io/projected/0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9-kube-api-access-q4t2g\") pod \"maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q\" (UID: \"0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q" Oct 31 13:01:14.281664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:14.276920 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q" Oct 31 13:01:14.848858 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:14.842384 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q"] Oct 31 13:01:15.636099 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:15.635950 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q" event={"ID":"0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9","Type":"ContainerStarted","Data":"ef995ecbad39fd47cade566536c20ff9964082b54e85b6ebf3d8edbf03058637"} Oct 31 13:01:16.650791 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:16.650754 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q" event={"ID":"0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9","Type":"ContainerStarted","Data":"4bc61ca98da9d3c4c1ee6bbab056a89bd71db98b3e579d1b6cf0afdd4dde6d69"} Oct 31 13:01:16.651269 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:16.650867 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q" Oct 31 13:01:16.703022 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:16.702955 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q" podStartSLOduration=2.930284338 podStartE2EDuration="3.702934439s" totalImagesPullingTime="772.650101ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:01:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:01:16.698345839 +0000 UTC m=+1727.405932137" watchObservedRunningTime="2026-10-31 13:01:16.702934439 +0000 UTC m=+1727.410520716" Oct 31 13:01:18.145564 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:18.145534 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42"] Oct 31 13:01:18.145977 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:18.145751 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" podUID="9e427b40-6255-497b-a5a3-7aa06c339de9" containerName="istio-proxy" containerID="cri-o://be5ae300e5daa207779b536bf6a633c60764381f859208b2e40944e5e0bbf6ac" gracePeriod=30 Oct 31 13:01:18.708302 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:01:18.708263 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:01:21.889805 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:21.889756 5693 patch_prober.go:29] interesting pod/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:01:21.890210 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:21.889831 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" podUID="9e427b40-6255-497b-a5a3-7aa06c339de9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:01:22.665912 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:22.665885 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q" Oct 31 13:01:23.429087 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:23.429024 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" Oct 31 13:01:23.534591 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:23.534560 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e427b40-6255-497b-a5a3-7aa06c339de9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e427b40-6255-497b-a5a3-7aa06c339de9-workload-socket\") pod \"9e427b40-6255-497b-a5a3-7aa06c339de9\" (UID: \"9e427b40-6255-497b-a5a3-7aa06c339de9\") " Oct 31 13:01:23.534799 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:23.534617 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e427b40-6255-497b-a5a3-7aa06c339de9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e427b40-6255-497b-a5a3-7aa06c339de9-workload-certs\") pod \"9e427b40-6255-497b-a5a3-7aa06c339de9\" (UID: \"9e427b40-6255-497b-a5a3-7aa06c339de9\") " Oct 31 13:01:23.534799 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:23.534675 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e427b40-6255-497b-a5a3-7aa06c339de9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e427b40-6255-497b-a5a3-7aa06c339de9-credential-socket\") pod \"9e427b40-6255-497b-a5a3-7aa06c339de9\" (UID: \"9e427b40-6255-497b-a5a3-7aa06c339de9\") " Oct 31 13:01:23.534799 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:23.534786 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e427b40-6255-497b-a5a3-7aa06c339de9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e427b40-6255-497b-a5a3-7aa06c339de9-istiod-ca-cert\") pod \"9e427b40-6255-497b-a5a3-7aa06c339de9\" (UID: \"9e427b40-6255-497b-a5a3-7aa06c339de9\") " Oct 31 13:01:23.534971 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:23.534868 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e427b40-6255-497b-a5a3-7aa06c339de9-istio-token\" (UniqueName: \"kubernetes.io/projected/9e427b40-6255-497b-a5a3-7aa06c339de9-istio-token\") pod \"9e427b40-6255-497b-a5a3-7aa06c339de9\" (UID: \"9e427b40-6255-497b-a5a3-7aa06c339de9\") " Oct 31 13:01:23.534971 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:23.534948 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e427b40-6255-497b-a5a3-7aa06c339de9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e427b40-6255-497b-a5a3-7aa06c339de9-istio-data\") pod \"9e427b40-6255-497b-a5a3-7aa06c339de9\" (UID: \"9e427b40-6255-497b-a5a3-7aa06c339de9\") " Oct 31 13:01:23.535077 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:23.534982 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e427b40-6255-497b-a5a3-7aa06c339de9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e427b40-6255-497b-a5a3-7aa06c339de9-istio-envoy\") pod \"9e427b40-6255-497b-a5a3-7aa06c339de9\" (UID: \"9e427b40-6255-497b-a5a3-7aa06c339de9\") " Oct 31 13:01:23.535077 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:23.535015 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e427b40-6255-497b-a5a3-7aa06c339de9-kube-api-access-q6bkq\" (UniqueName: \"kubernetes.io/projected/9e427b40-6255-497b-a5a3-7aa06c339de9-kube-api-access-q6bkq\") pod \"9e427b40-6255-497b-a5a3-7aa06c339de9\" (UID: \"9e427b40-6255-497b-a5a3-7aa06c339de9\") " Oct 31 13:01:23.536883 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:23.535212 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e427b40-6255-497b-a5a3-7aa06c339de9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e427b40-6255-497b-a5a3-7aa06c339de9-istio-podinfo\") pod \"9e427b40-6255-497b-a5a3-7aa06c339de9\" (UID: \"9e427b40-6255-497b-a5a3-7aa06c339de9\") " Oct 31 13:01:23.536883 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:23.536462 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e427b40-6255-497b-a5a3-7aa06c339de9-workload-certs" pod "9e427b40-6255-497b-a5a3-7aa06c339de9" (UID: "9e427b40-6255-497b-a5a3-7aa06c339de9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:01:23.536883 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:23.536567 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e427b40-6255-497b-a5a3-7aa06c339de9-workload-socket" pod "9e427b40-6255-497b-a5a3-7aa06c339de9" (UID: "9e427b40-6255-497b-a5a3-7aa06c339de9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:01:23.536883 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:23.536788 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e427b40-6255-497b-a5a3-7aa06c339de9-credential-socket" pod "9e427b40-6255-497b-a5a3-7aa06c339de9" (UID: "9e427b40-6255-497b-a5a3-7aa06c339de9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:01:23.536883 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:23.536859 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e427b40-6255-497b-a5a3-7aa06c339de9-istio-data" pod "9e427b40-6255-497b-a5a3-7aa06c339de9" (UID: "9e427b40-6255-497b-a5a3-7aa06c339de9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:01:23.537655 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:23.537602 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e427b40-6255-497b-a5a3-7aa06c339de9-istiod-ca-cert" pod "9e427b40-6255-497b-a5a3-7aa06c339de9" (UID: "9e427b40-6255-497b-a5a3-7aa06c339de9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:01:23.543124 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:23.543003 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e427b40-6255-497b-a5a3-7aa06c339de9-istio-token" pod "9e427b40-6255-497b-a5a3-7aa06c339de9" (UID: "9e427b40-6255-497b-a5a3-7aa06c339de9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:01:23.543124 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:23.543102 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e427b40-6255-497b-a5a3-7aa06c339de9-kube-api-access-q6bkq" pod "9e427b40-6255-497b-a5a3-7aa06c339de9" (UID: "9e427b40-6255-497b-a5a3-7aa06c339de9"). InnerVolumeSpecName "kube-api-access-q6bkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:01:23.543390 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:23.543360 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e427b40-6255-497b-a5a3-7aa06c339de9-istio-podinfo" pod "9e427b40-6255-497b-a5a3-7aa06c339de9" (UID: "9e427b40-6255-497b-a5a3-7aa06c339de9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:01:23.544345 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:23.544317 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e427b40-6255-497b-a5a3-7aa06c339de9-istio-envoy" pod "9e427b40-6255-497b-a5a3-7aa06c339de9" (UID: "9e427b40-6255-497b-a5a3-7aa06c339de9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:01:23.637867 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:23.637825 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e427b40-6255-497b-a5a3-7aa06c339de9-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:23.637867 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:23.637865 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e427b40-6255-497b-a5a3-7aa06c339de9-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:23.638104 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:23.637880 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e427b40-6255-497b-a5a3-7aa06c339de9-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:23.638104 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:23.637897 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e427b40-6255-497b-a5a3-7aa06c339de9-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:23.638104 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:23.637910 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e427b40-6255-497b-a5a3-7aa06c339de9-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:23.638104 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:23.637922 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e427b40-6255-497b-a5a3-7aa06c339de9-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:23.638104 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:23.637935 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e427b40-6255-497b-a5a3-7aa06c339de9-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:23.638104 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:23.637949 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e427b40-6255-497b-a5a3-7aa06c339de9-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:23.638104 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:23.637962 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6bkq\" (UniqueName: \"kubernetes.io/projected/9e427b40-6255-497b-a5a3-7aa06c339de9-kube-api-access-q6bkq\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:23.705513 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:23.705432 5693 generic.go:356] "Generic (PLEG): container finished" podID="9e427b40-6255-497b-a5a3-7aa06c339de9" containerID="be5ae300e5daa207779b536bf6a633c60764381f859208b2e40944e5e0bbf6ac" exitCode=0 Oct 31 13:01:23.705513 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:23.705475 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" Oct 31 13:01:23.705756 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:23.705517 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" event={"ID":"9e427b40-6255-497b-a5a3-7aa06c339de9","Type":"ContainerDied","Data":"be5ae300e5daa207779b536bf6a633c60764381f859208b2e40944e5e0bbf6ac"} Oct 31 13:01:23.705756 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:23.705575 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42" event={"ID":"9e427b40-6255-497b-a5a3-7aa06c339de9","Type":"ContainerDied","Data":"116f6c7fa019c0b40c7befa2b84fb529eff6f716b19ec663bdecc57c47825d28"} Oct 31 13:01:23.705756 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:23.705604 5693 scope.go:122] "RemoveContainer" containerID="be5ae300e5daa207779b536bf6a633c60764381f859208b2e40944e5e0bbf6ac" Oct 31 13:01:23.717877 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:23.717857 5693 scope.go:122] "RemoveContainer" containerID="be5ae300e5daa207779b536bf6a633c60764381f859208b2e40944e5e0bbf6ac" Oct 31 13:01:23.718234 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:01:23.718200 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be5ae300e5daa207779b536bf6a633c60764381f859208b2e40944e5e0bbf6ac\": container with ID starting with be5ae300e5daa207779b536bf6a633c60764381f859208b2e40944e5e0bbf6ac not found: ID does not exist" containerID="be5ae300e5daa207779b536bf6a633c60764381f859208b2e40944e5e0bbf6ac" Oct 31 13:01:23.718307 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:23.718246 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be5ae300e5daa207779b536bf6a633c60764381f859208b2e40944e5e0bbf6ac"} err="failed to get container status \"be5ae300e5daa207779b536bf6a633c60764381f859208b2e40944e5e0bbf6ac\": rpc error: code = NotFound desc = could not find container \"be5ae300e5daa207779b536bf6a633c60764381f859208b2e40944e5e0bbf6ac\": container with ID starting with be5ae300e5daa207779b536bf6a633c60764381f859208b2e40944e5e0bbf6ac not found: ID does not exist" Oct 31 13:01:23.743308 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:23.743281 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42"] Oct 31 13:01:23.764957 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:23.764927 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-l5w42"] Oct 31 13:01:24.527673 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:24.525513 5693 patch_prober.go:29] interesting pod/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": read tcp 10.133.0.2:59840->10.133.0.99:15021: read: connection reset by peer" start-of-body= Oct 31 13:01:24.527673 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:24.525576 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" podUID="1c8d2a80-a147-44f0-b49b-c3bd5b29fef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": read tcp 10.133.0.2:59840->10.133.0.99:15021: read: connection reset by peer" Oct 31 13:01:24.727771 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:24.723760 5693 generic.go:356] "Generic (PLEG): container finished" podID="1c8d2a80-a147-44f0-b49b-c3bd5b29fef0" containerID="04d076bc5e9ccd0ffcf480ba0b8e33c104bfc85033cf970007f7c761f5205dfb" exitCode=0 Oct 31 13:01:24.727771 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:24.723921 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" event={"ID":"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0","Type":"ContainerDied","Data":"04d076bc5e9ccd0ffcf480ba0b8e33c104bfc85033cf970007f7c761f5205dfb"} Oct 31 13:01:24.727771 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:24.724442 5693 scope.go:122] "RemoveContainer" containerID="04d076bc5e9ccd0ffcf480ba0b8e33c104bfc85033cf970007f7c761f5205dfb" Oct 31 13:01:24.727771 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:24.727101 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:01:24.727771 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:24.727193 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:01:24.727771 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:24.727238 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:01:25.415599 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:25.415566 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e427b40-6255-497b-a5a3-7aa06c339de9" path="/var/lib/kubelet/pods/9e427b40-6255-497b-a5a3-7aa06c339de9/volumes" Oct 31 13:01:25.784754 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:25.772406 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" event={"ID":"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0","Type":"ContainerStarted","Data":"c551421591d8d19afd9e7798a8a3d5e34c30cdf05365097a407cab874f6cac12"} Oct 31 13:01:25.860666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:25.859984 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" Oct 31 13:01:25.860666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:25.860103 5693 patch_prober.go:29] interesting pod/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 31 13:01:25.860666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:25.860150 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" podUID="1c8d2a80-a147-44f0-b49b-c3bd5b29fef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 31 13:01:26.855774 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:26.855732 5693 patch_prober.go:29] interesting pod/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 31 13:01:26.856269 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:26.855821 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" podUID="1c8d2a80-a147-44f0-b49b-c3bd5b29fef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 31 13:01:27.124852 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:27.124761 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-adfd11-758bb459dd-zvhmq"] Oct 31 13:01:27.125474 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:27.125452 5693 cpu_manager.go:414] "containerMap: removing container" podUID="9e427b40-6255-497b-a5a3-7aa06c339de9" containerName="istio-proxy" Oct 31 13:01:27.125584 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:27.125476 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e427b40-6255-497b-a5a3-7aa06c339de9" containerName="istio-proxy" Oct 31 13:01:27.125696 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:27.125682 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e427b40-6255-497b-a5a3-7aa06c339de9" containerName="istio-proxy" Oct 31 13:01:27.129758 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:27.129736 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-adfd11-758bb459dd-zvhmq" Oct 31 13:01:27.141778 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:27.141757 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-adfd11-dockercfg-928z5\"" Oct 31 13:01:27.147742 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:27.147713 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-adfd11-758bb459dd-zvhmq"] Oct 31 13:01:27.217662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:27.199471 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-adfd11\"" Oct 31 13:01:27.298730 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:27.223871 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74e3eaa3-e742-48ae-a5de-c923170b9f26-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-adfd11-758bb459dd-zvhmq\" (UID: \"74e3eaa3-e742-48ae-a5de-c923170b9f26\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-adfd11-758bb459dd-zvhmq" Oct 31 13:01:27.299149 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:27.299116 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5khk\" (UniqueName: \"kubernetes.io/projected/74e3eaa3-e742-48ae-a5de-c923170b9f26-kube-api-access-w5khk\") pod \"payload-processing-e2e-ipp-a-adfd11-758bb459dd-zvhmq\" (UID: \"74e3eaa3-e742-48ae-a5de-c923170b9f26\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-adfd11-758bb459dd-zvhmq" Oct 31 13:01:27.415596 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:27.415555 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74e3eaa3-e742-48ae-a5de-c923170b9f26-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-adfd11-758bb459dd-zvhmq\" (UID: \"74e3eaa3-e742-48ae-a5de-c923170b9f26\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-adfd11-758bb459dd-zvhmq" Oct 31 13:01:27.415980 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:27.415957 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5khk\" (UniqueName: \"kubernetes.io/projected/74e3eaa3-e742-48ae-a5de-c923170b9f26-kube-api-access-w5khk\") pod \"payload-processing-e2e-ipp-a-adfd11-758bb459dd-zvhmq\" (UID: \"74e3eaa3-e742-48ae-a5de-c923170b9f26\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-adfd11-758bb459dd-zvhmq" Oct 31 13:01:27.417611 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:27.417586 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74e3eaa3-e742-48ae-a5de-c923170b9f26-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-adfd11-758bb459dd-zvhmq\" (UID: \"74e3eaa3-e742-48ae-a5de-c923170b9f26\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-adfd11-758bb459dd-zvhmq" Oct 31 13:01:27.460442 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:27.460404 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5khk\" (UniqueName: \"kubernetes.io/projected/74e3eaa3-e742-48ae-a5de-c923170b9f26-kube-api-access-w5khk\") pod \"payload-processing-e2e-ipp-a-adfd11-758bb459dd-zvhmq\" (UID: \"74e3eaa3-e742-48ae-a5de-c923170b9f26\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-adfd11-758bb459dd-zvhmq" Oct 31 13:01:27.465040 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:27.465002 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-adfd11-758bb459dd-zvhmq" Oct 31 13:01:27.813199 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:27.813120 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-adfd11-758bb459dd-zvhmq"] Oct 31 13:01:27.820802 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:01:27.820777 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74e3eaa3_e742_48ae_a5de_c923170b9f26.slice/crio-d6b3b54fc01cf000d81a70d613b46fa7db661d2f3642d94083f3c6afebb5d7fe WatchSource:0}: Error finding container d6b3b54fc01cf000d81a70d613b46fa7db661d2f3642d94083f3c6afebb5d7fe: Status 404 returned error can't find the container with id d6b3b54fc01cf000d81a70d613b46fa7db661d2f3642d94083f3c6afebb5d7fe Oct 31 13:01:27.855058 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:27.855022 5693 patch_prober.go:29] interesting pod/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 31 13:01:27.855155 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:27.855086 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" podUID="1c8d2a80-a147-44f0-b49b-c3bd5b29fef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 31 13:01:28.826362 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:28.826333 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-adfd11-758bb459dd-zvhmq" event={"ID":"74e3eaa3-e742-48ae-a5de-c923170b9f26","Type":"ContainerStarted","Data":"1e84d7691aafe9444fcc1927e91786dd806ec8caf04ed3f3d865f6a159e4d22f"} Oct 31 13:01:28.826795 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:28.826383 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-adfd11-758bb459dd-zvhmq" event={"ID":"74e3eaa3-e742-48ae-a5de-c923170b9f26","Type":"ContainerStarted","Data":"d6b3b54fc01cf000d81a70d613b46fa7db661d2f3642d94083f3c6afebb5d7fe"} Oct 31 13:01:28.826795 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:28.826416 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-adfd11-758bb459dd-zvhmq" Oct 31 13:01:28.853863 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:28.853818 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-adfd11-758bb459dd-zvhmq" podStartSLOduration=1.853803468 podStartE2EDuration="1.853803468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:01:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:01:28.852944989 +0000 UTC m=+1739.560531284" watchObservedRunningTime="2026-10-31 13:01:28.853803468 +0000 UTC m=+1739.561389763" Oct 31 13:01:28.854547 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:28.854521 5693 patch_prober.go:29] interesting pod/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 31 13:01:28.854697 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:28.854573 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" podUID="1c8d2a80-a147-44f0-b49b-c3bd5b29fef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 31 13:01:29.854980 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:29.854945 5693 patch_prober.go:29] interesting pod/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 31 13:01:29.855371 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:29.854998 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" podUID="1c8d2a80-a147-44f0-b49b-c3bd5b29fef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 31 13:01:30.873451 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:30.873392 5693 patch_prober.go:29] interesting pod/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 31 13:01:30.874203 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:30.873455 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" podUID="1c8d2a80-a147-44f0-b49b-c3bd5b29fef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 31 13:01:31.863954 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:31.863922 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" Oct 31 13:01:31.864941 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:31.864915 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" Oct 31 13:01:31.866310 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:31.866285 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" Oct 31 13:01:35.075094 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:35.075060 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-558f58b547-wwbxw"] Oct 31 13:01:39.836767 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:39.836737 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-adfd11-758bb459dd-zvhmq" Oct 31 13:01:40.076978 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:40.076928 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-558f58b547-wwbxw" podUID="ce656873-15e8-4959-aa03-23bcd234a3f2" containerName="payload-pre-processing" containerID="cri-o://e5d07bc84702b5c6bd7dc9cba966ae0167f45375f660e3407aee31f359750bcd" gracePeriod=25 Oct 31 13:01:40.377590 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:40.377560 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-558f58b547-wwbxw" Oct 31 13:01:40.417175 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:40.415915 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce656873-15e8-4959-aa03-23bcd234a3f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce656873-15e8-4959-aa03-23bcd234a3f2-plugins-config-volume\") pod \"ce656873-15e8-4959-aa03-23bcd234a3f2\" (UID: \"ce656873-15e8-4959-aa03-23bcd234a3f2\") " Oct 31 13:01:40.417175 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:40.415977 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce656873-15e8-4959-aa03-23bcd234a3f2-kube-api-access-5wjj7\" (UniqueName: \"kubernetes.io/projected/ce656873-15e8-4959-aa03-23bcd234a3f2-kube-api-access-5wjj7\") pod \"ce656873-15e8-4959-aa03-23bcd234a3f2\" (UID: \"ce656873-15e8-4959-aa03-23bcd234a3f2\") " Oct 31 13:01:40.417888 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:40.417831 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce656873-15e8-4959-aa03-23bcd234a3f2-plugins-config-volume" pod "ce656873-15e8-4959-aa03-23bcd234a3f2" (UID: "ce656873-15e8-4959-aa03-23bcd234a3f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:01:40.426154 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:40.426115 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j"] Oct 31 13:01:40.426574 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:40.426542 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j" podUID="ddfdedb8-f94c-4b8b-a717-42b9b8b3642c" containerName="maas-api" containerID="cri-o://f240f8d2ac8eb471a2064dc890a34460cecd5a15bda5f437e23b3fa139eba818" gracePeriod=30 Oct 31 13:01:40.434125 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:40.434093 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce656873-15e8-4959-aa03-23bcd234a3f2-kube-api-access-5wjj7" pod "ce656873-15e8-4959-aa03-23bcd234a3f2" (UID: "ce656873-15e8-4959-aa03-23bcd234a3f2"). InnerVolumeSpecName "kube-api-access-5wjj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:01:40.518020 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:40.517992 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce656873-15e8-4959-aa03-23bcd234a3f2-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:40.518020 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:40.518018 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wjj7\" (UniqueName: \"kubernetes.io/projected/ce656873-15e8-4959-aa03-23bcd234a3f2-kube-api-access-5wjj7\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:40.952903 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:40.951818 5693 generic.go:356] "Generic (PLEG): container finished" podID="ddfdedb8-f94c-4b8b-a717-42b9b8b3642c" containerID="f240f8d2ac8eb471a2064dc890a34460cecd5a15bda5f437e23b3fa139eba818" exitCode=0 Oct 31 13:01:40.952903 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:40.951932 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j" event={"ID":"ddfdedb8-f94c-4b8b-a717-42b9b8b3642c","Type":"ContainerDied","Data":"f240f8d2ac8eb471a2064dc890a34460cecd5a15bda5f437e23b3fa139eba818"} Oct 31 13:01:40.952903 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:40.951981 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j" event={"ID":"ddfdedb8-f94c-4b8b-a717-42b9b8b3642c","Type":"ContainerDied","Data":"fa113e41f73eb249ea981e79540da2153193290174678cb4c3270a76a6e056c5"} Oct 31 13:01:40.952903 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:40.952004 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa113e41f73eb249ea981e79540da2153193290174678cb4c3270a76a6e056c5" Oct 31 13:01:40.953815 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:40.953790 5693 generic.go:356] "Generic (PLEG): container finished" podID="ce656873-15e8-4959-aa03-23bcd234a3f2" containerID="e5d07bc84702b5c6bd7dc9cba966ae0167f45375f660e3407aee31f359750bcd" exitCode=0 Oct 31 13:01:40.953904 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:40.953832 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-558f58b547-wwbxw" Oct 31 13:01:40.953904 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:40.953874 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-558f58b547-wwbxw" event={"ID":"ce656873-15e8-4959-aa03-23bcd234a3f2","Type":"ContainerDied","Data":"e5d07bc84702b5c6bd7dc9cba966ae0167f45375f660e3407aee31f359750bcd"} Oct 31 13:01:40.953989 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:40.953921 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-558f58b547-wwbxw" event={"ID":"ce656873-15e8-4959-aa03-23bcd234a3f2","Type":"ContainerDied","Data":"6d6fa0ba1055a80bcb5196d5f7736326331b6d9dde48cefc910094a5788955e3"} Oct 31 13:01:40.953989 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:40.953950 5693 scope.go:122] "RemoveContainer" containerID="e5d07bc84702b5c6bd7dc9cba966ae0167f45375f660e3407aee31f359750bcd" Oct 31 13:01:40.957070 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:40.957023 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j" Oct 31 13:01:40.976094 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:40.976075 5693 scope.go:122] "RemoveContainer" containerID="e5d07bc84702b5c6bd7dc9cba966ae0167f45375f660e3407aee31f359750bcd" Oct 31 13:01:40.976408 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:01:40.976383 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5d07bc84702b5c6bd7dc9cba966ae0167f45375f660e3407aee31f359750bcd\": container with ID starting with e5d07bc84702b5c6bd7dc9cba966ae0167f45375f660e3407aee31f359750bcd not found: ID does not exist" containerID="e5d07bc84702b5c6bd7dc9cba966ae0167f45375f660e3407aee31f359750bcd" Oct 31 13:01:40.976487 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:40.976420 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5d07bc84702b5c6bd7dc9cba966ae0167f45375f660e3407aee31f359750bcd"} err="failed to get container status \"e5d07bc84702b5c6bd7dc9cba966ae0167f45375f660e3407aee31f359750bcd\": rpc error: code = NotFound desc = could not find container \"e5d07bc84702b5c6bd7dc9cba966ae0167f45375f660e3407aee31f359750bcd\": container with ID starting with e5d07bc84702b5c6bd7dc9cba966ae0167f45375f660e3407aee31f359750bcd not found: ID does not exist" Oct 31 13:01:40.997236 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:40.997189 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-558f58b547-wwbxw"] Oct 31 13:01:41.002952 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:41.002923 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3c8c51-558f58b547-wwbxw"] Oct 31 13:01:41.033682 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:41.033538 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddfdedb8-f94c-4b8b-a717-42b9b8b3642c-kube-api-access-mhs7j\" (UniqueName: \"kubernetes.io/projected/ddfdedb8-f94c-4b8b-a717-42b9b8b3642c-kube-api-access-mhs7j\") pod \"ddfdedb8-f94c-4b8b-a717-42b9b8b3642c\" (UID: \"ddfdedb8-f94c-4b8b-a717-42b9b8b3642c\") " Oct 31 13:01:41.033879 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:41.033663 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddfdedb8-f94c-4b8b-a717-42b9b8b3642c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddfdedb8-f94c-4b8b-a717-42b9b8b3642c-openshift-service-ca\") pod \"ddfdedb8-f94c-4b8b-a717-42b9b8b3642c\" (UID: \"ddfdedb8-f94c-4b8b-a717-42b9b8b3642c\") " Oct 31 13:01:41.033879 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:41.033868 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddfdedb8-f94c-4b8b-a717-42b9b8b3642c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddfdedb8-f94c-4b8b-a717-42b9b8b3642c-metrics-certs\") pod \"ddfdedb8-f94c-4b8b-a717-42b9b8b3642c\" (UID: \"ddfdedb8-f94c-4b8b-a717-42b9b8b3642c\") " Oct 31 13:01:41.034144 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:41.034125 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddfdedb8-f94c-4b8b-a717-42b9b8b3642c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddfdedb8-f94c-4b8b-a717-42b9b8b3642c-maas-api-tls\") pod \"ddfdedb8-f94c-4b8b-a717-42b9b8b3642c\" (UID: \"ddfdedb8-f94c-4b8b-a717-42b9b8b3642c\") " Oct 31 13:01:41.034243 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:41.034201 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddfdedb8-f94c-4b8b-a717-42b9b8b3642c-openshift-service-ca" pod "ddfdedb8-f94c-4b8b-a717-42b9b8b3642c" (UID: "ddfdedb8-f94c-4b8b-a717-42b9b8b3642c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:01:41.034761 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:41.034729 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddfdedb8-f94c-4b8b-a717-42b9b8b3642c-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:41.037497 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:41.037337 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddfdedb8-f94c-4b8b-a717-42b9b8b3642c-maas-api-tls" pod "ddfdedb8-f94c-4b8b-a717-42b9b8b3642c" (UID: "ddfdedb8-f94c-4b8b-a717-42b9b8b3642c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:01:41.059075 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:41.059014 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddfdedb8-f94c-4b8b-a717-42b9b8b3642c-metrics-certs" pod "ddfdedb8-f94c-4b8b-a717-42b9b8b3642c" (UID: "ddfdedb8-f94c-4b8b-a717-42b9b8b3642c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:01:41.059823 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:41.059769 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddfdedb8-f94c-4b8b-a717-42b9b8b3642c-kube-api-access-mhs7j" pod "ddfdedb8-f94c-4b8b-a717-42b9b8b3642c" (UID: "ddfdedb8-f94c-4b8b-a717-42b9b8b3642c"). InnerVolumeSpecName "kube-api-access-mhs7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:01:41.136405 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:41.136369 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddfdedb8-f94c-4b8b-a717-42b9b8b3642c-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:41.136405 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:41.136406 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhs7j\" (UniqueName: \"kubernetes.io/projected/ddfdedb8-f94c-4b8b-a717-42b9b8b3642c-kube-api-access-mhs7j\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:41.136705 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:41.136424 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddfdedb8-f94c-4b8b-a717-42b9b8b3642c-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:41.409160 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:41.409129 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce656873-15e8-4959-aa03-23bcd234a3f2" path="/var/lib/kubelet/pods/ce656873-15e8-4959-aa03-23bcd234a3f2/volumes" Oct 31 13:01:41.961814 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:41.961787 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j" Oct 31 13:01:41.983313 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:41.983278 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j"] Oct 31 13:01:41.991098 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:41.991072 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j"] Oct 31 13:01:42.673917 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:42.673884 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl"] Oct 31 13:01:42.674814 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:42.674788 5693 cpu_manager.go:414] "containerMap: removing container" podUID="ddfdedb8-f94c-4b8b-a717-42b9b8b3642c" containerName="maas-api" Oct 31 13:01:42.674814 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:42.674814 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddfdedb8-f94c-4b8b-a717-42b9b8b3642c" containerName="maas-api" Oct 31 13:01:42.675036 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:42.674840 5693 cpu_manager.go:414] "containerMap: removing container" podUID="ce656873-15e8-4959-aa03-23bcd234a3f2" containerName="payload-pre-processing" Oct 31 13:01:42.675036 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:42.674850 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce656873-15e8-4959-aa03-23bcd234a3f2" containerName="payload-pre-processing" Oct 31 13:01:42.675155 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:42.675062 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddfdedb8-f94c-4b8b-a717-42b9b8b3642c" containerName="maas-api" Oct 31 13:01:42.675155 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:42.675080 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce656873-15e8-4959-aa03-23bcd234a3f2" containerName="payload-pre-processing" Oct 31 13:01:42.679925 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:42.679906 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" Oct 31 13:01:42.694546 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:42.694524 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-0db792-openshift-default-dockercfg-wgjcd\"" Oct 31 13:01:42.694992 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:42.694973 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl"] Oct 31 13:01:42.759085 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:42.759050 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85311012-2071-4e3d-ab91-f5f21c7e2ab5-istio-token\") pod \"e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl\" (UID: \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\") " pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" Oct 31 13:01:42.759296 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:42.759119 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85311012-2071-4e3d-ab91-f5f21c7e2ab5-workload-certs\") pod \"e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl\" (UID: \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\") " pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" Oct 31 13:01:42.759296 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:42.759167 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzx5b\" (UniqueName: \"kubernetes.io/projected/85311012-2071-4e3d-ab91-f5f21c7e2ab5-kube-api-access-kzx5b\") pod \"e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl\" (UID: \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\") " pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" Oct 31 13:01:42.759296 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:42.759276 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85311012-2071-4e3d-ab91-f5f21c7e2ab5-istio-envoy\") pod \"e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl\" (UID: \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\") " pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" Oct 31 13:01:42.759527 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:42.759329 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85311012-2071-4e3d-ab91-f5f21c7e2ab5-istio-data\") pod \"e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl\" (UID: \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\") " pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" Oct 31 13:01:42.759527 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:42.759396 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85311012-2071-4e3d-ab91-f5f21c7e2ab5-workload-socket\") pod \"e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl\" (UID: \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\") " pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" Oct 31 13:01:42.759527 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:42.759459 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85311012-2071-4e3d-ab91-f5f21c7e2ab5-istio-podinfo\") pod \"e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl\" (UID: \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\") " pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" Oct 31 13:01:42.759889 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:42.759537 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85311012-2071-4e3d-ab91-f5f21c7e2ab5-credential-socket\") pod \"e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl\" (UID: \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\") " pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" Oct 31 13:01:42.759889 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:42.759689 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85311012-2071-4e3d-ab91-f5f21c7e2ab5-istiod-ca-cert\") pod \"e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl\" (UID: \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\") " pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" Oct 31 13:01:42.861388 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:42.861356 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85311012-2071-4e3d-ab91-f5f21c7e2ab5-istio-envoy\") pod \"e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl\" (UID: \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\") " pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" Oct 31 13:01:42.861572 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:42.861411 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85311012-2071-4e3d-ab91-f5f21c7e2ab5-istio-data\") pod \"e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl\" (UID: \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\") " pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" Oct 31 13:01:42.861572 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:42.861462 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85311012-2071-4e3d-ab91-f5f21c7e2ab5-workload-socket\") pod \"e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl\" (UID: \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\") " pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" Oct 31 13:01:42.861572 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:42.861514 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85311012-2071-4e3d-ab91-f5f21c7e2ab5-istio-podinfo\") pod \"e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl\" (UID: \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\") " pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" Oct 31 13:01:42.861740 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:42.861585 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85311012-2071-4e3d-ab91-f5f21c7e2ab5-credential-socket\") pod \"e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl\" (UID: \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\") " pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" Oct 31 13:01:42.861740 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:42.861663 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85311012-2071-4e3d-ab91-f5f21c7e2ab5-istiod-ca-cert\") pod \"e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl\" (UID: \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\") " pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" Oct 31 13:01:42.861740 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:42.861697 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85311012-2071-4e3d-ab91-f5f21c7e2ab5-istio-token\") pod \"e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl\" (UID: \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\") " pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" Oct 31 13:01:42.861740 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:42.861735 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85311012-2071-4e3d-ab91-f5f21c7e2ab5-workload-certs\") pod \"e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl\" (UID: \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\") " pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" Oct 31 13:01:42.861939 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:42.861772 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzx5b\" (UniqueName: \"kubernetes.io/projected/85311012-2071-4e3d-ab91-f5f21c7e2ab5-kube-api-access-kzx5b\") pod \"e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl\" (UID: \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\") " pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" Oct 31 13:01:42.862511 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:42.862459 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85311012-2071-4e3d-ab91-f5f21c7e2ab5-istiod-ca-cert\") pod \"e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl\" (UID: \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\") " pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" Oct 31 13:01:42.862676 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:42.862622 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85311012-2071-4e3d-ab91-f5f21c7e2ab5-workload-socket\") pod \"e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl\" (UID: \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\") " pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" Oct 31 13:01:42.862773 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:42.862728 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85311012-2071-4e3d-ab91-f5f21c7e2ab5-credential-socket\") pod \"e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl\" (UID: \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\") " pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" Oct 31 13:01:42.862819 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:42.862800 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85311012-2071-4e3d-ab91-f5f21c7e2ab5-workload-certs\") pod \"e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl\" (UID: \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\") " pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" Oct 31 13:01:42.862973 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:42.862957 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85311012-2071-4e3d-ab91-f5f21c7e2ab5-istio-data\") pod \"e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl\" (UID: \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\") " pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" Oct 31 13:01:42.864049 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:42.864029 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85311012-2071-4e3d-ab91-f5f21c7e2ab5-istio-envoy\") pod \"e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl\" (UID: \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\") " pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" Oct 31 13:01:42.864564 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:42.864549 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85311012-2071-4e3d-ab91-f5f21c7e2ab5-istio-podinfo\") pod \"e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl\" (UID: \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\") " pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" Oct 31 13:01:42.875500 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:42.875474 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85311012-2071-4e3d-ab91-f5f21c7e2ab5-istio-token\") pod \"e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl\" (UID: \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\") " pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" Oct 31 13:01:42.879077 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:42.879055 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzx5b\" (UniqueName: \"kubernetes.io/projected/85311012-2071-4e3d-ab91-f5f21c7e2ab5-kube-api-access-kzx5b\") pod \"e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl\" (UID: \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\") " pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" Oct 31 13:01:42.998746 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:42.998217 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" Oct 31 13:01:43.412445 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:43.412406 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddfdedb8-f94c-4b8b-a717-42b9b8b3642c" path="/var/lib/kubelet/pods/ddfdedb8-f94c-4b8b-a717-42b9b8b3642c/volumes" Oct 31 13:01:43.418026 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:43.418000 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl"] Oct 31 13:01:43.419794 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:01:43.419759 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85311012_2071_4e3d_ab91_f5f21c7e2ab5.slice/crio-bd6caeb904172f717c026890698c7eb5ce5be0d63306d2e49f25ac94ca0714d8 WatchSource:0}: Error finding container bd6caeb904172f717c026890698c7eb5ce5be0d63306d2e49f25ac94ca0714d8: Status 404 returned error can't find the container with id bd6caeb904172f717c026890698c7eb5ce5be0d63306d2e49f25ac94ca0714d8 Oct 31 13:01:43.422478 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:43.422442 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:01:43.422568 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:43.422508 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:01:43.422568 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:43.422535 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:01:43.852934 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:43.852844 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56a83a-549fcf7864-9wx2j" podUID="ddfdedb8-f94c-4b8b-a717-42b9b8b3642c" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.100:8443/health\": context deadline exceeded" Oct 31 13:01:44.006161 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:44.006124 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" event={"ID":"85311012-2071-4e3d-ab91-f5f21c7e2ab5","Type":"ContainerStarted","Data":"ed445a1d4a7cdac6f139bdc73761a4aa41c6e5c80956082d2aa943866be2c0c6"} Oct 31 13:01:44.006551 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:44.006181 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" event={"ID":"85311012-2071-4e3d-ab91-f5f21c7e2ab5","Type":"ContainerStarted","Data":"bd6caeb904172f717c026890698c7eb5ce5be0d63306d2e49f25ac94ca0714d8"} Oct 31 13:01:44.032047 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:44.032004 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" podStartSLOduration=2.031989985 podStartE2EDuration="2.031989985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:01:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:01:44.03157268 +0000 UTC m=+1754.739158976" watchObservedRunningTime="2026-10-31 13:01:44.031989985 +0000 UTC m=+1754.739576280" Oct 31 13:01:44.998721 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:44.998685 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" Oct 31 13:01:45.004779 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:45.004755 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" Oct 31 13:01:45.013225 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:45.013199 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" Oct 31 13:01:45.014128 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:45.014109 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" Oct 31 13:01:45.791516 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:45.788766 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s"] Oct 31 13:01:45.791516 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:45.789008 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" podUID="b8b99947-a9a4-4b81-83ee-6cafebb48503" containerName="istio-proxy" containerID="cri-o://a11b518544fc0212d8560689d0f954d11e490bf75acb914a4da5a45aa978583f" gracePeriod=30 Oct 31 13:01:48.193546 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:48.192040 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0db792-854bc569bc-4tc49"] Oct 31 13:01:48.197461 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:48.197438 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db792-854bc569bc-4tc49" Oct 31 13:01:48.210793 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:48.210767 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-0db792\"" Oct 31 13:01:48.212069 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:48.212012 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-0db792-dockercfg-nbrw9\"" Oct 31 13:01:48.222509 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:48.222479 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0db792-854bc569bc-4tc49"] Oct 31 13:01:48.289143 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:48.289107 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afe96299-b227-472a-98a2-fafb8dcfca80-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0db792-854bc569bc-4tc49\" (UID: \"afe96299-b227-472a-98a2-fafb8dcfca80\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0db792-854bc569bc-4tc49" Oct 31 13:01:48.289317 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:48.289163 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk84z\" (UniqueName: \"kubernetes.io/projected/afe96299-b227-472a-98a2-fafb8dcfca80-kube-api-access-fk84z\") pod \"payload-processing-e2e-ipp-b-0db792-854bc569bc-4tc49\" (UID: \"afe96299-b227-472a-98a2-fafb8dcfca80\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0db792-854bc569bc-4tc49" Oct 31 13:01:48.404252 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:48.394676 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afe96299-b227-472a-98a2-fafb8dcfca80-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0db792-854bc569bc-4tc49\" (UID: \"afe96299-b227-472a-98a2-fafb8dcfca80\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0db792-854bc569bc-4tc49" Oct 31 13:01:48.404252 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:48.394760 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk84z\" (UniqueName: \"kubernetes.io/projected/afe96299-b227-472a-98a2-fafb8dcfca80-kube-api-access-fk84z\") pod \"payload-processing-e2e-ipp-b-0db792-854bc569bc-4tc49\" (UID: \"afe96299-b227-472a-98a2-fafb8dcfca80\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0db792-854bc569bc-4tc49" Oct 31 13:01:48.404252 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:48.396237 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afe96299-b227-472a-98a2-fafb8dcfca80-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0db792-854bc569bc-4tc49\" (UID: \"afe96299-b227-472a-98a2-fafb8dcfca80\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0db792-854bc569bc-4tc49" Oct 31 13:01:48.417676 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:48.416380 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk84z\" (UniqueName: \"kubernetes.io/projected/afe96299-b227-472a-98a2-fafb8dcfca80-kube-api-access-fk84z\") pod \"payload-processing-e2e-ipp-b-0db792-854bc569bc-4tc49\" (UID: \"afe96299-b227-472a-98a2-fafb8dcfca80\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0db792-854bc569bc-4tc49" Oct 31 13:01:48.527254 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:48.527151 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db792-854bc569bc-4tc49" Oct 31 13:01:48.712696 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:01:48.712633 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:01:48.860676 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:48.860625 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0db792-854bc569bc-4tc49"] Oct 31 13:01:48.864157 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:01:48.864125 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafe96299_b227_472a_98a2_fafb8dcfca80.slice/crio-3e072847888dc7a43cd067665e028cdfda2835cc637b9b785863525b35cd1b8a WatchSource:0}: Error finding container 3e072847888dc7a43cd067665e028cdfda2835cc637b9b785863525b35cd1b8a: Status 404 returned error can't find the container with id 3e072847888dc7a43cd067665e028cdfda2835cc637b9b785863525b35cd1b8a Oct 31 13:01:49.146680 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:49.144503 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db792-854bc569bc-4tc49" event={"ID":"afe96299-b227-472a-98a2-fafb8dcfca80","Type":"ContainerStarted","Data":"3e072847888dc7a43cd067665e028cdfda2835cc637b9b785863525b35cd1b8a"} Oct 31 13:01:50.152810 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:50.152773 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db792-854bc569bc-4tc49" event={"ID":"afe96299-b227-472a-98a2-fafb8dcfca80","Type":"ContainerStarted","Data":"fb81d6fc503e4ff8c2c5100d614cd8ebb11e49ed1d1ec2d46dd2430545d48fa5"} Oct 31 13:01:50.153254 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:50.152861 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db792-854bc569bc-4tc49" Oct 31 13:01:50.189542 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:50.189442 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db792-854bc569bc-4tc49" podStartSLOduration=2.189421145 podStartE2EDuration="2.189421145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:01:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:01:50.175302635 +0000 UTC m=+1760.882888932" watchObservedRunningTime="2026-10-31 13:01:50.189421145 +0000 UTC m=+1760.897007460" Oct 31 13:01:50.985437 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:50.985000 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-664c944bf9-lr9dl"] Oct 31 13:01:50.990240 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:50.990217 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-664c944bf9-lr9dl" Oct 31 13:01:51.001564 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.001532 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-664c944bf9-lr9dl"] Oct 31 13:01:51.001943 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.001863 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-48dcc1\"" Oct 31 13:01:51.003417 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.002019 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-48dcc1-dockercfg-kk89r\"" Oct 31 13:01:51.231711 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.226782 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24kpz\" (UniqueName: \"kubernetes.io/projected/c1565456-7221-4d1e-974a-a46efbfe4b91-kube-api-access-24kpz\") pod \"payload-pre-processing-e2e-ipp-b-48dcc1-664c944bf9-lr9dl\" (UID: \"c1565456-7221-4d1e-974a-a46efbfe4b91\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-664c944bf9-lr9dl" Oct 31 13:01:51.231711 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.227143 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1565456-7221-4d1e-974a-a46efbfe4b91-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-48dcc1-664c944bf9-lr9dl\" (UID: \"c1565456-7221-4d1e-974a-a46efbfe4b91\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-664c944bf9-lr9dl" Oct 31 13:01:51.249861 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.249605 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" event={"ID":"b8b99947-a9a4-4b81-83ee-6cafebb48503","Type":"ContainerDied","Data":"a11b518544fc0212d8560689d0f954d11e490bf75acb914a4da5a45aa978583f"} Oct 31 13:01:51.249861 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.249729 5693 scope.go:122] "RemoveContainer" containerID="206d0f7ab203dda912103ebe173a851f305b79c00aed7959dfa1da8b48f095a0" Oct 31 13:01:51.250789 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.250724 5693 generic.go:356] "Generic (PLEG): container finished" podID="b8b99947-a9a4-4b81-83ee-6cafebb48503" containerID="a11b518544fc0212d8560689d0f954d11e490bf75acb914a4da5a45aa978583f" exitCode=0 Oct 31 13:01:51.273209 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:01:51.273164 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"206d0f7ab203dda912103ebe173a851f305b79c00aed7959dfa1da8b48f095a0\": container with ID starting with 206d0f7ab203dda912103ebe173a851f305b79c00aed7959dfa1da8b48f095a0 not found: ID does not exist" containerID="206d0f7ab203dda912103ebe173a851f305b79c00aed7959dfa1da8b48f095a0" Oct 31 13:01:51.276806 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.276783 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" Oct 31 13:01:51.327928 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.327893 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b99947-a9a4-4b81-83ee-6cafebb48503-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b99947-a9a4-4b81-83ee-6cafebb48503-workload-socket\") pod \"b8b99947-a9a4-4b81-83ee-6cafebb48503\" (UID: \"b8b99947-a9a4-4b81-83ee-6cafebb48503\") " Oct 31 13:01:51.328100 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.327969 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b99947-a9a4-4b81-83ee-6cafebb48503-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8b99947-a9a4-4b81-83ee-6cafebb48503-istio-data\") pod \"b8b99947-a9a4-4b81-83ee-6cafebb48503\" (UID: \"b8b99947-a9a4-4b81-83ee-6cafebb48503\") " Oct 31 13:01:51.328100 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.328030 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b99947-a9a4-4b81-83ee-6cafebb48503-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b99947-a9a4-4b81-83ee-6cafebb48503-credential-socket\") pod \"b8b99947-a9a4-4b81-83ee-6cafebb48503\" (UID: \"b8b99947-a9a4-4b81-83ee-6cafebb48503\") " Oct 31 13:01:51.328100 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.328073 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b99947-a9a4-4b81-83ee-6cafebb48503-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8b99947-a9a4-4b81-83ee-6cafebb48503-workload-certs\") pod \"b8b99947-a9a4-4b81-83ee-6cafebb48503\" (UID: \"b8b99947-a9a4-4b81-83ee-6cafebb48503\") " Oct 31 13:01:51.328224 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.328110 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8b99947-a9a4-4b81-83ee-6cafebb48503-kube-api-access-krppg\" (UniqueName: \"kubernetes.io/projected/b8b99947-a9a4-4b81-83ee-6cafebb48503-kube-api-access-krppg\") pod \"b8b99947-a9a4-4b81-83ee-6cafebb48503\" (UID: \"b8b99947-a9a4-4b81-83ee-6cafebb48503\") " Oct 31 13:01:51.328224 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.328197 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8b99947-a9a4-4b81-83ee-6cafebb48503-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8b99947-a9a4-4b81-83ee-6cafebb48503-istiod-ca-cert\") pod \"b8b99947-a9a4-4b81-83ee-6cafebb48503\" (UID: \"b8b99947-a9a4-4b81-83ee-6cafebb48503\") " Oct 31 13:01:51.328319 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.328230 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b99947-a9a4-4b81-83ee-6cafebb48503-workload-socket" pod "b8b99947-a9a4-4b81-83ee-6cafebb48503" (UID: "b8b99947-a9a4-4b81-83ee-6cafebb48503"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:01:51.328408 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.328388 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b99947-a9a4-4b81-83ee-6cafebb48503-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8b99947-a9a4-4b81-83ee-6cafebb48503-istio-envoy\") pod \"b8b99947-a9a4-4b81-83ee-6cafebb48503\" (UID: \"b8b99947-a9a4-4b81-83ee-6cafebb48503\") " Oct 31 13:01:51.328517 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.328491 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8b99947-a9a4-4b81-83ee-6cafebb48503-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8b99947-a9a4-4b81-83ee-6cafebb48503-istio-podinfo\") pod \"b8b99947-a9a4-4b81-83ee-6cafebb48503\" (UID: \"b8b99947-a9a4-4b81-83ee-6cafebb48503\") " Oct 31 13:01:51.328593 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.328561 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8b99947-a9a4-4b81-83ee-6cafebb48503-istio-token\" (UniqueName: \"kubernetes.io/projected/b8b99947-a9a4-4b81-83ee-6cafebb48503-istio-token\") pod \"b8b99947-a9a4-4b81-83ee-6cafebb48503\" (UID: \"b8b99947-a9a4-4b81-83ee-6cafebb48503\") " Oct 31 13:01:51.328673 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.328591 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8b99947-a9a4-4b81-83ee-6cafebb48503-istiod-ca-cert" pod "b8b99947-a9a4-4b81-83ee-6cafebb48503" (UID: "b8b99947-a9a4-4b81-83ee-6cafebb48503"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:01:51.328865 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.328840 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24kpz\" (UniqueName: \"kubernetes.io/projected/c1565456-7221-4d1e-974a-a46efbfe4b91-kube-api-access-24kpz\") pod \"payload-pre-processing-e2e-ipp-b-48dcc1-664c944bf9-lr9dl\" (UID: \"c1565456-7221-4d1e-974a-a46efbfe4b91\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-664c944bf9-lr9dl" Oct 31 13:01:51.329262 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.329236 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1565456-7221-4d1e-974a-a46efbfe4b91-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-48dcc1-664c944bf9-lr9dl\" (UID: \"c1565456-7221-4d1e-974a-a46efbfe4b91\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-664c944bf9-lr9dl" Oct 31 13:01:51.329425 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.329406 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b99947-a9a4-4b81-83ee-6cafebb48503-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:51.329525 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.329432 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8b99947-a9a4-4b81-83ee-6cafebb48503-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:51.329741 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.328846 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b99947-a9a4-4b81-83ee-6cafebb48503-istio-data" pod "b8b99947-a9a4-4b81-83ee-6cafebb48503" (UID: "b8b99947-a9a4-4b81-83ee-6cafebb48503"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:01:51.329741 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.328991 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b99947-a9a4-4b81-83ee-6cafebb48503-credential-socket" pod "b8b99947-a9a4-4b81-83ee-6cafebb48503" (UID: "b8b99947-a9a4-4b81-83ee-6cafebb48503"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:01:51.329741 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.329160 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b99947-a9a4-4b81-83ee-6cafebb48503-workload-certs" pod "b8b99947-a9a4-4b81-83ee-6cafebb48503" (UID: "b8b99947-a9a4-4b81-83ee-6cafebb48503"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:01:51.331511 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.330319 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1565456-7221-4d1e-974a-a46efbfe4b91-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-48dcc1-664c944bf9-lr9dl\" (UID: \"c1565456-7221-4d1e-974a-a46efbfe4b91\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-664c944bf9-lr9dl" Oct 31 13:01:51.335981 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.335863 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b99947-a9a4-4b81-83ee-6cafebb48503-istio-envoy" pod "b8b99947-a9a4-4b81-83ee-6cafebb48503" (UID: "b8b99947-a9a4-4b81-83ee-6cafebb48503"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:01:51.336718 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.336690 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8b99947-a9a4-4b81-83ee-6cafebb48503-istio-podinfo" pod "b8b99947-a9a4-4b81-83ee-6cafebb48503" (UID: "b8b99947-a9a4-4b81-83ee-6cafebb48503"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:01:51.336718 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.336697 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b99947-a9a4-4b81-83ee-6cafebb48503-istio-token" pod "b8b99947-a9a4-4b81-83ee-6cafebb48503" (UID: "b8b99947-a9a4-4b81-83ee-6cafebb48503"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:01:51.338093 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.338071 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b99947-a9a4-4b81-83ee-6cafebb48503-kube-api-access-krppg" pod "b8b99947-a9a4-4b81-83ee-6cafebb48503" (UID: "b8b99947-a9a4-4b81-83ee-6cafebb48503"). InnerVolumeSpecName "kube-api-access-krppg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:01:51.351087 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.351039 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24kpz\" (UniqueName: \"kubernetes.io/projected/c1565456-7221-4d1e-974a-a46efbfe4b91-kube-api-access-24kpz\") pod \"payload-pre-processing-e2e-ipp-b-48dcc1-664c944bf9-lr9dl\" (UID: \"c1565456-7221-4d1e-974a-a46efbfe4b91\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-664c944bf9-lr9dl" Oct 31 13:01:51.430604 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.430570 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8b99947-a9a4-4b81-83ee-6cafebb48503-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:51.430604 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.430605 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8b99947-a9a4-4b81-83ee-6cafebb48503-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:51.430928 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.430623 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8b99947-a9a4-4b81-83ee-6cafebb48503-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:51.430928 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.430677 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8b99947-a9a4-4b81-83ee-6cafebb48503-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:51.430928 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.430693 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b99947-a9a4-4b81-83ee-6cafebb48503-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:51.430928 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.430797 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8b99947-a9a4-4b81-83ee-6cafebb48503-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:51.430928 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.430812 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krppg\" (UniqueName: \"kubernetes.io/projected/b8b99947-a9a4-4b81-83ee-6cafebb48503-kube-api-access-krppg\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:51.620572 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.619935 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-664c944bf9-lr9dl" Oct 31 13:01:51.913344 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:51.913315 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-664c944bf9-lr9dl"] Oct 31 13:01:51.914021 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:01:51.913991 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1565456_7221_4d1e_974a_a46efbfe4b91.slice/crio-86a8b35f8064a18e19b1afaa94331ed681313b7c24e0f59b1f384d75aa1b6a08 WatchSource:0}: Error finding container 86a8b35f8064a18e19b1afaa94331ed681313b7c24e0f59b1f384d75aa1b6a08: Status 404 returned error can't find the container with id 86a8b35f8064a18e19b1afaa94331ed681313b7c24e0f59b1f384d75aa1b6a08 Oct 31 13:01:52.258975 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:52.258875 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-664c944bf9-lr9dl" event={"ID":"c1565456-7221-4d1e-974a-a46efbfe4b91","Type":"ContainerStarted","Data":"4227905d1d7dd92c5e5618fcbddfa234d1df195e8bb7579720bddcef5b038ed2"} Oct 31 13:01:52.258975 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:52.258936 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-664c944bf9-lr9dl" event={"ID":"c1565456-7221-4d1e-974a-a46efbfe4b91","Type":"ContainerStarted","Data":"86a8b35f8064a18e19b1afaa94331ed681313b7c24e0f59b1f384d75aa1b6a08"} Oct 31 13:01:52.259479 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:52.259073 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-664c944bf9-lr9dl" Oct 31 13:01:52.260426 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:52.260407 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" Oct 31 13:01:52.260529 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:52.260452 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s" event={"ID":"b8b99947-a9a4-4b81-83ee-6cafebb48503","Type":"ContainerDied","Data":"7df62ab0e032848bf110bfa969fcffd762923d4eb95fc3654345afa92226afd5"} Oct 31 13:01:52.260529 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:52.260498 5693 scope.go:122] "RemoveContainer" containerID="a11b518544fc0212d8560689d0f954d11e490bf75acb914a4da5a45aa978583f" Oct 31 13:01:52.298789 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:52.298744 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-664c944bf9-lr9dl" podStartSLOduration=2.298730688 podStartE2EDuration="2.298730688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:01:52.282828999 +0000 UTC m=+1762.990415293" watchObservedRunningTime="2026-10-31 13:01:52.298730688 +0000 UTC m=+1763.006316984" Oct 31 13:01:52.301117 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:52.301097 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s"] Oct 31 13:01:52.311207 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:52.311169 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-56a83a-openshift-default-598b9f7c9-tkk8s"] Oct 31 13:01:53.409077 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:53.409045 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8b99947-a9a4-4b81-83ee-6cafebb48503" path="/var/lib/kubelet/pods/b8b99947-a9a4-4b81-83ee-6cafebb48503/volumes" Oct 31 13:01:54.059603 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:54.059555 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-fc4458d75-d86fk"] Oct 31 13:01:54.060567 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:54.060541 5693 cpu_manager.go:414] "containerMap: removing container" podUID="b8b99947-a9a4-4b81-83ee-6cafebb48503" containerName="istio-proxy" Oct 31 13:01:54.060567 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:54.060562 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8b99947-a9a4-4b81-83ee-6cafebb48503" containerName="istio-proxy" Oct 31 13:01:54.060781 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:54.060669 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8b99947-a9a4-4b81-83ee-6cafebb48503" containerName="istio-proxy" Oct 31 13:01:54.060781 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:54.060678 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8b99947-a9a4-4b81-83ee-6cafebb48503" containerName="istio-proxy" Oct 31 13:01:54.065498 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:54.065474 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-fc4458d75-d86fk" Oct 31 13:01:54.071748 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:54.071716 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4989f9\"" Oct 31 13:01:54.076627 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:54.076533 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-fc4458d75-d86fk"] Oct 31 13:01:54.077697 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:54.077147 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4989f9-dockercfg-gg756\"" Oct 31 13:01:54.218704 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:54.210092 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf5j5\" (UniqueName: \"kubernetes.io/projected/e00d7b50-2ee9-4019-9ccf-7d26cd06c118-kube-api-access-kf5j5\") pod \"payload-pre-processing-e2e-ipp-a-4989f9-fc4458d75-d86fk\" (UID: \"e00d7b50-2ee9-4019-9ccf-7d26cd06c118\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-fc4458d75-d86fk" Oct 31 13:01:54.218704 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:54.210229 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e00d7b50-2ee9-4019-9ccf-7d26cd06c118-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4989f9-fc4458d75-d86fk\" (UID: \"e00d7b50-2ee9-4019-9ccf-7d26cd06c118\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-fc4458d75-d86fk" Oct 31 13:01:54.313843 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:54.313749 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e00d7b50-2ee9-4019-9ccf-7d26cd06c118-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4989f9-fc4458d75-d86fk\" (UID: \"e00d7b50-2ee9-4019-9ccf-7d26cd06c118\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-fc4458d75-d86fk" Oct 31 13:01:54.314353 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:54.314327 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf5j5\" (UniqueName: \"kubernetes.io/projected/e00d7b50-2ee9-4019-9ccf-7d26cd06c118-kube-api-access-kf5j5\") pod \"payload-pre-processing-e2e-ipp-a-4989f9-fc4458d75-d86fk\" (UID: \"e00d7b50-2ee9-4019-9ccf-7d26cd06c118\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-fc4458d75-d86fk" Oct 31 13:01:54.315885 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:54.315858 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e00d7b50-2ee9-4019-9ccf-7d26cd06c118-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4989f9-fc4458d75-d86fk\" (UID: \"e00d7b50-2ee9-4019-9ccf-7d26cd06c118\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-fc4458d75-d86fk" Oct 31 13:01:54.358901 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:54.358740 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf5j5\" (UniqueName: \"kubernetes.io/projected/e00d7b50-2ee9-4019-9ccf-7d26cd06c118-kube-api-access-kf5j5\") pod \"payload-pre-processing-e2e-ipp-a-4989f9-fc4458d75-d86fk\" (UID: \"e00d7b50-2ee9-4019-9ccf-7d26cd06c118\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-fc4458d75-d86fk" Oct 31 13:01:54.387432 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:54.385232 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-fc4458d75-d86fk" Oct 31 13:01:54.677913 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:54.677885 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-fc4458d75-d86fk"] Oct 31 13:01:54.678994 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:01:54.678950 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode00d7b50_2ee9_4019_9ccf_7d26cd06c118.slice/crio-e4c6151b8bf822a58e5e46602092e2820ce47d1a21b6ba997703e55435160280 WatchSource:0}: Error finding container e4c6151b8bf822a58e5e46602092e2820ce47d1a21b6ba997703e55435160280: Status 404 returned error can't find the container with id e4c6151b8bf822a58e5e46602092e2820ce47d1a21b6ba997703e55435160280 Oct 31 13:01:55.382800 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:55.382760 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-fc4458d75-d86fk" event={"ID":"e00d7b50-2ee9-4019-9ccf-7d26cd06c118","Type":"ContainerStarted","Data":"42ce795b58089dabf4b1a5b6cd54c73eeb593ee1b98d81a10559bc1e1f09625e"} Oct 31 13:01:55.383179 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:55.383156 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-fc4458d75-d86fk" event={"ID":"e00d7b50-2ee9-4019-9ccf-7d26cd06c118","Type":"ContainerStarted","Data":"e4c6151b8bf822a58e5e46602092e2820ce47d1a21b6ba997703e55435160280"} Oct 31 13:01:55.383590 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:55.383554 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-fc4458d75-d86fk" Oct 31 13:01:55.405552 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:01:55.405503 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-fc4458d75-d86fk" podStartSLOduration=1.405488797 podStartE2EDuration="1.405488797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:01:55.402462768 +0000 UTC m=+1766.110049064" watchObservedRunningTime="2026-10-31 13:01:55.405488797 +0000 UTC m=+1766.113075092" Oct 31 13:02:00.449334 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:00.449297 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf"] Oct 31 13:02:00.450021 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:00.450004 5693 cpu_manager.go:414] "containerMap: removing container" podUID="b8b99947-a9a4-4b81-83ee-6cafebb48503" containerName="istio-proxy" Oct 31 13:02:00.450021 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:00.450022 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8b99947-a9a4-4b81-83ee-6cafebb48503" containerName="istio-proxy" Oct 31 13:02:00.454071 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:00.454044 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf" Oct 31 13:02:00.458010 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:00.457984 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-0db792-serving-cert\"" Oct 31 13:02:00.467548 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:00.467522 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf"] Oct 31 13:02:00.626258 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:00.626225 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dc63886-9566-45e8-9861-383040f4c4fe-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf\" (UID: \"2dc63886-9566-45e8-9861-383040f4c4fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf" Oct 31 13:02:00.626258 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:00.626261 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dc63886-9566-45e8-9861-383040f4c4fe-metrics-certs\") pod \"maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf\" (UID: \"2dc63886-9566-45e8-9861-383040f4c4fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf" Oct 31 13:02:00.626468 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:00.626281 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dc63886-9566-45e8-9861-383040f4c4fe-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf\" (UID: \"2dc63886-9566-45e8-9861-383040f4c4fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf" Oct 31 13:02:00.626503 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:00.626480 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p47qv\" (UniqueName: \"kubernetes.io/projected/2dc63886-9566-45e8-9861-383040f4c4fe-kube-api-access-p47qv\") pod \"maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf\" (UID: \"2dc63886-9566-45e8-9861-383040f4c4fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf" Oct 31 13:02:00.729390 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:00.729292 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dc63886-9566-45e8-9861-383040f4c4fe-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf\" (UID: \"2dc63886-9566-45e8-9861-383040f4c4fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf" Oct 31 13:02:00.729390 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:00.729350 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dc63886-9566-45e8-9861-383040f4c4fe-metrics-certs\") pod \"maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf\" (UID: \"2dc63886-9566-45e8-9861-383040f4c4fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf" Oct 31 13:02:00.729760 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:00.729731 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dc63886-9566-45e8-9861-383040f4c4fe-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf\" (UID: \"2dc63886-9566-45e8-9861-383040f4c4fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf" Oct 31 13:02:00.729960 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:00.729932 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p47qv\" (UniqueName: \"kubernetes.io/projected/2dc63886-9566-45e8-9861-383040f4c4fe-kube-api-access-p47qv\") pod \"maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf\" (UID: \"2dc63886-9566-45e8-9861-383040f4c4fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf" Oct 31 13:02:00.730803 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:00.730779 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dc63886-9566-45e8-9861-383040f4c4fe-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf\" (UID: \"2dc63886-9566-45e8-9861-383040f4c4fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf" Oct 31 13:02:00.732309 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:00.732280 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dc63886-9566-45e8-9861-383040f4c4fe-metrics-certs\") pod \"maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf\" (UID: \"2dc63886-9566-45e8-9861-383040f4c4fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf" Oct 31 13:02:00.732418 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:00.732376 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dc63886-9566-45e8-9861-383040f4c4fe-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf\" (UID: \"2dc63886-9566-45e8-9861-383040f4c4fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf" Oct 31 13:02:00.746791 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:00.746753 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p47qv\" (UniqueName: \"kubernetes.io/projected/2dc63886-9566-45e8-9861-383040f4c4fe-kube-api-access-p47qv\") pod \"maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf\" (UID: \"2dc63886-9566-45e8-9861-383040f4c4fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf" Oct 31 13:02:00.778584 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:00.778555 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf" Oct 31 13:02:01.146430 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:01.146388 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf"] Oct 31 13:02:01.254721 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:01.254695 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db792-854bc569bc-4tc49" Oct 31 13:02:01.423745 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:01.423708 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf" event={"ID":"2dc63886-9566-45e8-9861-383040f4c4fe","Type":"ContainerStarted","Data":"0187e66b59ea82334c059f861d328a6f357c3903726a39074b490a376892bdf6"} Oct 31 13:02:02.433201 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:02.433165 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf" event={"ID":"2dc63886-9566-45e8-9861-383040f4c4fe","Type":"ContainerStarted","Data":"c8336bab7c8e597efe2444a10e4392ca2aeaee1ba998fed336171c8bb97dacc0"} Oct 31 13:02:02.433733 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:02.433381 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf" Oct 31 13:02:02.455612 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:02.455565 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf" podStartSLOduration=1.8079872240000001 podStartE2EDuration="2.455551298s" totalImagesPullingTime="647.564074ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:02:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:02.453165937 +0000 UTC m=+1773.160752259" watchObservedRunningTime="2026-10-31 13:02:02.455551298 +0000 UTC m=+1773.163137592" Oct 31 13:02:03.270698 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:03.270635 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-664c944bf9-lr9dl" Oct 31 13:02:05.464914 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:05.464878 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5"] Oct 31 13:02:05.471087 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:05.471064 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5" Oct 31 13:02:05.477618 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:05.477579 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-4989f9-serving-cert\"" Oct 31 13:02:05.490163 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:05.490137 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5"] Oct 31 13:02:05.639694 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:05.639630 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/416c552c-b82d-472a-8f81-ac404b2c8a31-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5\" (UID: \"416c552c-b82d-472a-8f81-ac404b2c8a31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5" Oct 31 13:02:05.639871 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:05.639741 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/416c552c-b82d-472a-8f81-ac404b2c8a31-metrics-certs\") pod \"maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5\" (UID: \"416c552c-b82d-472a-8f81-ac404b2c8a31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5" Oct 31 13:02:05.639871 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:05.639833 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26hhx\" (UniqueName: \"kubernetes.io/projected/416c552c-b82d-472a-8f81-ac404b2c8a31-kube-api-access-26hhx\") pod \"maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5\" (UID: \"416c552c-b82d-472a-8f81-ac404b2c8a31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5" Oct 31 13:02:05.639871 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:05.639861 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/416c552c-b82d-472a-8f81-ac404b2c8a31-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5\" (UID: \"416c552c-b82d-472a-8f81-ac404b2c8a31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5" Oct 31 13:02:05.749709 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:05.742715 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/416c552c-b82d-472a-8f81-ac404b2c8a31-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5\" (UID: \"416c552c-b82d-472a-8f81-ac404b2c8a31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5" Oct 31 13:02:05.749709 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:05.742806 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/416c552c-b82d-472a-8f81-ac404b2c8a31-metrics-certs\") pod \"maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5\" (UID: \"416c552c-b82d-472a-8f81-ac404b2c8a31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5" Oct 31 13:02:05.749709 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:05.742917 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26hhx\" (UniqueName: \"kubernetes.io/projected/416c552c-b82d-472a-8f81-ac404b2c8a31-kube-api-access-26hhx\") pod \"maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5\" (UID: \"416c552c-b82d-472a-8f81-ac404b2c8a31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5" Oct 31 13:02:05.749709 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:05.742984 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/416c552c-b82d-472a-8f81-ac404b2c8a31-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5\" (UID: \"416c552c-b82d-472a-8f81-ac404b2c8a31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5" Oct 31 13:02:05.749709 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:05.745240 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/416c552c-b82d-472a-8f81-ac404b2c8a31-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5\" (UID: \"416c552c-b82d-472a-8f81-ac404b2c8a31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5" Oct 31 13:02:05.753341 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:05.753309 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/416c552c-b82d-472a-8f81-ac404b2c8a31-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5\" (UID: \"416c552c-b82d-472a-8f81-ac404b2c8a31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5" Oct 31 13:02:05.760027 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:05.759995 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/416c552c-b82d-472a-8f81-ac404b2c8a31-metrics-certs\") pod \"maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5\" (UID: \"416c552c-b82d-472a-8f81-ac404b2c8a31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5" Oct 31 13:02:05.777661 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:05.775761 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26hhx\" (UniqueName: \"kubernetes.io/projected/416c552c-b82d-472a-8f81-ac404b2c8a31-kube-api-access-26hhx\") pod \"maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5\" (UID: \"416c552c-b82d-472a-8f81-ac404b2c8a31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5" Oct 31 13:02:05.791010 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:05.790982 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5" Oct 31 13:02:06.167725 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:06.167584 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5"] Oct 31 13:02:06.174467 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:02:06.174406 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod416c552c_b82d_472a_8f81_ac404b2c8a31.slice/crio-bbbcaa15efda2e4956a5483c49c4b4e6f19dbae419f9d07714d6b5eb2227c969 WatchSource:0}: Error finding container bbbcaa15efda2e4956a5483c49c4b4e6f19dbae419f9d07714d6b5eb2227c969: Status 404 returned error can't find the container with id bbbcaa15efda2e4956a5483c49c4b4e6f19dbae419f9d07714d6b5eb2227c969 Oct 31 13:02:06.398568 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:06.398539 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-fc4458d75-d86fk" Oct 31 13:02:06.464106 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:06.463995 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5" event={"ID":"416c552c-b82d-472a-8f81-ac404b2c8a31","Type":"ContainerStarted","Data":"bbbcaa15efda2e4956a5483c49c4b4e6f19dbae419f9d07714d6b5eb2227c969"} Oct 31 13:02:07.471937 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:07.471893 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5" event={"ID":"416c552c-b82d-472a-8f81-ac404b2c8a31","Type":"ContainerStarted","Data":"f6f456283f425d33aed2c8186a59668730838c2ddd9af1ca170505245de5cf3c"} Oct 31 13:02:07.472367 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:07.471963 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5" Oct 31 13:02:07.494918 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:07.494872 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5" podStartSLOduration=2.075328889 podStartE2EDuration="2.494858495s" totalImagesPullingTime="419.529606ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:02:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:07.493102004 +0000 UTC m=+1778.200688299" watchObservedRunningTime="2026-10-31 13:02:07.494858495 +0000 UTC m=+1778.202444789" Oct 31 13:02:07.844680 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:07.844577 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs"] Oct 31 13:02:07.844873 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:07.844842 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs" podUID="659d976b-0eec-480e-ab29-c95814ab7e11" containerName="maas-api" containerID="cri-o://27e5dcbbe36f564fa3697d686d279fdbff9e0efeb24866354dcbfd5eaa1d28b7" gracePeriod=30 Oct 31 13:02:08.376428 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:08.376398 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs" Oct 31 13:02:08.446267 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:08.446240 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf" Oct 31 13:02:08.447266 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:08.447245 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/659d976b-0eec-480e-ab29-c95814ab7e11-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/659d976b-0eec-480e-ab29-c95814ab7e11-openshift-service-ca\") pod \"659d976b-0eec-480e-ab29-c95814ab7e11\" (UID: \"659d976b-0eec-480e-ab29-c95814ab7e11\") " Oct 31 13:02:08.447400 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:08.447283 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/659d976b-0eec-480e-ab29-c95814ab7e11-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/659d976b-0eec-480e-ab29-c95814ab7e11-maas-api-tls\") pod \"659d976b-0eec-480e-ab29-c95814ab7e11\" (UID: \"659d976b-0eec-480e-ab29-c95814ab7e11\") " Oct 31 13:02:08.447400 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:08.447319 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/659d976b-0eec-480e-ab29-c95814ab7e11-metrics-certs\" (UniqueName: \"kubernetes.io/secret/659d976b-0eec-480e-ab29-c95814ab7e11-metrics-certs\") pod \"659d976b-0eec-480e-ab29-c95814ab7e11\" (UID: \"659d976b-0eec-480e-ab29-c95814ab7e11\") " Oct 31 13:02:08.447400 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:08.447367 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/659d976b-0eec-480e-ab29-c95814ab7e11-kube-api-access-pgbst\" (UniqueName: \"kubernetes.io/projected/659d976b-0eec-480e-ab29-c95814ab7e11-kube-api-access-pgbst\") pod \"659d976b-0eec-480e-ab29-c95814ab7e11\" (UID: \"659d976b-0eec-480e-ab29-c95814ab7e11\") " Oct 31 13:02:08.448051 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:08.448026 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/659d976b-0eec-480e-ab29-c95814ab7e11-openshift-service-ca" pod "659d976b-0eec-480e-ab29-c95814ab7e11" (UID: "659d976b-0eec-480e-ab29-c95814ab7e11"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:02:08.450034 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:08.450007 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/659d976b-0eec-480e-ab29-c95814ab7e11-metrics-certs" pod "659d976b-0eec-480e-ab29-c95814ab7e11" (UID: "659d976b-0eec-480e-ab29-c95814ab7e11"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:02:08.450195 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:08.450170 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/659d976b-0eec-480e-ab29-c95814ab7e11-maas-api-tls" pod "659d976b-0eec-480e-ab29-c95814ab7e11" (UID: "659d976b-0eec-480e-ab29-c95814ab7e11"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:02:08.450274 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:08.450173 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/659d976b-0eec-480e-ab29-c95814ab7e11-kube-api-access-pgbst" pod "659d976b-0eec-480e-ab29-c95814ab7e11" (UID: "659d976b-0eec-480e-ab29-c95814ab7e11"). InnerVolumeSpecName "kube-api-access-pgbst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:02:08.481162 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:08.481136 5693 generic.go:356] "Generic (PLEG): container finished" podID="659d976b-0eec-480e-ab29-c95814ab7e11" containerID="27e5dcbbe36f564fa3697d686d279fdbff9e0efeb24866354dcbfd5eaa1d28b7" exitCode=0 Oct 31 13:02:08.481585 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:08.481183 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs" Oct 31 13:02:08.481585 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:08.481221 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs" event={"ID":"659d976b-0eec-480e-ab29-c95814ab7e11","Type":"ContainerDied","Data":"27e5dcbbe36f564fa3697d686d279fdbff9e0efeb24866354dcbfd5eaa1d28b7"} Oct 31 13:02:08.481585 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:08.481272 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs" event={"ID":"659d976b-0eec-480e-ab29-c95814ab7e11","Type":"ContainerDied","Data":"6e319d78905c1265b58a07bbc6c058df4ea05e1ffbd2d27e948965ef3b5abf02"} Oct 31 13:02:08.481585 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:08.481293 5693 scope.go:122] "RemoveContainer" containerID="27e5dcbbe36f564fa3697d686d279fdbff9e0efeb24866354dcbfd5eaa1d28b7" Oct 31 13:02:08.492828 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:08.492808 5693 scope.go:122] "RemoveContainer" containerID="27e5dcbbe36f564fa3697d686d279fdbff9e0efeb24866354dcbfd5eaa1d28b7" Oct 31 13:02:08.493109 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:02:08.493091 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27e5dcbbe36f564fa3697d686d279fdbff9e0efeb24866354dcbfd5eaa1d28b7\": container with ID starting with 27e5dcbbe36f564fa3697d686d279fdbff9e0efeb24866354dcbfd5eaa1d28b7 not found: ID does not exist" containerID="27e5dcbbe36f564fa3697d686d279fdbff9e0efeb24866354dcbfd5eaa1d28b7" Oct 31 13:02:08.493190 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:08.493115 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27e5dcbbe36f564fa3697d686d279fdbff9e0efeb24866354dcbfd5eaa1d28b7"} err="failed to get container status \"27e5dcbbe36f564fa3697d686d279fdbff9e0efeb24866354dcbfd5eaa1d28b7\": rpc error: code = NotFound desc = could not find container \"27e5dcbbe36f564fa3697d686d279fdbff9e0efeb24866354dcbfd5eaa1d28b7\": container with ID starting with 27e5dcbbe36f564fa3697d686d279fdbff9e0efeb24866354dcbfd5eaa1d28b7 not found: ID does not exist" Oct 31 13:02:08.526140 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:08.526110 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs"] Oct 31 13:02:08.532813 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:08.532785 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dc2db4-67df8f656d-2d7rs"] Oct 31 13:02:08.549599 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:08.549572 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/659d976b-0eec-480e-ab29-c95814ab7e11-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:08.549820 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:08.549791 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/659d976b-0eec-480e-ab29-c95814ab7e11-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:08.549919 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:08.549823 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/659d976b-0eec-480e-ab29-c95814ab7e11-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:08.549919 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:08.549838 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgbst\" (UniqueName: \"kubernetes.io/projected/659d976b-0eec-480e-ab29-c95814ab7e11-kube-api-access-pgbst\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:09.408662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:09.408606 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="659d976b-0eec-480e-ab29-c95814ab7e11" path="/var/lib/kubelet/pods/659d976b-0eec-480e-ab29-c95814ab7e11/volumes" Oct 31 13:02:13.488076 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:13.488046 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5" Oct 31 13:02:16.843825 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:16.843773 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0db792-854bc569bc-4tc49"] Oct 31 13:02:18.811742 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:02:18.811301 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:02:21.273130 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:21.273096 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-64f86ffbff-5fcnb"] Oct 31 13:02:21.273781 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:21.273320 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-64f86ffbff-5fcnb" podUID="820722a4-2f7a-45b0-911a-fc744c660284" containerName="payload-pre-processing" containerID="cri-o://acdfe5595168d409baca1b8e25529285338071072f041698521c460c928e8f01" gracePeriod=30 Oct 31 13:02:21.625417 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:21.625325 5693 generic.go:356] "Generic (PLEG): container finished" podID="820722a4-2f7a-45b0-911a-fc744c660284" containerID="acdfe5595168d409baca1b8e25529285338071072f041698521c460c928e8f01" exitCode=0 Oct 31 13:02:21.625776 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:21.625751 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-64f86ffbff-5fcnb" event={"ID":"820722a4-2f7a-45b0-911a-fc744c660284","Type":"ContainerDied","Data":"acdfe5595168d409baca1b8e25529285338071072f041698521c460c928e8f01"} Oct 31 13:02:21.806156 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:21.805255 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-64f86ffbff-5fcnb" Oct 31 13:02:21.840101 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:21.840070 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/820722a4-2f7a-45b0-911a-fc744c660284-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/820722a4-2f7a-45b0-911a-fc744c660284-plugins-config-volume\") pod \"820722a4-2f7a-45b0-911a-fc744c660284\" (UID: \"820722a4-2f7a-45b0-911a-fc744c660284\") " Oct 31 13:02:21.840522 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:21.840496 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/820722a4-2f7a-45b0-911a-fc744c660284-plugins-config-volume" pod "820722a4-2f7a-45b0-911a-fc744c660284" (UID: "820722a4-2f7a-45b0-911a-fc744c660284"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:02:21.844130 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:21.844096 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db792-854bc569bc-4tc49" podUID="afe96299-b227-472a-98a2-fafb8dcfca80" containerName="payload-processing" containerID="cri-o://fb81d6fc503e4ff8c2c5100d614cd8ebb11e49ed1d1ec2d46dd2430545d48fa5" gracePeriod=25 Oct 31 13:02:21.943362 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:21.942536 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/820722a4-2f7a-45b0-911a-fc744c660284-kube-api-access-f6z9p\" (UniqueName: \"kubernetes.io/projected/820722a4-2f7a-45b0-911a-fc744c660284-kube-api-access-f6z9p\") pod \"820722a4-2f7a-45b0-911a-fc744c660284\" (UID: \"820722a4-2f7a-45b0-911a-fc744c660284\") " Oct 31 13:02:21.943362 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:21.943066 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/820722a4-2f7a-45b0-911a-fc744c660284-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:21.946735 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:21.946699 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/820722a4-2f7a-45b0-911a-fc744c660284-kube-api-access-f6z9p" pod "820722a4-2f7a-45b0-911a-fc744c660284" (UID: "820722a4-2f7a-45b0-911a-fc744c660284"). InnerVolumeSpecName "kube-api-access-f6z9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:02:22.045278 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:22.045231 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6z9p\" (UniqueName: \"kubernetes.io/projected/820722a4-2f7a-45b0-911a-fc744c660284-kube-api-access-f6z9p\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:22.116448 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:22.116424 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db792-854bc569bc-4tc49" Oct 31 13:02:22.146362 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:22.146335 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afe96299-b227-472a-98a2-fafb8dcfca80-kube-api-access-fk84z\" (UniqueName: \"kubernetes.io/projected/afe96299-b227-472a-98a2-fafb8dcfca80-kube-api-access-fk84z\") pod \"afe96299-b227-472a-98a2-fafb8dcfca80\" (UID: \"afe96299-b227-472a-98a2-fafb8dcfca80\") " Oct 31 13:02:22.146533 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:22.146418 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afe96299-b227-472a-98a2-fafb8dcfca80-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afe96299-b227-472a-98a2-fafb8dcfca80-plugins-config-volume\") pod \"afe96299-b227-472a-98a2-fafb8dcfca80\" (UID: \"afe96299-b227-472a-98a2-fafb8dcfca80\") " Oct 31 13:02:22.146870 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:22.146845 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afe96299-b227-472a-98a2-fafb8dcfca80-plugins-config-volume" pod "afe96299-b227-472a-98a2-fafb8dcfca80" (UID: "afe96299-b227-472a-98a2-fafb8dcfca80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:02:22.147216 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:22.147196 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afe96299-b227-472a-98a2-fafb8dcfca80-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:22.149156 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:22.149127 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afe96299-b227-472a-98a2-fafb8dcfca80-kube-api-access-fk84z" pod "afe96299-b227-472a-98a2-fafb8dcfca80" (UID: "afe96299-b227-472a-98a2-fafb8dcfca80"). InnerVolumeSpecName "kube-api-access-fk84z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:02:22.248760 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:22.248724 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk84z\" (UniqueName: \"kubernetes.io/projected/afe96299-b227-472a-98a2-fafb8dcfca80-kube-api-access-fk84z\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:22.633847 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:22.633757 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-64f86ffbff-5fcnb" Oct 31 13:02:22.633847 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:22.633756 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-64f86ffbff-5fcnb" event={"ID":"820722a4-2f7a-45b0-911a-fc744c660284","Type":"ContainerDied","Data":"f75de74786817626a858a0f65ee7e9fba114ab551be41c5114e11c664087e16a"} Oct 31 13:02:22.634384 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:22.633904 5693 scope.go:122] "RemoveContainer" containerID="acdfe5595168d409baca1b8e25529285338071072f041698521c460c928e8f01" Oct 31 13:02:22.635471 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:22.635317 5693 generic.go:356] "Generic (PLEG): container finished" podID="afe96299-b227-472a-98a2-fafb8dcfca80" containerID="fb81d6fc503e4ff8c2c5100d614cd8ebb11e49ed1d1ec2d46dd2430545d48fa5" exitCode=0 Oct 31 13:02:22.635471 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:22.635366 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db792-854bc569bc-4tc49" Oct 31 13:02:22.635471 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:22.635417 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db792-854bc569bc-4tc49" event={"ID":"afe96299-b227-472a-98a2-fafb8dcfca80","Type":"ContainerDied","Data":"fb81d6fc503e4ff8c2c5100d614cd8ebb11e49ed1d1ec2d46dd2430545d48fa5"} Oct 31 13:02:22.635471 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:22.635466 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db792-854bc569bc-4tc49" event={"ID":"afe96299-b227-472a-98a2-fafb8dcfca80","Type":"ContainerDied","Data":"3e072847888dc7a43cd067665e028cdfda2835cc637b9b785863525b35cd1b8a"} Oct 31 13:02:22.647201 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:22.647182 5693 scope.go:122] "RemoveContainer" containerID="fb81d6fc503e4ff8c2c5100d614cd8ebb11e49ed1d1ec2d46dd2430545d48fa5" Oct 31 13:02:22.657166 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:22.657149 5693 scope.go:122] "RemoveContainer" containerID="fb81d6fc503e4ff8c2c5100d614cd8ebb11e49ed1d1ec2d46dd2430545d48fa5" Oct 31 13:02:22.657430 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:02:22.657404 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb81d6fc503e4ff8c2c5100d614cd8ebb11e49ed1d1ec2d46dd2430545d48fa5\": container with ID starting with fb81d6fc503e4ff8c2c5100d614cd8ebb11e49ed1d1ec2d46dd2430545d48fa5 not found: ID does not exist" containerID="fb81d6fc503e4ff8c2c5100d614cd8ebb11e49ed1d1ec2d46dd2430545d48fa5" Oct 31 13:02:22.657494 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:22.657443 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb81d6fc503e4ff8c2c5100d614cd8ebb11e49ed1d1ec2d46dd2430545d48fa5"} err="failed to get container status \"fb81d6fc503e4ff8c2c5100d614cd8ebb11e49ed1d1ec2d46dd2430545d48fa5\": rpc error: code = NotFound desc = could not find container \"fb81d6fc503e4ff8c2c5100d614cd8ebb11e49ed1d1ec2d46dd2430545d48fa5\": container with ID starting with fb81d6fc503e4ff8c2c5100d614cd8ebb11e49ed1d1ec2d46dd2430545d48fa5 not found: ID does not exist" Oct 31 13:02:22.663187 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:22.663162 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-64f86ffbff-5fcnb"] Oct 31 13:02:22.671740 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:22.671714 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bc9ae8-64f86ffbff-5fcnb"] Oct 31 13:02:22.698743 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:22.697384 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0db792-854bc569bc-4tc49"] Oct 31 13:02:22.701126 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:22.701105 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0db792-854bc569bc-4tc49"] Oct 31 13:02:22.967819 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:22.967787 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-fc4458d75-d86fk"] Oct 31 13:02:23.468578 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:23.467549 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="820722a4-2f7a-45b0-911a-fc744c660284" path="/var/lib/kubelet/pods/820722a4-2f7a-45b0-911a-fc744c660284/volumes" Oct 31 13:02:23.468795 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:23.468693 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afe96299-b227-472a-98a2-fafb8dcfca80" path="/var/lib/kubelet/pods/afe96299-b227-472a-98a2-fafb8dcfca80/volumes" Oct 31 13:02:27.969149 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:27.969090 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-fc4458d75-d86fk" podUID="e00d7b50-2ee9-4019-9ccf-7d26cd06c118" containerName="payload-pre-processing" containerID="cri-o://42ce795b58089dabf4b1a5b6cd54c73eeb593ee1b98d81a10559bc1e1f09625e" gracePeriod=25 Oct 31 13:02:28.301287 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:28.301256 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-fc4458d75-d86fk" Oct 31 13:02:28.379822 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:28.379789 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e00d7b50-2ee9-4019-9ccf-7d26cd06c118-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e00d7b50-2ee9-4019-9ccf-7d26cd06c118-plugins-config-volume\") pod \"e00d7b50-2ee9-4019-9ccf-7d26cd06c118\" (UID: \"e00d7b50-2ee9-4019-9ccf-7d26cd06c118\") " Oct 31 13:02:28.380208 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:28.379852 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e00d7b50-2ee9-4019-9ccf-7d26cd06c118-kube-api-access-kf5j5\" (UniqueName: \"kubernetes.io/projected/e00d7b50-2ee9-4019-9ccf-7d26cd06c118-kube-api-access-kf5j5\") pod \"e00d7b50-2ee9-4019-9ccf-7d26cd06c118\" (UID: \"e00d7b50-2ee9-4019-9ccf-7d26cd06c118\") " Oct 31 13:02:28.380316 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:28.380248 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e00d7b50-2ee9-4019-9ccf-7d26cd06c118-plugins-config-volume" pod "e00d7b50-2ee9-4019-9ccf-7d26cd06c118" (UID: "e00d7b50-2ee9-4019-9ccf-7d26cd06c118"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:02:28.380524 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:28.380507 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e00d7b50-2ee9-4019-9ccf-7d26cd06c118-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:28.382250 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:28.382225 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e00d7b50-2ee9-4019-9ccf-7d26cd06c118-kube-api-access-kf5j5" pod "e00d7b50-2ee9-4019-9ccf-7d26cd06c118" (UID: "e00d7b50-2ee9-4019-9ccf-7d26cd06c118"). InnerVolumeSpecName "kube-api-access-kf5j5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:02:28.483147 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:28.483115 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf5j5\" (UniqueName: \"kubernetes.io/projected/e00d7b50-2ee9-4019-9ccf-7d26cd06c118-kube-api-access-kf5j5\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:28.691307 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:28.691268 5693 generic.go:356] "Generic (PLEG): container finished" podID="e00d7b50-2ee9-4019-9ccf-7d26cd06c118" containerID="42ce795b58089dabf4b1a5b6cd54c73eeb593ee1b98d81a10559bc1e1f09625e" exitCode=0 Oct 31 13:02:28.691521 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:28.691334 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-fc4458d75-d86fk" Oct 31 13:02:28.691521 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:28.691356 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-fc4458d75-d86fk" event={"ID":"e00d7b50-2ee9-4019-9ccf-7d26cd06c118","Type":"ContainerDied","Data":"42ce795b58089dabf4b1a5b6cd54c73eeb593ee1b98d81a10559bc1e1f09625e"} Oct 31 13:02:28.691521 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:28.691449 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-fc4458d75-d86fk" event={"ID":"e00d7b50-2ee9-4019-9ccf-7d26cd06c118","Type":"ContainerDied","Data":"e4c6151b8bf822a58e5e46602092e2820ce47d1a21b6ba997703e55435160280"} Oct 31 13:02:28.691521 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:28.691481 5693 scope.go:122] "RemoveContainer" containerID="42ce795b58089dabf4b1a5b6cd54c73eeb593ee1b98d81a10559bc1e1f09625e" Oct 31 13:02:28.702696 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:28.702675 5693 scope.go:122] "RemoveContainer" containerID="42ce795b58089dabf4b1a5b6cd54c73eeb593ee1b98d81a10559bc1e1f09625e" Oct 31 13:02:28.702997 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:02:28.702976 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42ce795b58089dabf4b1a5b6cd54c73eeb593ee1b98d81a10559bc1e1f09625e\": container with ID starting with 42ce795b58089dabf4b1a5b6cd54c73eeb593ee1b98d81a10559bc1e1f09625e not found: ID does not exist" containerID="42ce795b58089dabf4b1a5b6cd54c73eeb593ee1b98d81a10559bc1e1f09625e" Oct 31 13:02:28.703067 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:28.703006 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42ce795b58089dabf4b1a5b6cd54c73eeb593ee1b98d81a10559bc1e1f09625e"} err="failed to get container status \"42ce795b58089dabf4b1a5b6cd54c73eeb593ee1b98d81a10559bc1e1f09625e\": rpc error: code = NotFound desc = could not find container \"42ce795b58089dabf4b1a5b6cd54c73eeb593ee1b98d81a10559bc1e1f09625e\": container with ID starting with 42ce795b58089dabf4b1a5b6cd54c73eeb593ee1b98d81a10559bc1e1f09625e not found: ID does not exist" Oct 31 13:02:28.723326 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:28.723300 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-fc4458d75-d86fk"] Oct 31 13:02:28.730423 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:28.730398 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4989f9-fc4458d75-d86fk"] Oct 31 13:02:29.443596 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:29.443502 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e00d7b50-2ee9-4019-9ccf-7d26cd06c118" path="/var/lib/kubelet/pods/e00d7b50-2ee9-4019-9ccf-7d26cd06c118/volumes" Oct 31 13:02:38.859906 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:38.859811 5693 generic.go:356] "Generic (PLEG): container finished" podID="85311012-2071-4e3d-ab91-f5f21c7e2ab5" containerID="ed445a1d4a7cdac6f139bdc73761a4aa41c6e5c80956082d2aa943866be2c0c6" exitCode=0 Oct 31 13:02:38.859906 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:38.859880 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" event={"ID":"85311012-2071-4e3d-ab91-f5f21c7e2ab5","Type":"ContainerDied","Data":"ed445a1d4a7cdac6f139bdc73761a4aa41c6e5c80956082d2aa943866be2c0c6"} Oct 31 13:02:38.860435 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:38.860302 5693 scope.go:122] "RemoveContainer" containerID="ed445a1d4a7cdac6f139bdc73761a4aa41c6e5c80956082d2aa943866be2c0c6" Oct 31 13:02:38.861861 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:38.861830 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:02:38.861974 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:38.861902 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:02:38.862042 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:38.861961 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:02:39.867731 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:39.867700 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" event={"ID":"85311012-2071-4e3d-ab91-f5f21c7e2ab5","Type":"ContainerStarted","Data":"48f72a31a82a11aa6d296e90bed8822e9aec92345fd38bed9293eca1a57ac34c"} Oct 31 13:02:40.002665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:40.002051 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" Oct 31 13:02:40.002665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:40.002139 5693 patch_prober.go:29] interesting pod/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 31 13:02:40.002665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:40.002189 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" podUID="85311012-2071-4e3d-ab91-f5f21c7e2ab5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 31 13:02:41.001864 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:41.001831 5693 patch_prober.go:29] interesting pod/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 31 13:02:41.002512 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:41.002478 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" podUID="85311012-2071-4e3d-ab91-f5f21c7e2ab5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 31 13:02:42.006667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:41.999042 5693 patch_prober.go:29] interesting pod/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 31 13:02:42.006667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:41.999121 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" podUID="85311012-2071-4e3d-ab91-f5f21c7e2ab5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 31 13:02:42.851023 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:42.850983 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q"] Oct 31 13:02:42.851564 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:42.851512 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q" podUID="0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9" containerName="maas-api" containerID="cri-o://4bc61ca98da9d3c4c1ee6bbab056a89bd71db98b3e579d1b6cf0afdd4dde6d69" gracePeriod=30 Oct 31 13:02:43.003145 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:43.003114 5693 patch_prober.go:29] interesting pod/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 31 13:02:43.003383 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:43.003359 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" podUID="85311012-2071-4e3d-ab91-f5f21c7e2ab5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 31 13:02:43.410497 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:43.410474 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q" Oct 31 13:02:43.510311 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:43.510266 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9-openshift-service-ca\") pod \"0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9\" (UID: \"0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9\") " Oct 31 13:02:43.511497 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:43.511477 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9-maas-api-tls\") pod \"0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9\" (UID: \"0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9\") " Oct 31 13:02:43.511982 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:43.511847 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9-openshift-service-ca" pod "0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9" (UID: "0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:02:43.512162 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:43.512145 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9-metrics-certs\") pod \"0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9\" (UID: \"0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9\") " Oct 31 13:02:43.512328 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:43.512311 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9-kube-api-access-q4t2g\" (UniqueName: \"kubernetes.io/projected/0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9-kube-api-access-q4t2g\") pod \"0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9\" (UID: \"0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9\") " Oct 31 13:02:43.513037 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:43.513022 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:43.515674 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:43.515280 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9-maas-api-tls" pod "0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9" (UID: "0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:02:43.515674 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:43.515379 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9-metrics-certs" pod "0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9" (UID: "0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:02:43.517786 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:43.517769 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9-kube-api-access-q4t2g" pod "0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9" (UID: "0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9"). InnerVolumeSpecName "kube-api-access-q4t2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:02:43.635207 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:43.634921 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:43.635207 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:43.634961 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:43.635207 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:43.634978 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4t2g\" (UniqueName: \"kubernetes.io/projected/0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9-kube-api-access-q4t2g\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:43.939021 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:43.938981 5693 generic.go:356] "Generic (PLEG): container finished" podID="0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9" containerID="4bc61ca98da9d3c4c1ee6bbab056a89bd71db98b3e579d1b6cf0afdd4dde6d69" exitCode=0 Oct 31 13:02:43.939216 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:43.939035 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q" Oct 31 13:02:43.939216 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:43.939060 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q" event={"ID":"0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9","Type":"ContainerDied","Data":"4bc61ca98da9d3c4c1ee6bbab056a89bd71db98b3e579d1b6cf0afdd4dde6d69"} Oct 31 13:02:43.939216 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:43.939117 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q" event={"ID":"0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9","Type":"ContainerDied","Data":"ef995ecbad39fd47cade566536c20ff9964082b54e85b6ebf3d8edbf03058637"} Oct 31 13:02:43.939216 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:43.939141 5693 scope.go:122] "RemoveContainer" containerID="4bc61ca98da9d3c4c1ee6bbab056a89bd71db98b3e579d1b6cf0afdd4dde6d69" Oct 31 13:02:43.950404 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:43.950385 5693 scope.go:122] "RemoveContainer" containerID="4bc61ca98da9d3c4c1ee6bbab056a89bd71db98b3e579d1b6cf0afdd4dde6d69" Oct 31 13:02:43.950691 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:02:43.950671 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bc61ca98da9d3c4c1ee6bbab056a89bd71db98b3e579d1b6cf0afdd4dde6d69\": container with ID starting with 4bc61ca98da9d3c4c1ee6bbab056a89bd71db98b3e579d1b6cf0afdd4dde6d69 not found: ID does not exist" containerID="4bc61ca98da9d3c4c1ee6bbab056a89bd71db98b3e579d1b6cf0afdd4dde6d69" Oct 31 13:02:43.950747 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:43.950699 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bc61ca98da9d3c4c1ee6bbab056a89bd71db98b3e579d1b6cf0afdd4dde6d69"} err="failed to get container status \"4bc61ca98da9d3c4c1ee6bbab056a89bd71db98b3e579d1b6cf0afdd4dde6d69\": rpc error: code = NotFound desc = could not find container \"4bc61ca98da9d3c4c1ee6bbab056a89bd71db98b3e579d1b6cf0afdd4dde6d69\": container with ID starting with 4bc61ca98da9d3c4c1ee6bbab056a89bd71db98b3e579d1b6cf0afdd4dde6d69 not found: ID does not exist" Oct 31 13:02:43.963137 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:43.963116 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q"] Oct 31 13:02:43.968783 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:43.968762 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c8c51-5fd9487c78-kb27q"] Oct 31 13:02:44.001094 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:44.001053 5693 patch_prober.go:29] interesting pod/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 31 13:02:44.001282 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:44.001136 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" podUID="85311012-2071-4e3d-ab91-f5f21c7e2ab5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 31 13:02:44.326655 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:44.325192 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-664c944bf9-lr9dl"] Oct 31 13:02:44.326655 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:44.325494 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-664c944bf9-lr9dl" podUID="c1565456-7221-4d1e-974a-a46efbfe4b91" containerName="payload-pre-processing" containerID="cri-o://4227905d1d7dd92c5e5618fcbddfa234d1df195e8bb7579720bddcef5b038ed2" gracePeriod=30 Oct 31 13:02:44.921204 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:44.920922 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-664c944bf9-lr9dl" Oct 31 13:02:44.949323 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:44.949286 5693 generic.go:356] "Generic (PLEG): container finished" podID="c1565456-7221-4d1e-974a-a46efbfe4b91" containerID="4227905d1d7dd92c5e5618fcbddfa234d1df195e8bb7579720bddcef5b038ed2" exitCode=0 Oct 31 13:02:44.949488 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:44.949473 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-664c944bf9-lr9dl" event={"ID":"c1565456-7221-4d1e-974a-a46efbfe4b91","Type":"ContainerDied","Data":"4227905d1d7dd92c5e5618fcbddfa234d1df195e8bb7579720bddcef5b038ed2"} Oct 31 13:02:44.949548 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:44.949523 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-664c944bf9-lr9dl" event={"ID":"c1565456-7221-4d1e-974a-a46efbfe4b91","Type":"ContainerDied","Data":"86a8b35f8064a18e19b1afaa94331ed681313b7c24e0f59b1f384d75aa1b6a08"} Oct 31 13:02:44.949604 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:44.949552 5693 scope.go:122] "RemoveContainer" containerID="4227905d1d7dd92c5e5618fcbddfa234d1df195e8bb7579720bddcef5b038ed2" Oct 31 13:02:44.949688 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:44.949673 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-664c944bf9-lr9dl" Oct 31 13:02:44.964351 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:44.964313 5693 scope.go:122] "RemoveContainer" containerID="4227905d1d7dd92c5e5618fcbddfa234d1df195e8bb7579720bddcef5b038ed2" Oct 31 13:02:44.964694 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:02:44.964629 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4227905d1d7dd92c5e5618fcbddfa234d1df195e8bb7579720bddcef5b038ed2\": container with ID starting with 4227905d1d7dd92c5e5618fcbddfa234d1df195e8bb7579720bddcef5b038ed2 not found: ID does not exist" containerID="4227905d1d7dd92c5e5618fcbddfa234d1df195e8bb7579720bddcef5b038ed2" Oct 31 13:02:44.964888 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:44.964860 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4227905d1d7dd92c5e5618fcbddfa234d1df195e8bb7579720bddcef5b038ed2"} err="failed to get container status \"4227905d1d7dd92c5e5618fcbddfa234d1df195e8bb7579720bddcef5b038ed2\": rpc error: code = NotFound desc = could not find container \"4227905d1d7dd92c5e5618fcbddfa234d1df195e8bb7579720bddcef5b038ed2\": container with ID starting with 4227905d1d7dd92c5e5618fcbddfa234d1df195e8bb7579720bddcef5b038ed2 not found: ID does not exist" Oct 31 13:02:45.023690 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:45.022178 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" Oct 31 13:02:45.027684 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:45.024778 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1565456-7221-4d1e-974a-a46efbfe4b91-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1565456-7221-4d1e-974a-a46efbfe4b91-plugins-config-volume\") pod \"c1565456-7221-4d1e-974a-a46efbfe4b91\" (UID: \"c1565456-7221-4d1e-974a-a46efbfe4b91\") " Oct 31 13:02:45.027684 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:45.025624 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1565456-7221-4d1e-974a-a46efbfe4b91-kube-api-access-24kpz\" (UniqueName: \"kubernetes.io/projected/c1565456-7221-4d1e-974a-a46efbfe4b91-kube-api-access-24kpz\") pod \"c1565456-7221-4d1e-974a-a46efbfe4b91\" (UID: \"c1565456-7221-4d1e-974a-a46efbfe4b91\") " Oct 31 13:02:45.027684 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:45.025255 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1565456-7221-4d1e-974a-a46efbfe4b91-plugins-config-volume" pod "c1565456-7221-4d1e-974a-a46efbfe4b91" (UID: "c1565456-7221-4d1e-974a-a46efbfe4b91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:02:45.027684 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:45.027008 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1565456-7221-4d1e-974a-a46efbfe4b91-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:45.035990 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:45.028461 5693 patch_prober.go:29] interesting pod/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 31 13:02:45.035990 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:45.028512 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" podUID="85311012-2071-4e3d-ab91-f5f21c7e2ab5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 31 13:02:45.063162 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:45.049287 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1565456-7221-4d1e-974a-a46efbfe4b91-kube-api-access-24kpz" pod "c1565456-7221-4d1e-974a-a46efbfe4b91" (UID: "c1565456-7221-4d1e-974a-a46efbfe4b91"). InnerVolumeSpecName "kube-api-access-24kpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:02:45.128777 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:45.128689 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24kpz\" (UniqueName: \"kubernetes.io/projected/c1565456-7221-4d1e-974a-a46efbfe4b91-kube-api-access-24kpz\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:45.279990 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:45.279952 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-664c944bf9-lr9dl"] Oct 31 13:02:45.294540 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:45.294511 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-664c944bf9-lr9dl"] Oct 31 13:02:45.411539 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:45.411499 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9" path="/var/lib/kubelet/pods/0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9/volumes" Oct 31 13:02:45.412059 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:45.412038 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1565456-7221-4d1e-974a-a46efbfe4b91" path="/var/lib/kubelet/pods/c1565456-7221-4d1e-974a-a46efbfe4b91/volumes" Oct 31 13:02:46.003416 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:46.003387 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" Oct 31 13:02:46.004729 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:46.004703 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" Oct 31 13:02:48.687008 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:02:48.686972 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:02:59.660934 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:59.660898 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m"] Oct 31 13:02:59.661438 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:59.661148 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m" podUID="6a1ee165-b894-429a-b703-d99a2839af4f" containerName="maas-api" containerID="cri-o://e688e6a208d6b3b99cf4c1a5fd59cf1fa1ef10c4f7e59576127a62b530d98195" gracePeriod=30 Oct 31 13:02:59.746248 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:59.746192 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-adfd11-758bb459dd-zvhmq"] Oct 31 13:02:59.746971 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:59.746918 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-adfd11-758bb459dd-zvhmq" podUID="74e3eaa3-e742-48ae-a5de-c923170b9f26" containerName="payload-processing" containerID="cri-o://1e84d7691aafe9444fcc1927e91786dd806ec8caf04ed3f3d865f6a159e4d22f" gracePeriod=30 Oct 31 13:02:59.835091 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:02:59.835056 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-adfd11-758bb459dd-zvhmq" podUID="74e3eaa3-e742-48ae-a5de-c923170b9f26" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.106:9004: connect: connection refused" Oct 31 13:03:00.194178 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:00.181967 5693 generic.go:356] "Generic (PLEG): container finished" podID="6a1ee165-b894-429a-b703-d99a2839af4f" containerID="e688e6a208d6b3b99cf4c1a5fd59cf1fa1ef10c4f7e59576127a62b530d98195" exitCode=0 Oct 31 13:03:00.194178 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:00.182045 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m" event={"ID":"6a1ee165-b894-429a-b703-d99a2839af4f","Type":"ContainerDied","Data":"e688e6a208d6b3b99cf4c1a5fd59cf1fa1ef10c4f7e59576127a62b530d98195"} Oct 31 13:03:00.222144 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:00.221791 5693 generic.go:356] "Generic (PLEG): container finished" podID="74e3eaa3-e742-48ae-a5de-c923170b9f26" containerID="1e84d7691aafe9444fcc1927e91786dd806ec8caf04ed3f3d865f6a159e4d22f" exitCode=0 Oct 31 13:03:00.222144 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:00.221844 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-adfd11-758bb459dd-zvhmq" event={"ID":"74e3eaa3-e742-48ae-a5de-c923170b9f26","Type":"ContainerDied","Data":"1e84d7691aafe9444fcc1927e91786dd806ec8caf04ed3f3d865f6a159e4d22f"} Oct 31 13:03:00.262481 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:00.262411 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m" Oct 31 13:03:00.275988 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:00.275962 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-adfd11-758bb459dd-zvhmq" Oct 31 13:03:00.278314 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:00.278291 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a1ee165-b894-429a-b703-d99a2839af4f-kube-api-access-74g45\" (UniqueName: \"kubernetes.io/projected/6a1ee165-b894-429a-b703-d99a2839af4f-kube-api-access-74g45\") pod \"6a1ee165-b894-429a-b703-d99a2839af4f\" (UID: \"6a1ee165-b894-429a-b703-d99a2839af4f\") " Oct 31 13:03:00.278488 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:00.278470 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a1ee165-b894-429a-b703-d99a2839af4f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a1ee165-b894-429a-b703-d99a2839af4f-maas-api-tls\") pod \"6a1ee165-b894-429a-b703-d99a2839af4f\" (UID: \"6a1ee165-b894-429a-b703-d99a2839af4f\") " Oct 31 13:03:00.278536 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:00.278510 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74e3eaa3-e742-48ae-a5de-c923170b9f26-kube-api-access-w5khk\" (UniqueName: \"kubernetes.io/projected/74e3eaa3-e742-48ae-a5de-c923170b9f26-kube-api-access-w5khk\") pod \"74e3eaa3-e742-48ae-a5de-c923170b9f26\" (UID: \"74e3eaa3-e742-48ae-a5de-c923170b9f26\") " Oct 31 13:03:00.278606 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:00.278590 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74e3eaa3-e742-48ae-a5de-c923170b9f26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74e3eaa3-e742-48ae-a5de-c923170b9f26-plugins-config-volume\") pod \"74e3eaa3-e742-48ae-a5de-c923170b9f26\" (UID: \"74e3eaa3-e742-48ae-a5de-c923170b9f26\") " Oct 31 13:03:00.278724 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:00.278707 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a1ee165-b894-429a-b703-d99a2839af4f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a1ee165-b894-429a-b703-d99a2839af4f-metrics-certs\") pod \"6a1ee165-b894-429a-b703-d99a2839af4f\" (UID: \"6a1ee165-b894-429a-b703-d99a2839af4f\") " Oct 31 13:03:00.278896 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:00.278819 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a1ee165-b894-429a-b703-d99a2839af4f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a1ee165-b894-429a-b703-d99a2839af4f-openshift-service-ca\") pod \"6a1ee165-b894-429a-b703-d99a2839af4f\" (UID: \"6a1ee165-b894-429a-b703-d99a2839af4f\") " Oct 31 13:03:00.279236 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:00.279216 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a1ee165-b894-429a-b703-d99a2839af4f-openshift-service-ca" pod "6a1ee165-b894-429a-b703-d99a2839af4f" (UID: "6a1ee165-b894-429a-b703-d99a2839af4f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:03:00.279300 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:00.279240 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74e3eaa3-e742-48ae-a5de-c923170b9f26-plugins-config-volume" pod "74e3eaa3-e742-48ae-a5de-c923170b9f26" (UID: "74e3eaa3-e742-48ae-a5de-c923170b9f26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:03:00.281256 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:00.281234 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a1ee165-b894-429a-b703-d99a2839af4f-kube-api-access-74g45" pod "6a1ee165-b894-429a-b703-d99a2839af4f" (UID: "6a1ee165-b894-429a-b703-d99a2839af4f"). InnerVolumeSpecName "kube-api-access-74g45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:03:00.281359 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:00.281262 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74e3eaa3-e742-48ae-a5de-c923170b9f26-kube-api-access-w5khk" pod "74e3eaa3-e742-48ae-a5de-c923170b9f26" (UID: "74e3eaa3-e742-48ae-a5de-c923170b9f26"). InnerVolumeSpecName "kube-api-access-w5khk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:03:00.282370 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:00.282346 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a1ee165-b894-429a-b703-d99a2839af4f-metrics-certs" pod "6a1ee165-b894-429a-b703-d99a2839af4f" (UID: "6a1ee165-b894-429a-b703-d99a2839af4f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:03:00.283697 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:00.283670 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a1ee165-b894-429a-b703-d99a2839af4f-maas-api-tls" pod "6a1ee165-b894-429a-b703-d99a2839af4f" (UID: "6a1ee165-b894-429a-b703-d99a2839af4f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:03:00.380310 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:00.380274 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74e3eaa3-e742-48ae-a5de-c923170b9f26-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:00.380310 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:00.380304 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a1ee165-b894-429a-b703-d99a2839af4f-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:00.380310 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:00.380314 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a1ee165-b894-429a-b703-d99a2839af4f-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:00.380538 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:00.380323 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74g45\" (UniqueName: \"kubernetes.io/projected/6a1ee165-b894-429a-b703-d99a2839af4f-kube-api-access-74g45\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:00.380538 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:00.380332 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a1ee165-b894-429a-b703-d99a2839af4f-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:00.380538 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:00.380351 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5khk\" (UniqueName: \"kubernetes.io/projected/74e3eaa3-e742-48ae-a5de-c923170b9f26-kube-api-access-w5khk\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:01.229897 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:01.229868 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m" Oct 31 13:03:01.230371 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:01.229866 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m" event={"ID":"6a1ee165-b894-429a-b703-d99a2839af4f","Type":"ContainerDied","Data":"2159e177bef853bc6f8161d4297d8c93d47d3c6d6969535fd040d017fd16d46c"} Oct 31 13:03:01.230371 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:01.230001 5693 scope.go:122] "RemoveContainer" containerID="e688e6a208d6b3b99cf4c1a5fd59cf1fa1ef10c4f7e59576127a62b530d98195" Oct 31 13:03:01.231372 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:01.231321 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-adfd11-758bb459dd-zvhmq" event={"ID":"74e3eaa3-e742-48ae-a5de-c923170b9f26","Type":"ContainerDied","Data":"d6b3b54fc01cf000d81a70d613b46fa7db661d2f3642d94083f3c6afebb5d7fe"} Oct 31 13:03:01.231372 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:01.231346 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-adfd11-758bb459dd-zvhmq" Oct 31 13:03:01.240845 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:01.240721 5693 scope.go:122] "RemoveContainer" containerID="1e84d7691aafe9444fcc1927e91786dd806ec8caf04ed3f3d865f6a159e4d22f" Oct 31 13:03:01.259803 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:01.259771 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-adfd11-758bb459dd-zvhmq"] Oct 31 13:03:01.263347 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:01.263324 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-adfd11-758bb459dd-zvhmq"] Oct 31 13:03:01.276024 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:01.276000 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m"] Oct 31 13:03:01.282243 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:01.282223 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-adfd11-86bfd5cb4f-pb56m"] Oct 31 13:03:01.408550 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:01.408519 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a1ee165-b894-429a-b703-d99a2839af4f" path="/var/lib/kubelet/pods/6a1ee165-b894-429a-b703-d99a2839af4f/volumes" Oct 31 13:03:01.409027 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:01.409011 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74e3eaa3-e742-48ae-a5de-c923170b9f26" path="/var/lib/kubelet/pods/74e3eaa3-e742-48ae-a5de-c923170b9f26/volumes" Oct 31 13:03:02.139979 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:02.139947 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg"] Oct 31 13:03:02.140318 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:02.140221 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" podUID="1c8d2a80-a147-44f0-b49b-c3bd5b29fef0" containerName="istio-proxy" containerID="cri-o://c551421591d8d19afd9e7798a8a3d5e34c30cdf05365097a407cab874f6cac12" gracePeriod=30 Oct 31 13:03:03.910199 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:03.910169 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-49bffb-7678b64f8f-zdp9v"] Oct 31 13:03:03.910750 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:03.910737 5693 cpu_manager.go:414] "containerMap: removing container" podUID="e00d7b50-2ee9-4019-9ccf-7d26cd06c118" containerName="payload-pre-processing" Oct 31 13:03:03.910816 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:03.910751 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e00d7b50-2ee9-4019-9ccf-7d26cd06c118" containerName="payload-pre-processing" Oct 31 13:03:03.910816 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:03.910764 5693 cpu_manager.go:414] "containerMap: removing container" podUID="c1565456-7221-4d1e-974a-a46efbfe4b91" containerName="payload-pre-processing" Oct 31 13:03:03.910816 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:03.910769 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1565456-7221-4d1e-974a-a46efbfe4b91" containerName="payload-pre-processing" Oct 31 13:03:03.910816 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:03.910782 5693 cpu_manager.go:414] "containerMap: removing container" podUID="820722a4-2f7a-45b0-911a-fc744c660284" containerName="payload-pre-processing" Oct 31 13:03:03.910816 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:03.910786 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="820722a4-2f7a-45b0-911a-fc744c660284" containerName="payload-pre-processing" Oct 31 13:03:03.910816 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:03.910797 5693 cpu_manager.go:414] "containerMap: removing container" podUID="659d976b-0eec-480e-ab29-c95814ab7e11" containerName="maas-api" Oct 31 13:03:03.910816 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:03.910802 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="659d976b-0eec-480e-ab29-c95814ab7e11" containerName="maas-api" Oct 31 13:03:03.910816 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:03.910809 5693 cpu_manager.go:414] "containerMap: removing container" podUID="0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9" containerName="maas-api" Oct 31 13:03:03.910816 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:03.910814 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9" containerName="maas-api" Oct 31 13:03:03.911102 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:03.910830 5693 cpu_manager.go:414] "containerMap: removing container" podUID="6a1ee165-b894-429a-b703-d99a2839af4f" containerName="maas-api" Oct 31 13:03:03.911102 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:03.910834 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a1ee165-b894-429a-b703-d99a2839af4f" containerName="maas-api" Oct 31 13:03:03.911102 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:03.910843 5693 cpu_manager.go:414] "containerMap: removing container" podUID="74e3eaa3-e742-48ae-a5de-c923170b9f26" containerName="payload-processing" Oct 31 13:03:03.911102 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:03.910847 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74e3eaa3-e742-48ae-a5de-c923170b9f26" containerName="payload-processing" Oct 31 13:03:03.911102 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:03.910858 5693 cpu_manager.go:414] "containerMap: removing container" podUID="afe96299-b227-472a-98a2-fafb8dcfca80" containerName="payload-processing" Oct 31 13:03:03.911102 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:03.910863 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afe96299-b227-472a-98a2-fafb8dcfca80" containerName="payload-processing" Oct 31 13:03:03.911102 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:03.910949 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="74e3eaa3-e742-48ae-a5de-c923170b9f26" containerName="payload-processing" Oct 31 13:03:03.911102 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:03.910957 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a1ee165-b894-429a-b703-d99a2839af4f" containerName="maas-api" Oct 31 13:03:03.911102 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:03.910964 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1565456-7221-4d1e-974a-a46efbfe4b91" containerName="payload-pre-processing" Oct 31 13:03:03.911102 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:03.910971 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="afe96299-b227-472a-98a2-fafb8dcfca80" containerName="payload-processing" Oct 31 13:03:03.911102 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:03.910980 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e5b8170-8f4b-4577-81d8-bd4b4f1db5d9" containerName="maas-api" Oct 31 13:03:03.911102 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:03.910986 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="820722a4-2f7a-45b0-911a-fc744c660284" containerName="payload-pre-processing" Oct 31 13:03:03.911102 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:03.910991 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="e00d7b50-2ee9-4019-9ccf-7d26cd06c118" containerName="payload-pre-processing" Oct 31 13:03:03.911102 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:03.910995 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="659d976b-0eec-480e-ab29-c95814ab7e11" containerName="maas-api" Oct 31 13:03:03.915781 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:03.915756 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-49bffb-7678b64f8f-zdp9v" Oct 31 13:03:03.937333 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:03.937308 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-49bffb\"" Oct 31 13:03:03.942136 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:03.942112 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/084961ae-b1f7-4f97-aca7-21e9cc5ae882-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-49bffb-7678b64f8f-zdp9v\" (UID: \"084961ae-b1f7-4f97-aca7-21e9cc5ae882\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-49bffb-7678b64f8f-zdp9v" Oct 31 13:03:03.942240 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:03.942190 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tt7b\" (UniqueName: \"kubernetes.io/projected/084961ae-b1f7-4f97-aca7-21e9cc5ae882-kube-api-access-2tt7b\") pod \"payload-processing-e2e-praxis-a-49bffb-7678b64f8f-zdp9v\" (UID: \"084961ae-b1f7-4f97-aca7-21e9cc5ae882\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-49bffb-7678b64f8f-zdp9v" Oct 31 13:03:03.949337 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:03.949313 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-49bffb-74bccc9cbc-6mlwp"] Oct 31 13:03:03.954760 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:03.954736 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-49bffb-7678b64f8f-zdp9v"] Oct 31 13:03:03.954863 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:03.954826 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-49bffb-74bccc9cbc-6mlwp" Oct 31 13:03:03.975844 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:03.975825 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-49bffb-dockercfg-br7wn\"" Oct 31 13:03:03.981331 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:03.981303 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-49bffb-74bccc9cbc-6mlwp"] Oct 31 13:03:04.043582 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:04.043551 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39047437-bef9-4407-b5e6-2781f6811546-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-49bffb-74bccc9cbc-6mlwp\" (UID: \"39047437-bef9-4407-b5e6-2781f6811546\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-49bffb-74bccc9cbc-6mlwp" Oct 31 13:03:04.043810 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:04.043625 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldvrr\" (UniqueName: \"kubernetes.io/projected/39047437-bef9-4407-b5e6-2781f6811546-kube-api-access-ldvrr\") pod \"payload-pre-processing-e2e-praxis-a-49bffb-74bccc9cbc-6mlwp\" (UID: \"39047437-bef9-4407-b5e6-2781f6811546\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-49bffb-74bccc9cbc-6mlwp" Oct 31 13:03:04.043886 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:04.043856 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/084961ae-b1f7-4f97-aca7-21e9cc5ae882-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-49bffb-7678b64f8f-zdp9v\" (UID: \"084961ae-b1f7-4f97-aca7-21e9cc5ae882\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-49bffb-7678b64f8f-zdp9v" Oct 31 13:03:04.043989 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:04.043969 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tt7b\" (UniqueName: \"kubernetes.io/projected/084961ae-b1f7-4f97-aca7-21e9cc5ae882-kube-api-access-2tt7b\") pod \"payload-processing-e2e-praxis-a-49bffb-7678b64f8f-zdp9v\" (UID: \"084961ae-b1f7-4f97-aca7-21e9cc5ae882\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-49bffb-7678b64f8f-zdp9v" Oct 31 13:03:04.044773 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:04.044749 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/084961ae-b1f7-4f97-aca7-21e9cc5ae882-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-49bffb-7678b64f8f-zdp9v\" (UID: \"084961ae-b1f7-4f97-aca7-21e9cc5ae882\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-49bffb-7678b64f8f-zdp9v" Oct 31 13:03:04.124686 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:04.108544 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tt7b\" (UniqueName: \"kubernetes.io/projected/084961ae-b1f7-4f97-aca7-21e9cc5ae882-kube-api-access-2tt7b\") pod \"payload-processing-e2e-praxis-a-49bffb-7678b64f8f-zdp9v\" (UID: \"084961ae-b1f7-4f97-aca7-21e9cc5ae882\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-49bffb-7678b64f8f-zdp9v" Oct 31 13:03:04.145751 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:04.145709 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39047437-bef9-4407-b5e6-2781f6811546-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-49bffb-74bccc9cbc-6mlwp\" (UID: \"39047437-bef9-4407-b5e6-2781f6811546\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-49bffb-74bccc9cbc-6mlwp" Oct 31 13:03:04.146889 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:04.146862 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldvrr\" (UniqueName: \"kubernetes.io/projected/39047437-bef9-4407-b5e6-2781f6811546-kube-api-access-ldvrr\") pod \"payload-pre-processing-e2e-praxis-a-49bffb-74bccc9cbc-6mlwp\" (UID: \"39047437-bef9-4407-b5e6-2781f6811546\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-49bffb-74bccc9cbc-6mlwp" Oct 31 13:03:04.151666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:04.146742 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39047437-bef9-4407-b5e6-2781f6811546-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-49bffb-74bccc9cbc-6mlwp\" (UID: \"39047437-bef9-4407-b5e6-2781f6811546\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-49bffb-74bccc9cbc-6mlwp" Oct 31 13:03:04.188729 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:04.184787 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldvrr\" (UniqueName: \"kubernetes.io/projected/39047437-bef9-4407-b5e6-2781f6811546-kube-api-access-ldvrr\") pod \"payload-pre-processing-e2e-praxis-a-49bffb-74bccc9cbc-6mlwp\" (UID: \"39047437-bef9-4407-b5e6-2781f6811546\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-49bffb-74bccc9cbc-6mlwp" Oct 31 13:03:04.230666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:04.229938 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-49bffb-7678b64f8f-zdp9v" Oct 31 13:03:04.269982 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:04.269950 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-49bffb-74bccc9cbc-6mlwp" Oct 31 13:03:04.601441 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:04.601407 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-49bffb-7678b64f8f-zdp9v"] Oct 31 13:03:04.601591 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:03:04.601525 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod084961ae_b1f7_4f97_aca7_21e9cc5ae882.slice/crio-85560b811426a8f05907132e478d04414a47c1981c370320710d873c71a56398 WatchSource:0}: Error finding container 85560b811426a8f05907132e478d04414a47c1981c370320710d873c71a56398: Status 404 returned error can't find the container with id 85560b811426a8f05907132e478d04414a47c1981c370320710d873c71a56398 Oct 31 13:03:04.655466 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:04.655441 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-49bffb-74bccc9cbc-6mlwp"] Oct 31 13:03:04.655714 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:03:04.655692 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39047437_bef9_4407_b5e6_2781f6811546.slice/crio-e16f5d81df5c90e1221b29fcfdfc364fcd739cefae99a8800e90dbb740d6bc2d WatchSource:0}: Error finding container e16f5d81df5c90e1221b29fcfdfc364fcd739cefae99a8800e90dbb740d6bc2d: Status 404 returned error can't find the container with id e16f5d81df5c90e1221b29fcfdfc364fcd739cefae99a8800e90dbb740d6bc2d Oct 31 13:03:05.317290 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:05.317249 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-49bffb-7678b64f8f-zdp9v" event={"ID":"084961ae-b1f7-4f97-aca7-21e9cc5ae882","Type":"ContainerStarted","Data":"43ff6dabfbbc97921a16be1f9337dd09c242f324b39f5f498d7bdf17ffafa72e"} Oct 31 13:03:05.317770 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:05.317317 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-49bffb-7678b64f8f-zdp9v" event={"ID":"084961ae-b1f7-4f97-aca7-21e9cc5ae882","Type":"ContainerStarted","Data":"85560b811426a8f05907132e478d04414a47c1981c370320710d873c71a56398"} Oct 31 13:03:05.317770 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:05.317348 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-49bffb-7678b64f8f-zdp9v" Oct 31 13:03:05.319043 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:05.319018 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-49bffb-74bccc9cbc-6mlwp" event={"ID":"39047437-bef9-4407-b5e6-2781f6811546","Type":"ContainerStarted","Data":"694d53bae5e1409ceaeebd0b031b0cae66a9c1d57cba4ac86138305a3cf54925"} Oct 31 13:03:05.319150 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:05.319061 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-49bffb-74bccc9cbc-6mlwp" Oct 31 13:03:05.319150 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:05.319076 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-49bffb-74bccc9cbc-6mlwp" event={"ID":"39047437-bef9-4407-b5e6-2781f6811546","Type":"ContainerStarted","Data":"e16f5d81df5c90e1221b29fcfdfc364fcd739cefae99a8800e90dbb740d6bc2d"} Oct 31 13:03:05.351865 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:05.351528 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-49bffb-7678b64f8f-zdp9v" podStartSLOduration=2.351509983 podStartE2EDuration="2.351509983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:05.339365271 +0000 UTC m=+1836.046951568" watchObservedRunningTime="2026-10-31 13:03:05.351509983 +0000 UTC m=+1836.059096279" Oct 31 13:03:05.367207 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:05.367146 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-49bffb-74bccc9cbc-6mlwp" podStartSLOduration=2.367124508 podStartE2EDuration="2.367124508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:05.364307838 +0000 UTC m=+1836.071894134" watchObservedRunningTime="2026-10-31 13:03:05.367124508 +0000 UTC m=+1836.074710802" Oct 31 13:03:07.346299 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:07.345953 5693 generic.go:356] "Generic (PLEG): container finished" podID="1c8d2a80-a147-44f0-b49b-c3bd5b29fef0" containerID="c551421591d8d19afd9e7798a8a3d5e34c30cdf05365097a407cab874f6cac12" exitCode=0 Oct 31 13:03:07.346299 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:07.346065 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" event={"ID":"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0","Type":"ContainerDied","Data":"c551421591d8d19afd9e7798a8a3d5e34c30cdf05365097a407cab874f6cac12"} Oct 31 13:03:07.346299 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:07.346193 5693 scope.go:122] "RemoveContainer" containerID="04d076bc5e9ccd0ffcf480ba0b8e33c104bfc85033cf970007f7c761f5205dfb" Oct 31 13:03:07.448294 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:07.448271 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" Oct 31 13:03:07.528707 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:07.528675 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-workload-socket\") pod \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\" (UID: \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\") " Oct 31 13:03:07.528905 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:07.528740 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-credential-socket\") pod \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\" (UID: \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\") " Oct 31 13:03:07.528905 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:07.528781 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-istio-podinfo\") pod \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\" (UID: \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\") " Oct 31 13:03:07.528905 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:07.528831 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-istiod-ca-cert\") pod \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\" (UID: \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\") " Oct 31 13:03:07.528905 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:07.528862 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-workload-certs\") pod \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\" (UID: \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\") " Oct 31 13:03:07.529117 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:07.528923 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-istio-data\") pod \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\" (UID: \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\") " Oct 31 13:03:07.529117 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:07.529044 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-credential-socket" pod "1c8d2a80-a147-44f0-b49b-c3bd5b29fef0" (UID: "1c8d2a80-a147-44f0-b49b-c3bd5b29fef0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:03:07.529117 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:07.529077 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-istio-token\" (UniqueName: \"kubernetes.io/projected/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-istio-token\") pod \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\" (UID: \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\") " Oct 31 13:03:07.529276 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:07.529163 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-kube-api-access-hmv5d\" (UniqueName: \"kubernetes.io/projected/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-kube-api-access-hmv5d\") pod \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\" (UID: \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\") " Oct 31 13:03:07.529276 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:07.529231 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-istio-envoy\") pod \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\" (UID: \"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0\") " Oct 31 13:03:07.529596 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:07.529499 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-istio-data" pod "1c8d2a80-a147-44f0-b49b-c3bd5b29fef0" (UID: "1c8d2a80-a147-44f0-b49b-c3bd5b29fef0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:03:07.529596 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:07.529566 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-workload-certs" pod "1c8d2a80-a147-44f0-b49b-c3bd5b29fef0" (UID: "1c8d2a80-a147-44f0-b49b-c3bd5b29fef0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:03:07.530149 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:07.530131 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-istiod-ca-cert" pod "1c8d2a80-a147-44f0-b49b-c3bd5b29fef0" (UID: "1c8d2a80-a147-44f0-b49b-c3bd5b29fef0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:03:07.530280 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:07.530265 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-workload-socket" pod "1c8d2a80-a147-44f0-b49b-c3bd5b29fef0" (UID: "1c8d2a80-a147-44f0-b49b-c3bd5b29fef0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:03:07.530973 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:07.530455 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:07.531133 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:07.531118 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:07.531220 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:07.531209 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:07.531957 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:07.531934 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-istio-token" pod "1c8d2a80-a147-44f0-b49b-c3bd5b29fef0" (UID: "1c8d2a80-a147-44f0-b49b-c3bd5b29fef0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:03:07.532062 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:07.532015 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-istio-envoy" pod "1c8d2a80-a147-44f0-b49b-c3bd5b29fef0" (UID: "1c8d2a80-a147-44f0-b49b-c3bd5b29fef0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:03:07.533311 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:07.533284 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-istio-podinfo" pod "1c8d2a80-a147-44f0-b49b-c3bd5b29fef0" (UID: "1c8d2a80-a147-44f0-b49b-c3bd5b29fef0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:03:07.533982 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:07.533947 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-kube-api-access-hmv5d" pod "1c8d2a80-a147-44f0-b49b-c3bd5b29fef0" (UID: "1c8d2a80-a147-44f0-b49b-c3bd5b29fef0"). InnerVolumeSpecName "kube-api-access-hmv5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:03:07.632923 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:07.632891 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:07.632923 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:07.632919 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:07.632923 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:07.632929 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:07.633160 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:07.632938 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:07.633160 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:07.632947 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmv5d\" (UniqueName: \"kubernetes.io/projected/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-kube-api-access-hmv5d\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:07.633160 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:07.632955 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:08.356482 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:08.356449 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" event={"ID":"1c8d2a80-a147-44f0-b49b-c3bd5b29fef0","Type":"ContainerDied","Data":"8e4dc2bc5f22c2ded4d803bcab6f8d36fd5e75950264574d325e1ecac23acdd5"} Oct 31 13:03:08.356926 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:08.356505 5693 scope.go:122] "RemoveContainer" containerID="c551421591d8d19afd9e7798a8a3d5e34c30cdf05365097a407cab874f6cac12" Oct 31 13:03:08.356926 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:08.356588 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg" Oct 31 13:03:08.385686 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:08.385569 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg"] Oct 31 13:03:08.393180 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:08.393156 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-adfd11-openshift-default-7d4bd788bf-h2ccg"] Oct 31 13:03:09.448711 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:09.430911 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c8d2a80-a147-44f0-b49b-c3bd5b29fef0" path="/var/lib/kubelet/pods/1c8d2a80-a147-44f0-b49b-c3bd5b29fef0/volumes" Oct 31 13:03:16.339662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:16.336539 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-49bffb-74bccc9cbc-6mlwp" Oct 31 13:03:16.346082 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:16.345816 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-49bffb-7678b64f8f-zdp9v" Oct 31 13:03:17.142808 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:17.142776 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf"] Oct 31 13:03:17.143299 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:17.143235 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf" podUID="2dc63886-9566-45e8-9861-383040f4c4fe" containerName="maas-api" containerID="cri-o://c8336bab7c8e597efe2444a10e4392ca2aeaee1ba998fed336171c8bb97dacc0" gracePeriod=30 Oct 31 13:03:17.438915 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:17.438882 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf" Oct 31 13:03:17.451377 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:17.451349 5693 generic.go:356] "Generic (PLEG): container finished" podID="2dc63886-9566-45e8-9861-383040f4c4fe" containerID="c8336bab7c8e597efe2444a10e4392ca2aeaee1ba998fed336171c8bb97dacc0" exitCode=0 Oct 31 13:03:17.451520 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:17.451396 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf" Oct 31 13:03:17.451520 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:17.451427 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf" event={"ID":"2dc63886-9566-45e8-9861-383040f4c4fe","Type":"ContainerDied","Data":"c8336bab7c8e597efe2444a10e4392ca2aeaee1ba998fed336171c8bb97dacc0"} Oct 31 13:03:17.451520 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:17.451485 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf" event={"ID":"2dc63886-9566-45e8-9861-383040f4c4fe","Type":"ContainerDied","Data":"0187e66b59ea82334c059f861d328a6f357c3903726a39074b490a376892bdf6"} Oct 31 13:03:17.451520 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:17.451508 5693 scope.go:122] "RemoveContainer" containerID="c8336bab7c8e597efe2444a10e4392ca2aeaee1ba998fed336171c8bb97dacc0" Oct 31 13:03:17.461338 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:17.461305 5693 scope.go:122] "RemoveContainer" containerID="c8336bab7c8e597efe2444a10e4392ca2aeaee1ba998fed336171c8bb97dacc0" Oct 31 13:03:17.461738 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:03:17.461687 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8336bab7c8e597efe2444a10e4392ca2aeaee1ba998fed336171c8bb97dacc0\": container with ID starting with c8336bab7c8e597efe2444a10e4392ca2aeaee1ba998fed336171c8bb97dacc0 not found: ID does not exist" containerID="c8336bab7c8e597efe2444a10e4392ca2aeaee1ba998fed336171c8bb97dacc0" Oct 31 13:03:17.461738 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:17.461726 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8336bab7c8e597efe2444a10e4392ca2aeaee1ba998fed336171c8bb97dacc0"} err="failed to get container status \"c8336bab7c8e597efe2444a10e4392ca2aeaee1ba998fed336171c8bb97dacc0\": rpc error: code = NotFound desc = could not find container \"c8336bab7c8e597efe2444a10e4392ca2aeaee1ba998fed336171c8bb97dacc0\": container with ID starting with c8336bab7c8e597efe2444a10e4392ca2aeaee1ba998fed336171c8bb97dacc0 not found: ID does not exist" Oct 31 13:03:17.692279 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:17.692185 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dc63886-9566-45e8-9861-383040f4c4fe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dc63886-9566-45e8-9861-383040f4c4fe-metrics-certs\") pod \"2dc63886-9566-45e8-9861-383040f4c4fe\" (UID: \"2dc63886-9566-45e8-9861-383040f4c4fe\") " Oct 31 13:03:17.692279 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:17.692252 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dc63886-9566-45e8-9861-383040f4c4fe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dc63886-9566-45e8-9861-383040f4c4fe-openshift-service-ca\") pod \"2dc63886-9566-45e8-9861-383040f4c4fe\" (UID: \"2dc63886-9566-45e8-9861-383040f4c4fe\") " Oct 31 13:03:17.692533 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:17.692313 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dc63886-9566-45e8-9861-383040f4c4fe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dc63886-9566-45e8-9861-383040f4c4fe-maas-api-tls\") pod \"2dc63886-9566-45e8-9861-383040f4c4fe\" (UID: \"2dc63886-9566-45e8-9861-383040f4c4fe\") " Oct 31 13:03:17.692533 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:17.692394 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dc63886-9566-45e8-9861-383040f4c4fe-kube-api-access-p47qv\" (UniqueName: \"kubernetes.io/projected/2dc63886-9566-45e8-9861-383040f4c4fe-kube-api-access-p47qv\") pod \"2dc63886-9566-45e8-9861-383040f4c4fe\" (UID: \"2dc63886-9566-45e8-9861-383040f4c4fe\") " Oct 31 13:03:17.692947 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:17.692919 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dc63886-9566-45e8-9861-383040f4c4fe-openshift-service-ca" pod "2dc63886-9566-45e8-9861-383040f4c4fe" (UID: "2dc63886-9566-45e8-9861-383040f4c4fe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:03:17.695157 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:17.695110 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dc63886-9566-45e8-9861-383040f4c4fe-metrics-certs" pod "2dc63886-9566-45e8-9861-383040f4c4fe" (UID: "2dc63886-9566-45e8-9861-383040f4c4fe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:03:17.695280 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:17.695153 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dc63886-9566-45e8-9861-383040f4c4fe-kube-api-access-p47qv" pod "2dc63886-9566-45e8-9861-383040f4c4fe" (UID: "2dc63886-9566-45e8-9861-383040f4c4fe"). InnerVolumeSpecName "kube-api-access-p47qv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:03:17.695772 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:17.695752 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dc63886-9566-45e8-9861-383040f4c4fe-maas-api-tls" pod "2dc63886-9566-45e8-9861-383040f4c4fe" (UID: "2dc63886-9566-45e8-9861-383040f4c4fe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:03:17.798107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:17.798009 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dc63886-9566-45e8-9861-383040f4c4fe-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:17.798107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:17.798050 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dc63886-9566-45e8-9861-383040f4c4fe-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:17.798107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:17.798066 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dc63886-9566-45e8-9861-383040f4c4fe-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:17.798107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:17.798080 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p47qv\" (UniqueName: \"kubernetes.io/projected/2dc63886-9566-45e8-9861-383040f4c4fe-kube-api-access-p47qv\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:17.813797 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:17.813729 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf"] Oct 31 13:03:17.819314 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:17.818554 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db792-87dbc659f-lfwrf"] Oct 31 13:03:18.689565 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:03:18.689530 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:03:19.409693 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:19.409634 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dc63886-9566-45e8-9861-383040f4c4fe" path="/var/lib/kubelet/pods/2dc63886-9566-45e8-9861-383040f4c4fe/volumes" Oct 31 13:03:20.471359 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:20.471326 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl"] Oct 31 13:03:20.472176 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:20.472141 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" podUID="85311012-2071-4e3d-ab91-f5f21c7e2ab5" containerName="istio-proxy" containerID="cri-o://48f72a31a82a11aa6d296e90bed8822e9aec92345fd38bed9293eca1a57ac34c" gracePeriod=30 Oct 31 13:03:25.854916 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:25.854884 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" Oct 31 13:03:25.900262 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:25.900223 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85311012-2071-4e3d-ab91-f5f21c7e2ab5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85311012-2071-4e3d-ab91-f5f21c7e2ab5-istio-podinfo\") pod \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\" (UID: \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\") " Oct 31 13:03:25.900448 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:25.900355 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85311012-2071-4e3d-ab91-f5f21c7e2ab5-istio-token\" (UniqueName: \"kubernetes.io/projected/85311012-2071-4e3d-ab91-f5f21c7e2ab5-istio-token\") pod \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\" (UID: \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\") " Oct 31 13:03:25.900448 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:25.900395 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85311012-2071-4e3d-ab91-f5f21c7e2ab5-kube-api-access-kzx5b\" (UniqueName: \"kubernetes.io/projected/85311012-2071-4e3d-ab91-f5f21c7e2ab5-kube-api-access-kzx5b\") pod \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\" (UID: \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\") " Oct 31 13:03:25.900574 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:25.900457 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85311012-2071-4e3d-ab91-f5f21c7e2ab5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85311012-2071-4e3d-ab91-f5f21c7e2ab5-workload-certs\") pod \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\" (UID: \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\") " Oct 31 13:03:25.900574 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:25.900490 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85311012-2071-4e3d-ab91-f5f21c7e2ab5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85311012-2071-4e3d-ab91-f5f21c7e2ab5-credential-socket\") pod \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\" (UID: \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\") " Oct 31 13:03:25.900574 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:25.900537 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85311012-2071-4e3d-ab91-f5f21c7e2ab5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85311012-2071-4e3d-ab91-f5f21c7e2ab5-istio-data\") pod \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\" (UID: \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\") " Oct 31 13:03:25.900755 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:25.900580 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85311012-2071-4e3d-ab91-f5f21c7e2ab5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85311012-2071-4e3d-ab91-f5f21c7e2ab5-istio-envoy\") pod \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\" (UID: \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\") " Oct 31 13:03:25.900938 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:25.900897 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85311012-2071-4e3d-ab91-f5f21c7e2ab5-workload-certs" pod "85311012-2071-4e3d-ab91-f5f21c7e2ab5" (UID: "85311012-2071-4e3d-ab91-f5f21c7e2ab5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:03:25.901097 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:25.901063 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85311012-2071-4e3d-ab91-f5f21c7e2ab5-istio-data" pod "85311012-2071-4e3d-ab91-f5f21c7e2ab5" (UID: "85311012-2071-4e3d-ab91-f5f21c7e2ab5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:03:25.901335 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:25.901311 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85311012-2071-4e3d-ab91-f5f21c7e2ab5-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:25.901597 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:25.901338 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85311012-2071-4e3d-ab91-f5f21c7e2ab5-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:25.901597 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:25.901401 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85311012-2071-4e3d-ab91-f5f21c7e2ab5-credential-socket" pod "85311012-2071-4e3d-ab91-f5f21c7e2ab5" (UID: "85311012-2071-4e3d-ab91-f5f21c7e2ab5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:03:25.903144 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:25.903116 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85311012-2071-4e3d-ab91-f5f21c7e2ab5-istio-podinfo" pod "85311012-2071-4e3d-ab91-f5f21c7e2ab5" (UID: "85311012-2071-4e3d-ab91-f5f21c7e2ab5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:03:25.903312 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:25.903288 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85311012-2071-4e3d-ab91-f5f21c7e2ab5-istio-token" pod "85311012-2071-4e3d-ab91-f5f21c7e2ab5" (UID: "85311012-2071-4e3d-ab91-f5f21c7e2ab5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:03:25.903912 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:25.903873 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85311012-2071-4e3d-ab91-f5f21c7e2ab5-istio-envoy" pod "85311012-2071-4e3d-ab91-f5f21c7e2ab5" (UID: "85311012-2071-4e3d-ab91-f5f21c7e2ab5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:03:25.904031 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:25.903991 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85311012-2071-4e3d-ab91-f5f21c7e2ab5-kube-api-access-kzx5b" pod "85311012-2071-4e3d-ab91-f5f21c7e2ab5" (UID: "85311012-2071-4e3d-ab91-f5f21c7e2ab5"). InnerVolumeSpecName "kube-api-access-kzx5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:03:26.002569 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:26.002488 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85311012-2071-4e3d-ab91-f5f21c7e2ab5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85311012-2071-4e3d-ab91-f5f21c7e2ab5-workload-socket\") pod \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\" (UID: \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\") " Oct 31 13:03:26.002569 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:26.002547 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85311012-2071-4e3d-ab91-f5f21c7e2ab5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85311012-2071-4e3d-ab91-f5f21c7e2ab5-istiod-ca-cert\") pod \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\" (UID: \"85311012-2071-4e3d-ab91-f5f21c7e2ab5\") " Oct 31 13:03:26.002804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:26.002780 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85311012-2071-4e3d-ab91-f5f21c7e2ab5-workload-socket" pod "85311012-2071-4e3d-ab91-f5f21c7e2ab5" (UID: "85311012-2071-4e3d-ab91-f5f21c7e2ab5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:03:26.003002 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:26.002980 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85311012-2071-4e3d-ab91-f5f21c7e2ab5-istiod-ca-cert" pod "85311012-2071-4e3d-ab91-f5f21c7e2ab5" (UID: "85311012-2071-4e3d-ab91-f5f21c7e2ab5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:03:26.003251 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:26.003239 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85311012-2071-4e3d-ab91-f5f21c7e2ab5-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:26.003291 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:26.003257 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzx5b\" (UniqueName: \"kubernetes.io/projected/85311012-2071-4e3d-ab91-f5f21c7e2ab5-kube-api-access-kzx5b\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:26.003291 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:26.003268 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85311012-2071-4e3d-ab91-f5f21c7e2ab5-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:26.003291 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:26.003278 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85311012-2071-4e3d-ab91-f5f21c7e2ab5-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:26.003379 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:26.003286 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85311012-2071-4e3d-ab91-f5f21c7e2ab5-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:26.003379 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:26.003301 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85311012-2071-4e3d-ab91-f5f21c7e2ab5-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:26.003379 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:26.003309 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85311012-2071-4e3d-ab91-f5f21c7e2ab5-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:26.514994 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:26.514960 5693 generic.go:356] "Generic (PLEG): container finished" podID="85311012-2071-4e3d-ab91-f5f21c7e2ab5" containerID="48f72a31a82a11aa6d296e90bed8822e9aec92345fd38bed9293eca1a57ac34c" exitCode=0 Oct 31 13:03:26.515177 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:26.515013 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" Oct 31 13:03:26.515177 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:26.515039 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" event={"ID":"85311012-2071-4e3d-ab91-f5f21c7e2ab5","Type":"ContainerDied","Data":"48f72a31a82a11aa6d296e90bed8822e9aec92345fd38bed9293eca1a57ac34c"} Oct 31 13:03:26.515177 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:26.515091 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl" event={"ID":"85311012-2071-4e3d-ab91-f5f21c7e2ab5","Type":"ContainerDied","Data":"bd6caeb904172f717c026890698c7eb5ce5be0d63306d2e49f25ac94ca0714d8"} Oct 31 13:03:26.515177 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:26.515114 5693 scope.go:122] "RemoveContainer" containerID="48f72a31a82a11aa6d296e90bed8822e9aec92345fd38bed9293eca1a57ac34c" Oct 31 13:03:26.530325 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:26.530304 5693 scope.go:122] "RemoveContainer" containerID="ed445a1d4a7cdac6f139bdc73761a4aa41c6e5c80956082d2aa943866be2c0c6" Oct 31 13:03:26.539873 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:26.539857 5693 scope.go:122] "RemoveContainer" containerID="48f72a31a82a11aa6d296e90bed8822e9aec92345fd38bed9293eca1a57ac34c" Oct 31 13:03:26.540142 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:03:26.540124 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48f72a31a82a11aa6d296e90bed8822e9aec92345fd38bed9293eca1a57ac34c\": container with ID starting with 48f72a31a82a11aa6d296e90bed8822e9aec92345fd38bed9293eca1a57ac34c not found: ID does not exist" containerID="48f72a31a82a11aa6d296e90bed8822e9aec92345fd38bed9293eca1a57ac34c" Oct 31 13:03:26.540217 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:26.540157 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48f72a31a82a11aa6d296e90bed8822e9aec92345fd38bed9293eca1a57ac34c"} err="failed to get container status \"48f72a31a82a11aa6d296e90bed8822e9aec92345fd38bed9293eca1a57ac34c\": rpc error: code = NotFound desc = could not find container \"48f72a31a82a11aa6d296e90bed8822e9aec92345fd38bed9293eca1a57ac34c\": container with ID starting with 48f72a31a82a11aa6d296e90bed8822e9aec92345fd38bed9293eca1a57ac34c not found: ID does not exist" Oct 31 13:03:26.540217 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:26.540188 5693 scope.go:122] "RemoveContainer" containerID="ed445a1d4a7cdac6f139bdc73761a4aa41c6e5c80956082d2aa943866be2c0c6" Oct 31 13:03:26.540430 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:03:26.540414 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed445a1d4a7cdac6f139bdc73761a4aa41c6e5c80956082d2aa943866be2c0c6\": container with ID starting with ed445a1d4a7cdac6f139bdc73761a4aa41c6e5c80956082d2aa943866be2c0c6 not found: ID does not exist" containerID="ed445a1d4a7cdac6f139bdc73761a4aa41c6e5c80956082d2aa943866be2c0c6" Oct 31 13:03:26.540469 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:26.540437 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed445a1d4a7cdac6f139bdc73761a4aa41c6e5c80956082d2aa943866be2c0c6"} err="failed to get container status \"ed445a1d4a7cdac6f139bdc73761a4aa41c6e5c80956082d2aa943866be2c0c6\": rpc error: code = NotFound desc = could not find container \"ed445a1d4a7cdac6f139bdc73761a4aa41c6e5c80956082d2aa943866be2c0c6\": container with ID starting with ed445a1d4a7cdac6f139bdc73761a4aa41c6e5c80956082d2aa943866be2c0c6 not found: ID does not exist" Oct 31 13:03:26.552484 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:26.552460 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl"] Oct 31 13:03:26.564135 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:26.555809 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-0db792-openshift-default-798bd88897-67jjl"] Oct 31 13:03:27.426270 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:27.426230 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85311012-2071-4e3d-ab91-f5f21c7e2ab5" path="/var/lib/kubelet/pods/85311012-2071-4e3d-ab91-f5f21c7e2ab5/volumes" Oct 31 13:03:29.119593 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.119529 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t"] Oct 31 13:03:29.120578 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.120546 5693 cpu_manager.go:414] "containerMap: removing container" podUID="1c8d2a80-a147-44f0-b49b-c3bd5b29fef0" containerName="istio-proxy" Oct 31 13:03:29.120762 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.120581 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c8d2a80-a147-44f0-b49b-c3bd5b29fef0" containerName="istio-proxy" Oct 31 13:03:29.120762 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.120594 5693 cpu_manager.go:414] "containerMap: removing container" podUID="1c8d2a80-a147-44f0-b49b-c3bd5b29fef0" containerName="istio-proxy" Oct 31 13:03:29.120762 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.120603 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c8d2a80-a147-44f0-b49b-c3bd5b29fef0" containerName="istio-proxy" Oct 31 13:03:29.120762 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.120617 5693 cpu_manager.go:414] "containerMap: removing container" podUID="85311012-2071-4e3d-ab91-f5f21c7e2ab5" containerName="istio-proxy" Oct 31 13:03:29.120762 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.120622 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85311012-2071-4e3d-ab91-f5f21c7e2ab5" containerName="istio-proxy" Oct 31 13:03:29.120762 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.120676 5693 cpu_manager.go:414] "containerMap: removing container" podUID="2dc63886-9566-45e8-9861-383040f4c4fe" containerName="maas-api" Oct 31 13:03:29.120762 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.120684 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dc63886-9566-45e8-9861-383040f4c4fe" containerName="maas-api" Oct 31 13:03:29.122804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.120794 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="85311012-2071-4e3d-ab91-f5f21c7e2ab5" containerName="istio-proxy" Oct 31 13:03:29.122804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.120838 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c8d2a80-a147-44f0-b49b-c3bd5b29fef0" containerName="istio-proxy" Oct 31 13:03:29.122804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.120851 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dc63886-9566-45e8-9861-383040f4c4fe" containerName="maas-api" Oct 31 13:03:29.122804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.120861 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c8d2a80-a147-44f0-b49b-c3bd5b29fef0" containerName="istio-proxy" Oct 31 13:03:29.127077 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.127059 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" Oct 31 13:03:29.135099 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.135070 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-204e6c-openshift-default-dockercfg-76s2z\"" Oct 31 13:03:29.143812 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.143758 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t"] Oct 31 13:03:29.161373 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.161342 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f339605-e98a-4957-88d8-59afe07f2a7b-workload-certs\") pod \"e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t\" (UID: \"0f339605-e98a-4957-88d8-59afe07f2a7b\") " pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" Oct 31 13:03:29.161542 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.161392 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f339605-e98a-4957-88d8-59afe07f2a7b-workload-socket\") pod \"e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t\" (UID: \"0f339605-e98a-4957-88d8-59afe07f2a7b\") " pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" Oct 31 13:03:29.161610 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.161553 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f339605-e98a-4957-88d8-59afe07f2a7b-istio-data\") pod \"e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t\" (UID: \"0f339605-e98a-4957-88d8-59afe07f2a7b\") " pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" Oct 31 13:03:29.161610 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.161600 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f339605-e98a-4957-88d8-59afe07f2a7b-istio-token\") pod \"e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t\" (UID: \"0f339605-e98a-4957-88d8-59afe07f2a7b\") " pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" Oct 31 13:03:29.161771 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.161747 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wknz6\" (UniqueName: \"kubernetes.io/projected/0f339605-e98a-4957-88d8-59afe07f2a7b-kube-api-access-wknz6\") pod \"e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t\" (UID: \"0f339605-e98a-4957-88d8-59afe07f2a7b\") " pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" Oct 31 13:03:29.161876 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.161853 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f339605-e98a-4957-88d8-59afe07f2a7b-istiod-ca-cert\") pod \"e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t\" (UID: \"0f339605-e98a-4957-88d8-59afe07f2a7b\") " pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" Oct 31 13:03:29.161934 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.161919 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f339605-e98a-4957-88d8-59afe07f2a7b-credential-socket\") pod \"e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t\" (UID: \"0f339605-e98a-4957-88d8-59afe07f2a7b\") " pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" Oct 31 13:03:29.162009 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.161993 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f339605-e98a-4957-88d8-59afe07f2a7b-istio-envoy\") pod \"e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t\" (UID: \"0f339605-e98a-4957-88d8-59afe07f2a7b\") " pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" Oct 31 13:03:29.162065 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.162041 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f339605-e98a-4957-88d8-59afe07f2a7b-istio-podinfo\") pod \"e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t\" (UID: \"0f339605-e98a-4957-88d8-59afe07f2a7b\") " pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" Oct 31 13:03:29.181497 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.181465 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-758f48-6785db86d7-j4phq"] Oct 31 13:03:29.182454 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.182433 5693 cpu_manager.go:414] "containerMap: removing container" podUID="85311012-2071-4e3d-ab91-f5f21c7e2ab5" containerName="istio-proxy" Oct 31 13:03:29.182518 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.182459 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85311012-2071-4e3d-ab91-f5f21c7e2ab5" containerName="istio-proxy" Oct 31 13:03:29.182630 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.182617 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="85311012-2071-4e3d-ab91-f5f21c7e2ab5" containerName="istio-proxy" Oct 31 13:03:29.186793 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.186765 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-758f48-6785db86d7-j4phq" Oct 31 13:03:29.193602 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.193581 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-758f48\"" Oct 31 13:03:29.197680 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.196894 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-758f48-6785db86d7-j4phq"] Oct 31 13:03:29.227541 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.227500 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-758f48-6456c959dc-665vf"] Oct 31 13:03:29.231884 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.231862 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-758f48-6456c959dc-665vf" Oct 31 13:03:29.240832 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.240814 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-758f48-dockercfg-cjz7t\"" Oct 31 13:03:29.263163 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.263140 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f339605-e98a-4957-88d8-59afe07f2a7b-istio-envoy\") pod \"e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t\" (UID: \"0f339605-e98a-4957-88d8-59afe07f2a7b\") " pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" Oct 31 13:03:29.263264 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.263179 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f339605-e98a-4957-88d8-59afe07f2a7b-istio-podinfo\") pod \"e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t\" (UID: \"0f339605-e98a-4957-88d8-59afe07f2a7b\") " pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" Oct 31 13:03:29.263264 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.263210 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f339605-e98a-4957-88d8-59afe07f2a7b-workload-certs\") pod \"e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t\" (UID: \"0f339605-e98a-4957-88d8-59afe07f2a7b\") " pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" Oct 31 13:03:29.263264 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.263250 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f339605-e98a-4957-88d8-59afe07f2a7b-workload-socket\") pod \"e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t\" (UID: \"0f339605-e98a-4957-88d8-59afe07f2a7b\") " pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" Oct 31 13:03:29.263399 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.263343 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f339605-e98a-4957-88d8-59afe07f2a7b-istio-data\") pod \"e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t\" (UID: \"0f339605-e98a-4957-88d8-59afe07f2a7b\") " pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" Oct 31 13:03:29.263399 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.263380 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f339605-e98a-4957-88d8-59afe07f2a7b-istio-token\") pod \"e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t\" (UID: \"0f339605-e98a-4957-88d8-59afe07f2a7b\") " pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" Oct 31 13:03:29.263511 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.263414 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fs2g\" (UniqueName: \"kubernetes.io/projected/5022d397-de96-44f9-a83b-f28678b75b84-kube-api-access-5fs2g\") pod \"payload-pre-processing-e2e-praxis-a-758f48-6785db86d7-j4phq\" (UID: \"5022d397-de96-44f9-a83b-f28678b75b84\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-758f48-6785db86d7-j4phq" Oct 31 13:03:29.263511 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.263482 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wknz6\" (UniqueName: \"kubernetes.io/projected/0f339605-e98a-4957-88d8-59afe07f2a7b-kube-api-access-wknz6\") pod \"e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t\" (UID: \"0f339605-e98a-4957-88d8-59afe07f2a7b\") " pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" Oct 31 13:03:29.263623 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.263521 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9krvq\" (UniqueName: \"kubernetes.io/projected/106bba13-d6d7-4c17-802e-bcf194297900-kube-api-access-9krvq\") pod \"payload-processing-e2e-praxis-a-758f48-6456c959dc-665vf\" (UID: \"106bba13-d6d7-4c17-802e-bcf194297900\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-758f48-6456c959dc-665vf" Oct 31 13:03:29.263623 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.263553 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5022d397-de96-44f9-a83b-f28678b75b84-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-758f48-6785db86d7-j4phq\" (UID: \"5022d397-de96-44f9-a83b-f28678b75b84\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-758f48-6785db86d7-j4phq" Oct 31 13:03:29.263623 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.263589 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f339605-e98a-4957-88d8-59afe07f2a7b-workload-certs\") pod \"e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t\" (UID: \"0f339605-e98a-4957-88d8-59afe07f2a7b\") " pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" Oct 31 13:03:29.263623 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.263597 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/106bba13-d6d7-4c17-802e-bcf194297900-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-758f48-6456c959dc-665vf\" (UID: \"106bba13-d6d7-4c17-802e-bcf194297900\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-758f48-6456c959dc-665vf" Oct 31 13:03:29.263797 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.263631 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f339605-e98a-4957-88d8-59afe07f2a7b-workload-socket\") pod \"e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t\" (UID: \"0f339605-e98a-4957-88d8-59afe07f2a7b\") " pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" Oct 31 13:03:29.263797 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.263686 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f339605-e98a-4957-88d8-59afe07f2a7b-istio-data\") pod \"e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t\" (UID: \"0f339605-e98a-4957-88d8-59afe07f2a7b\") " pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" Oct 31 13:03:29.263888 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.263795 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f339605-e98a-4957-88d8-59afe07f2a7b-istiod-ca-cert\") pod \"e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t\" (UID: \"0f339605-e98a-4957-88d8-59afe07f2a7b\") " pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" Oct 31 13:03:29.263888 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.263823 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f339605-e98a-4957-88d8-59afe07f2a7b-credential-socket\") pod \"e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t\" (UID: \"0f339605-e98a-4957-88d8-59afe07f2a7b\") " pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" Oct 31 13:03:29.264091 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.264072 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f339605-e98a-4957-88d8-59afe07f2a7b-credential-socket\") pod \"e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t\" (UID: \"0f339605-e98a-4957-88d8-59afe07f2a7b\") " pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" Oct 31 13:03:29.264394 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.264372 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f339605-e98a-4957-88d8-59afe07f2a7b-istiod-ca-cert\") pod \"e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t\" (UID: \"0f339605-e98a-4957-88d8-59afe07f2a7b\") " pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" Oct 31 13:03:29.265580 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.265553 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f339605-e98a-4957-88d8-59afe07f2a7b-istio-envoy\") pod \"e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t\" (UID: \"0f339605-e98a-4957-88d8-59afe07f2a7b\") " pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" Oct 31 13:03:29.265860 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.265841 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f339605-e98a-4957-88d8-59afe07f2a7b-istio-podinfo\") pod \"e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t\" (UID: \"0f339605-e98a-4957-88d8-59afe07f2a7b\") " pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" Oct 31 13:03:29.292148 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.292110 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-758f48-6456c959dc-665vf"] Oct 31 13:03:29.310086 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.310052 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f339605-e98a-4957-88d8-59afe07f2a7b-istio-token\") pod \"e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t\" (UID: \"0f339605-e98a-4957-88d8-59afe07f2a7b\") " pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" Oct 31 13:03:29.310227 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.310109 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wknz6\" (UniqueName: \"kubernetes.io/projected/0f339605-e98a-4957-88d8-59afe07f2a7b-kube-api-access-wknz6\") pod \"e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t\" (UID: \"0f339605-e98a-4957-88d8-59afe07f2a7b\") " pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" Oct 31 13:03:29.364689 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.364634 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fs2g\" (UniqueName: \"kubernetes.io/projected/5022d397-de96-44f9-a83b-f28678b75b84-kube-api-access-5fs2g\") pod \"payload-pre-processing-e2e-praxis-a-758f48-6785db86d7-j4phq\" (UID: \"5022d397-de96-44f9-a83b-f28678b75b84\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-758f48-6785db86d7-j4phq" Oct 31 13:03:29.364824 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.364749 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9krvq\" (UniqueName: \"kubernetes.io/projected/106bba13-d6d7-4c17-802e-bcf194297900-kube-api-access-9krvq\") pod \"payload-processing-e2e-praxis-a-758f48-6456c959dc-665vf\" (UID: \"106bba13-d6d7-4c17-802e-bcf194297900\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-758f48-6456c959dc-665vf" Oct 31 13:03:29.364824 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.364773 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5022d397-de96-44f9-a83b-f28678b75b84-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-758f48-6785db86d7-j4phq\" (UID: \"5022d397-de96-44f9-a83b-f28678b75b84\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-758f48-6785db86d7-j4phq" Oct 31 13:03:29.364824 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.364801 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/106bba13-d6d7-4c17-802e-bcf194297900-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-758f48-6456c959dc-665vf\" (UID: \"106bba13-d6d7-4c17-802e-bcf194297900\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-758f48-6456c959dc-665vf" Oct 31 13:03:29.365428 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.365407 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5022d397-de96-44f9-a83b-f28678b75b84-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-758f48-6785db86d7-j4phq\" (UID: \"5022d397-de96-44f9-a83b-f28678b75b84\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-758f48-6785db86d7-j4phq" Oct 31 13:03:29.365467 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.365455 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/106bba13-d6d7-4c17-802e-bcf194297900-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-758f48-6456c959dc-665vf\" (UID: \"106bba13-d6d7-4c17-802e-bcf194297900\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-758f48-6456c959dc-665vf" Oct 31 13:03:29.419668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.413887 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fs2g\" (UniqueName: \"kubernetes.io/projected/5022d397-de96-44f9-a83b-f28678b75b84-kube-api-access-5fs2g\") pod \"payload-pre-processing-e2e-praxis-a-758f48-6785db86d7-j4phq\" (UID: \"5022d397-de96-44f9-a83b-f28678b75b84\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-758f48-6785db86d7-j4phq" Oct 31 13:03:29.464834 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.455939 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-204e6c-openshift-default-dockercfg-76s2z\"" Oct 31 13:03:29.464834 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.461727 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" Oct 31 13:03:29.467081 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.467019 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9krvq\" (UniqueName: \"kubernetes.io/projected/106bba13-d6d7-4c17-802e-bcf194297900-kube-api-access-9krvq\") pod \"payload-processing-e2e-praxis-a-758f48-6456c959dc-665vf\" (UID: \"106bba13-d6d7-4c17-802e-bcf194297900\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-758f48-6456c959dc-665vf" Oct 31 13:03:29.507152 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.507122 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-758f48-6785db86d7-j4phq" Oct 31 13:03:29.590870 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.590843 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-758f48-dockercfg-cjz7t\"" Oct 31 13:03:29.607602 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.607579 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-758f48-6456c959dc-665vf" Oct 31 13:03:29.871783 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.871745 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t"] Oct 31 13:03:29.874011 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:03:29.873986 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f339605_e98a_4957_88d8_59afe07f2a7b.slice/crio-5dff25b60f6d732180b5c9ebe9a2fe23c973c0dfe8602644ccba341c5c45c396 WatchSource:0}: Error finding container 5dff25b60f6d732180b5c9ebe9a2fe23c973c0dfe8602644ccba341c5c45c396: Status 404 returned error can't find the container with id 5dff25b60f6d732180b5c9ebe9a2fe23c973c0dfe8602644ccba341c5c45c396 Oct 31 13:03:29.878066 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.878014 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:03:29.878343 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.878311 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:03:29.878577 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.878549 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:03:29.911488 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.911456 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-758f48-6785db86d7-j4phq"] Oct 31 13:03:29.925050 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:03:29.925012 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5022d397_de96_44f9_a83b_f28678b75b84.slice/crio-032fe138747a98266e67143af695348f3551739d5d1c50b5873cf7cf56b32378 WatchSource:0}: Error finding container 032fe138747a98266e67143af695348f3551739d5d1c50b5873cf7cf56b32378: Status 404 returned error can't find the container with id 032fe138747a98266e67143af695348f3551739d5d1c50b5873cf7cf56b32378 Oct 31 13:03:29.930815 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:29.930787 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-758f48-6456c959dc-665vf"] Oct 31 13:03:29.932157 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:03:29.932127 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod106bba13_d6d7_4c17_802e_bcf194297900.slice/crio-920341ef52d6744c3e170e6d9a458b41032bae30ff5750eec4377487baa71009 WatchSource:0}: Error finding container 920341ef52d6744c3e170e6d9a458b41032bae30ff5750eec4377487baa71009: Status 404 returned error can't find the container with id 920341ef52d6744c3e170e6d9a458b41032bae30ff5750eec4377487baa71009 Oct 31 13:03:30.621884 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:30.621845 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" event={"ID":"0f339605-e98a-4957-88d8-59afe07f2a7b","Type":"ContainerStarted","Data":"4903c0f767d66841eb4600396088e4dabad0bb51de52a827da3d27f1d333a42e"} Oct 31 13:03:30.622378 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:30.621915 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" event={"ID":"0f339605-e98a-4957-88d8-59afe07f2a7b","Type":"ContainerStarted","Data":"5dff25b60f6d732180b5c9ebe9a2fe23c973c0dfe8602644ccba341c5c45c396"} Oct 31 13:03:30.637622 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:30.635622 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-758f48-6456c959dc-665vf" event={"ID":"106bba13-d6d7-4c17-802e-bcf194297900","Type":"ContainerStarted","Data":"57fc39968384db080e7fa6aa3c1aaa2b285b85e955881527a412b143fa5dbb29"} Oct 31 13:03:30.637622 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:30.635697 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-758f48-6456c959dc-665vf" event={"ID":"106bba13-d6d7-4c17-802e-bcf194297900","Type":"ContainerStarted","Data":"920341ef52d6744c3e170e6d9a458b41032bae30ff5750eec4377487baa71009"} Oct 31 13:03:30.637622 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:30.636855 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-758f48-6456c959dc-665vf" Oct 31 13:03:30.639920 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:30.638504 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-758f48-6785db86d7-j4phq" event={"ID":"5022d397-de96-44f9-a83b-f28678b75b84","Type":"ContainerStarted","Data":"f0669d25033fd1e26c6788894fc59cba73c04b2b1d71b921b2277cd33dd404ad"} Oct 31 13:03:30.639920 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:30.638550 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-758f48-6785db86d7-j4phq" event={"ID":"5022d397-de96-44f9-a83b-f28678b75b84","Type":"ContainerStarted","Data":"032fe138747a98266e67143af695348f3551739d5d1c50b5873cf7cf56b32378"} Oct 31 13:03:30.639920 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:30.639239 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-758f48-6785db86d7-j4phq" Oct 31 13:03:30.672394 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:30.672212 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" podStartSLOduration=1.6721920460000002 podStartE2EDuration="1.672192046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:30.665718898 +0000 UTC m=+1861.373305196" watchObservedRunningTime="2026-10-31 13:03:30.672192046 +0000 UTC m=+1861.379778344" Oct 31 13:03:30.700461 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:30.700410 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-758f48-6785db86d7-j4phq" podStartSLOduration=1.700396278 podStartE2EDuration="1.700396278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:30.69903942 +0000 UTC m=+1861.406625717" watchObservedRunningTime="2026-10-31 13:03:30.700396278 +0000 UTC m=+1861.407982572" Oct 31 13:03:31.476703 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:31.471016 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" Oct 31 13:03:31.496662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:31.488984 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" Oct 31 13:03:31.538677 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:31.533034 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-758f48-6456c959dc-665vf" podStartSLOduration=2.533014752 podStartE2EDuration="2.533014752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:30.738824726 +0000 UTC m=+1861.446411024" watchObservedRunningTime="2026-10-31 13:03:31.533014752 +0000 UTC m=+1862.240601048" Oct 31 13:03:31.647291 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:31.647179 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" Oct 31 13:03:31.648161 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:31.648139 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" Oct 31 13:03:34.835679 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:34.835624 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-204e6c-789f8c8f7c-tzjf8"] Oct 31 13:03:34.840156 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:34.840131 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-204e6c-789f8c8f7c-tzjf8" Oct 31 13:03:34.854172 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:34.854146 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-204e6c-789f8c8f7c-tzjf8"] Oct 31 13:03:34.858511 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:34.858483 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-204e6c-dockercfg-dczwf\"" Oct 31 13:03:34.858654 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:34.858520 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-204e6c\"" Oct 31 13:03:34.886060 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:34.886028 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-204e6c-76ffbdd775-7b6th"] Oct 31 13:03:34.891224 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:34.891197 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-204e6c-76ffbdd775-7b6th" Oct 31 13:03:34.898116 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:34.897309 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-204e6c-76ffbdd775-7b6th"] Oct 31 13:03:34.914735 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:34.914707 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcp7q\" (UniqueName: \"kubernetes.io/projected/75dac63c-3308-45cc-84b0-be41aa27da1a-kube-api-access-dcp7q\") pod \"payload-pre-processing-e2e-praxis-b-204e6c-76ffbdd775-7b6th\" (UID: \"75dac63c-3308-45cc-84b0-be41aa27da1a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-204e6c-76ffbdd775-7b6th" Oct 31 13:03:34.914873 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:34.914834 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75dac63c-3308-45cc-84b0-be41aa27da1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-204e6c-76ffbdd775-7b6th\" (UID: \"75dac63c-3308-45cc-84b0-be41aa27da1a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-204e6c-76ffbdd775-7b6th" Oct 31 13:03:34.914941 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:34.914918 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce01e69-4d72-471e-a60a-ec028a9b2f87-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-204e6c-789f8c8f7c-tzjf8\" (UID: \"5ce01e69-4d72-471e-a60a-ec028a9b2f87\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-204e6c-789f8c8f7c-tzjf8" Oct 31 13:03:34.915004 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:34.914991 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf87z\" (UniqueName: \"kubernetes.io/projected/5ce01e69-4d72-471e-a60a-ec028a9b2f87-kube-api-access-zf87z\") pod \"payload-processing-e2e-praxis-b-204e6c-789f8c8f7c-tzjf8\" (UID: \"5ce01e69-4d72-471e-a60a-ec028a9b2f87\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-204e6c-789f8c8f7c-tzjf8" Oct 31 13:03:35.032124 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:35.032039 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcp7q\" (UniqueName: \"kubernetes.io/projected/75dac63c-3308-45cc-84b0-be41aa27da1a-kube-api-access-dcp7q\") pod \"payload-pre-processing-e2e-praxis-b-204e6c-76ffbdd775-7b6th\" (UID: \"75dac63c-3308-45cc-84b0-be41aa27da1a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-204e6c-76ffbdd775-7b6th" Oct 31 13:03:35.032124 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:35.032120 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75dac63c-3308-45cc-84b0-be41aa27da1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-204e6c-76ffbdd775-7b6th\" (UID: \"75dac63c-3308-45cc-84b0-be41aa27da1a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-204e6c-76ffbdd775-7b6th" Oct 31 13:03:35.032394 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:35.032172 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce01e69-4d72-471e-a60a-ec028a9b2f87-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-204e6c-789f8c8f7c-tzjf8\" (UID: \"5ce01e69-4d72-471e-a60a-ec028a9b2f87\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-204e6c-789f8c8f7c-tzjf8" Oct 31 13:03:35.032394 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:35.032247 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf87z\" (UniqueName: \"kubernetes.io/projected/5ce01e69-4d72-471e-a60a-ec028a9b2f87-kube-api-access-zf87z\") pod \"payload-processing-e2e-praxis-b-204e6c-789f8c8f7c-tzjf8\" (UID: \"5ce01e69-4d72-471e-a60a-ec028a9b2f87\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-204e6c-789f8c8f7c-tzjf8" Oct 31 13:03:35.048263 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:35.036474 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce01e69-4d72-471e-a60a-ec028a9b2f87-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-204e6c-789f8c8f7c-tzjf8\" (UID: \"5ce01e69-4d72-471e-a60a-ec028a9b2f87\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-204e6c-789f8c8f7c-tzjf8" Oct 31 13:03:35.059667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:35.049062 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75dac63c-3308-45cc-84b0-be41aa27da1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-204e6c-76ffbdd775-7b6th\" (UID: \"75dac63c-3308-45cc-84b0-be41aa27da1a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-204e6c-76ffbdd775-7b6th" Oct 31 13:03:35.063672 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:35.062347 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcp7q\" (UniqueName: \"kubernetes.io/projected/75dac63c-3308-45cc-84b0-be41aa27da1a-kube-api-access-dcp7q\") pod \"payload-pre-processing-e2e-praxis-b-204e6c-76ffbdd775-7b6th\" (UID: \"75dac63c-3308-45cc-84b0-be41aa27da1a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-204e6c-76ffbdd775-7b6th" Oct 31 13:03:35.095715 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:35.089579 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf87z\" (UniqueName: \"kubernetes.io/projected/5ce01e69-4d72-471e-a60a-ec028a9b2f87-kube-api-access-zf87z\") pod \"payload-processing-e2e-praxis-b-204e6c-789f8c8f7c-tzjf8\" (UID: \"5ce01e69-4d72-471e-a60a-ec028a9b2f87\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-204e6c-789f8c8f7c-tzjf8" Oct 31 13:03:35.157482 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:35.157450 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-204e6c-789f8c8f7c-tzjf8" Oct 31 13:03:35.211311 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:35.211043 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-204e6c-76ffbdd775-7b6th" Oct 31 13:03:35.499830 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:35.499773 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-204e6c-789f8c8f7c-tzjf8"] Oct 31 13:03:35.525262 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:03:35.525225 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ce01e69_4d72_471e_a60a_ec028a9b2f87.slice/crio-10f611e4205a5ba49dd6fc5a6e4a92b50fcbbcb79f943e680cb16e295185543b WatchSource:0}: Error finding container 10f611e4205a5ba49dd6fc5a6e4a92b50fcbbcb79f943e680cb16e295185543b: Status 404 returned error can't find the container with id 10f611e4205a5ba49dd6fc5a6e4a92b50fcbbcb79f943e680cb16e295185543b Oct 31 13:03:35.678896 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:35.678585 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-204e6c-789f8c8f7c-tzjf8" event={"ID":"5ce01e69-4d72-471e-a60a-ec028a9b2f87","Type":"ContainerStarted","Data":"10f611e4205a5ba49dd6fc5a6e4a92b50fcbbcb79f943e680cb16e295185543b"} Oct 31 13:03:35.783172 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:35.783072 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-204e6c-76ffbdd775-7b6th"] Oct 31 13:03:35.812947 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:03:35.812907 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75dac63c_3308_45cc_84b0_be41aa27da1a.slice/crio-90f1551315046677d13cd3242d21169a424f74c195ec874f23431ae6e1ce3208 WatchSource:0}: Error finding container 90f1551315046677d13cd3242d21169a424f74c195ec874f23431ae6e1ce3208: Status 404 returned error can't find the container with id 90f1551315046677d13cd3242d21169a424f74c195ec874f23431ae6e1ce3208 Oct 31 13:03:36.687961 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:36.687925 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-204e6c-76ffbdd775-7b6th" event={"ID":"75dac63c-3308-45cc-84b0-be41aa27da1a","Type":"ContainerStarted","Data":"0cb826a4474b229708cf2af686f7c537e75c2b208af851af0eca36902d0698bf"} Oct 31 13:03:36.688446 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:36.687993 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-204e6c-76ffbdd775-7b6th" event={"ID":"75dac63c-3308-45cc-84b0-be41aa27da1a","Type":"ContainerStarted","Data":"90f1551315046677d13cd3242d21169a424f74c195ec874f23431ae6e1ce3208"} Oct 31 13:03:36.688446 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:36.688037 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-204e6c-76ffbdd775-7b6th" Oct 31 13:03:36.690001 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:36.689976 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-204e6c-789f8c8f7c-tzjf8" event={"ID":"5ce01e69-4d72-471e-a60a-ec028a9b2f87","Type":"ContainerStarted","Data":"52fc0bea6825729f7529ad6cb75710667332eaa3c31f0678c4cc316a34d0ac8a"} Oct 31 13:03:36.690157 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:36.690101 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-204e6c-789f8c8f7c-tzjf8" Oct 31 13:03:36.715511 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:36.715460 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-204e6c-76ffbdd775-7b6th" podStartSLOduration=2.715446925 podStartE2EDuration="2.715446925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:36.712820267 +0000 UTC m=+1867.420406563" watchObservedRunningTime="2026-10-31 13:03:36.715446925 +0000 UTC m=+1867.423033220" Oct 31 13:03:36.752015 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:36.751959 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-204e6c-789f8c8f7c-tzjf8" podStartSLOduration=2.751941843 podStartE2EDuration="2.751941843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:36.748349546 +0000 UTC m=+1867.455935842" watchObservedRunningTime="2026-10-31 13:03:36.751941843 +0000 UTC m=+1867.459528139" Oct 31 13:03:36.788290 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:36.788256 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c347ff-86d57944cc-sf94h"] Oct 31 13:03:36.793790 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:36.793764 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c347ff-86d57944cc-sf94h" Oct 31 13:03:36.811336 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:36.811311 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c347ff\"" Oct 31 13:03:36.813098 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:36.813077 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c347ff-dockercfg-7q7ht\"" Oct 31 13:03:36.817078 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:36.817052 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c347ff-86d57944cc-sf94h"] Oct 31 13:03:36.866613 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:36.866581 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c5d33c5-5bd9-4386-a5d0-f905d505dc9a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c347ff-86d57944cc-sf94h\" (UID: \"5c5d33c5-5bd9-4386-a5d0-f905d505dc9a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c347ff-86d57944cc-sf94h" Oct 31 13:03:36.866815 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:36.866756 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2fwm\" (UniqueName: \"kubernetes.io/projected/5c5d33c5-5bd9-4386-a5d0-f905d505dc9a-kube-api-access-g2fwm\") pod \"payload-processing-e2e-ipp-b-c347ff-86d57944cc-sf94h\" (UID: \"5c5d33c5-5bd9-4386-a5d0-f905d505dc9a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c347ff-86d57944cc-sf94h" Oct 31 13:03:36.979717 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:36.971896 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c5d33c5-5bd9-4386-a5d0-f905d505dc9a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c347ff-86d57944cc-sf94h\" (UID: \"5c5d33c5-5bd9-4386-a5d0-f905d505dc9a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c347ff-86d57944cc-sf94h" Oct 31 13:03:36.979717 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:36.972048 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2fwm\" (UniqueName: \"kubernetes.io/projected/5c5d33c5-5bd9-4386-a5d0-f905d505dc9a-kube-api-access-g2fwm\") pod \"payload-processing-e2e-ipp-b-c347ff-86d57944cc-sf94h\" (UID: \"5c5d33c5-5bd9-4386-a5d0-f905d505dc9a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c347ff-86d57944cc-sf94h" Oct 31 13:03:36.979717 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:36.973288 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c5d33c5-5bd9-4386-a5d0-f905d505dc9a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c347ff-86d57944cc-sf94h\" (UID: \"5c5d33c5-5bd9-4386-a5d0-f905d505dc9a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c347ff-86d57944cc-sf94h" Oct 31 13:03:37.020704 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:37.020650 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2fwm\" (UniqueName: \"kubernetes.io/projected/5c5d33c5-5bd9-4386-a5d0-f905d505dc9a-kube-api-access-g2fwm\") pod \"payload-processing-e2e-ipp-b-c347ff-86d57944cc-sf94h\" (UID: \"5c5d33c5-5bd9-4386-a5d0-f905d505dc9a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c347ff-86d57944cc-sf94h" Oct 31 13:03:37.107859 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:37.107825 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c347ff-86d57944cc-sf94h" Oct 31 13:03:37.407106 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:03:37.407073 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c5d33c5_5bd9_4386_a5d0_f905d505dc9a.slice/crio-c8a86249f7e1535a3189b7702e00cbbbb08d70e70bdace3c89de2fa5f9e8cd7e WatchSource:0}: Error finding container c8a86249f7e1535a3189b7702e00cbbbb08d70e70bdace3c89de2fa5f9e8cd7e: Status 404 returned error can't find the container with id c8a86249f7e1535a3189b7702e00cbbbb08d70e70bdace3c89de2fa5f9e8cd7e Oct 31 13:03:37.411572 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:37.411548 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c347ff-86d57944cc-sf94h"] Oct 31 13:03:37.698112 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:37.698025 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c347ff-86d57944cc-sf94h" event={"ID":"5c5d33c5-5bd9-4386-a5d0-f905d505dc9a","Type":"ContainerStarted","Data":"12aa97db67f73eacf3a258cd934b46fe4ef53a726f666ec16382d68c7e89991e"} Oct 31 13:03:37.698570 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:37.698409 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c347ff-86d57944cc-sf94h" Oct 31 13:03:37.698570 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:37.698449 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c347ff-86d57944cc-sf94h" event={"ID":"5c5d33c5-5bd9-4386-a5d0-f905d505dc9a","Type":"ContainerStarted","Data":"c8a86249f7e1535a3189b7702e00cbbbb08d70e70bdace3c89de2fa5f9e8cd7e"} Oct 31 13:03:37.733505 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:37.731517 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c347ff-86d57944cc-sf94h" podStartSLOduration=1.7314976020000001 podStartE2EDuration="1.731497602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:37.72993112 +0000 UTC m=+1868.437517416" watchObservedRunningTime="2026-10-31 13:03:37.731497602 +0000 UTC m=+1868.439083902" Oct 31 13:03:41.070566 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:41.070531 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-758f48-c94487788-czlnw"] Oct 31 13:03:41.075238 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:41.075216 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-758f48-c94487788-czlnw" Oct 31 13:03:41.088822 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:41.088793 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-758f48-c94487788-czlnw"] Oct 31 13:03:41.089027 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:41.088819 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-758f48-serving-cert\"" Oct 31 13:03:41.134634 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:41.134599 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq4m2\" (UniqueName: \"kubernetes.io/projected/964ebb98-f4fe-43c0-acda-95c3cc1631c1-kube-api-access-cq4m2\") pod \"maas-api-e2e-praxis-a-758f48-c94487788-czlnw\" (UID: \"964ebb98-f4fe-43c0-acda-95c3cc1631c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-758f48-c94487788-czlnw" Oct 31 13:03:41.134826 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:41.134679 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/964ebb98-f4fe-43c0-acda-95c3cc1631c1-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-758f48-c94487788-czlnw\" (UID: \"964ebb98-f4fe-43c0-acda-95c3cc1631c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-758f48-c94487788-czlnw" Oct 31 13:03:41.134826 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:41.134801 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/964ebb98-f4fe-43c0-acda-95c3cc1631c1-maas-api-tls\") pod \"maas-api-e2e-praxis-a-758f48-c94487788-czlnw\" (UID: \"964ebb98-f4fe-43c0-acda-95c3cc1631c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-758f48-c94487788-czlnw" Oct 31 13:03:41.134826 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:41.134819 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/964ebb98-f4fe-43c0-acda-95c3cc1631c1-metrics-certs\") pod \"maas-api-e2e-praxis-a-758f48-c94487788-czlnw\" (UID: \"964ebb98-f4fe-43c0-acda-95c3cc1631c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-758f48-c94487788-czlnw" Oct 31 13:03:41.239946 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:41.238355 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/964ebb98-f4fe-43c0-acda-95c3cc1631c1-maas-api-tls\") pod \"maas-api-e2e-praxis-a-758f48-c94487788-czlnw\" (UID: \"964ebb98-f4fe-43c0-acda-95c3cc1631c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-758f48-c94487788-czlnw" Oct 31 13:03:41.239946 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:41.238410 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/964ebb98-f4fe-43c0-acda-95c3cc1631c1-metrics-certs\") pod \"maas-api-e2e-praxis-a-758f48-c94487788-czlnw\" (UID: \"964ebb98-f4fe-43c0-acda-95c3cc1631c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-758f48-c94487788-czlnw" Oct 31 13:03:41.239946 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:41.239218 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq4m2\" (UniqueName: \"kubernetes.io/projected/964ebb98-f4fe-43c0-acda-95c3cc1631c1-kube-api-access-cq4m2\") pod \"maas-api-e2e-praxis-a-758f48-c94487788-czlnw\" (UID: \"964ebb98-f4fe-43c0-acda-95c3cc1631c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-758f48-c94487788-czlnw" Oct 31 13:03:41.239946 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:41.239270 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/964ebb98-f4fe-43c0-acda-95c3cc1631c1-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-758f48-c94487788-czlnw\" (UID: \"964ebb98-f4fe-43c0-acda-95c3cc1631c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-758f48-c94487788-czlnw" Oct 31 13:03:41.240295 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:41.240116 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/964ebb98-f4fe-43c0-acda-95c3cc1631c1-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-758f48-c94487788-czlnw\" (UID: \"964ebb98-f4fe-43c0-acda-95c3cc1631c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-758f48-c94487788-czlnw" Oct 31 13:03:41.246411 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:41.246382 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/964ebb98-f4fe-43c0-acda-95c3cc1631c1-metrics-certs\") pod \"maas-api-e2e-praxis-a-758f48-c94487788-czlnw\" (UID: \"964ebb98-f4fe-43c0-acda-95c3cc1631c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-758f48-c94487788-czlnw" Oct 31 13:03:41.246583 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:41.246435 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/964ebb98-f4fe-43c0-acda-95c3cc1631c1-maas-api-tls\") pod \"maas-api-e2e-praxis-a-758f48-c94487788-czlnw\" (UID: \"964ebb98-f4fe-43c0-acda-95c3cc1631c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-758f48-c94487788-czlnw" Oct 31 13:03:41.255528 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:41.255494 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq4m2\" (UniqueName: \"kubernetes.io/projected/964ebb98-f4fe-43c0-acda-95c3cc1631c1-kube-api-access-cq4m2\") pod \"maas-api-e2e-praxis-a-758f48-c94487788-czlnw\" (UID: \"964ebb98-f4fe-43c0-acda-95c3cc1631c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-758f48-c94487788-czlnw" Oct 31 13:03:41.402094 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:41.402069 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-758f48-c94487788-czlnw" Oct 31 13:03:41.722514 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:41.722427 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-758f48-c94487788-czlnw"] Oct 31 13:03:41.738297 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:41.738272 5693 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 13:03:42.657045 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:42.657015 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-758f48-6456c959dc-665vf" Oct 31 13:03:42.657523 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:42.657145 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-758f48-6785db86d7-j4phq" Oct 31 13:03:42.769776 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:42.769707 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-758f48-c94487788-czlnw" event={"ID":"964ebb98-f4fe-43c0-acda-95c3cc1631c1","Type":"ContainerStarted","Data":"15bdd43cdbe827b6d8f976d63163f5d026da4c0147a599e0392c26dcd0e2acae"} Oct 31 13:03:42.769969 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:42.769790 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-758f48-c94487788-czlnw" event={"ID":"964ebb98-f4fe-43c0-acda-95c3cc1631c1","Type":"ContainerStarted","Data":"b9f3a8bd867d9fdc7abc03abc26f9fb4f028fdcdcfa86379d61f07e802b4c8b5"} Oct 31 13:03:42.769969 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:42.769839 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-758f48-c94487788-czlnw" Oct 31 13:03:42.819566 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:42.819516 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-758f48-c94487788-czlnw" podStartSLOduration=1.4402572390000001 podStartE2EDuration="1.819501731s" totalImagesPullingTime="379.244492ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:42.817910091 +0000 UTC m=+1873.525496386" watchObservedRunningTime="2026-10-31 13:03:42.819501731 +0000 UTC m=+1873.527088050" Oct 31 13:03:43.253428 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:43.253395 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf"] Oct 31 13:03:43.257793 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:43.257775 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" Oct 31 13:03:43.264936 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:43.264913 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-49bffb-openshift-default-dockercfg-7qsc9\"" Oct 31 13:03:43.277140 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:43.277105 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf"] Oct 31 13:03:43.415139 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:43.415102 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-workload-socket\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf\" (UID: \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" Oct 31 13:03:43.415450 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:43.415228 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-workload-certs\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf\" (UID: \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" Oct 31 13:03:43.415450 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:43.415376 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-credential-socket\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf\" (UID: \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" Oct 31 13:03:43.415450 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:43.415431 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-istio-data\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf\" (UID: \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" Oct 31 13:03:43.415774 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:43.415499 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-istio-token\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf\" (UID: \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" Oct 31 13:03:43.415774 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:43.415541 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-istiod-ca-cert\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf\" (UID: \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" Oct 31 13:03:43.415774 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:43.415577 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-istio-podinfo\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf\" (UID: \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" Oct 31 13:03:43.415774 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:43.415623 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-istio-envoy\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf\" (UID: \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" Oct 31 13:03:43.415983 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:43.415817 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m25tl\" (UniqueName: \"kubernetes.io/projected/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-kube-api-access-m25tl\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf\" (UID: \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" Oct 31 13:03:43.517230 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:43.517146 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-workload-certs\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf\" (UID: \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" Oct 31 13:03:43.517400 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:43.517237 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-credential-socket\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf\" (UID: \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" Oct 31 13:03:43.517400 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:43.517294 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-istio-data\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf\" (UID: \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" Oct 31 13:03:43.517400 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:43.517357 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-istio-token\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf\" (UID: \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" Oct 31 13:03:43.517400 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:43.517398 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-istiod-ca-cert\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf\" (UID: \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" Oct 31 13:03:43.517618 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:43.517426 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-istio-podinfo\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf\" (UID: \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" Oct 31 13:03:43.517618 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:43.517461 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-istio-envoy\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf\" (UID: \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" Oct 31 13:03:43.517777 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:43.517624 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-workload-certs\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf\" (UID: \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" Oct 31 13:03:43.517777 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:43.517748 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-credential-socket\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf\" (UID: \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" Oct 31 13:03:43.543387 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:43.517989 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-istio-data\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf\" (UID: \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" Oct 31 13:03:43.543387 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:43.518076 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m25tl\" (UniqueName: \"kubernetes.io/projected/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-kube-api-access-m25tl\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf\" (UID: \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" Oct 31 13:03:43.543387 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:43.518211 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-istiod-ca-cert\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf\" (UID: \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" Oct 31 13:03:43.543387 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:43.518216 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-workload-socket\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf\" (UID: \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" Oct 31 13:03:43.543387 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:43.520887 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-istio-envoy\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf\" (UID: \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" Oct 31 13:03:43.552951 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:43.548336 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-workload-socket\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf\" (UID: \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" Oct 31 13:03:43.555224 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:43.555066 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m25tl\" (UniqueName: \"kubernetes.io/projected/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-kube-api-access-m25tl\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf\" (UID: \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" Oct 31 13:03:43.555391 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:43.555365 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-istio-podinfo\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf\" (UID: \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" Oct 31 13:03:43.557223 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:43.557151 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-istio-token\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf\" (UID: \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" Oct 31 13:03:43.572864 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:43.572837 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" Oct 31 13:03:43.886203 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:43.886176 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf"] Oct 31 13:03:43.887457 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:03:43.887433 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebbd6fa9_5e1a_4244_8c0e_5e6f4c8df758.slice/crio-ef9ce215813d80cbec25579338b50a752c537e6745cbc7665e0868b68e16a5f1 WatchSource:0}: Error finding container ef9ce215813d80cbec25579338b50a752c537e6745cbc7665e0868b68e16a5f1: Status 404 returned error can't find the container with id ef9ce215813d80cbec25579338b50a752c537e6745cbc7665e0868b68e16a5f1 Oct 31 13:03:43.894748 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:43.894712 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:03:43.894847 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:43.894784 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:03:43.894847 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:43.894810 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:03:44.820135 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:44.820089 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" event={"ID":"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758","Type":"ContainerStarted","Data":"f35347918fc0ed154824bf23d1ab19e548ff352e8fb09680571cf4fba939104f"} Oct 31 13:03:44.820284 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:44.820168 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" event={"ID":"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758","Type":"ContainerStarted","Data":"ef9ce215813d80cbec25579338b50a752c537e6745cbc7665e0868b68e16a5f1"} Oct 31 13:03:45.573260 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:45.573226 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" Oct 31 13:03:45.574860 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:45.574830 5693 patch_prober.go:29] interesting pod/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 31 13:03:45.575006 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:45.574885 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" podUID="ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 31 13:03:46.573354 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:46.573320 5693 patch_prober.go:29] interesting pod/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 31 13:03:46.573845 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:46.573378 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" podUID="ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 31 13:03:47.573484 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:47.573449 5693 patch_prober.go:29] interesting pod/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 31 13:03:47.573883 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:47.573520 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" podUID="ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 31 13:03:47.774712 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:47.766853 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-204e6c-76ffbdd775-7b6th" Oct 31 13:03:47.783661 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:47.776801 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-204e6c-789f8c8f7c-tzjf8" Oct 31 13:03:47.794661 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:47.794299 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" podStartSLOduration=4.794276428 podStartE2EDuration="4.794276428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:44.853032885 +0000 UTC m=+1875.560619181" watchObservedRunningTime="2026-10-31 13:03:47.794276428 +0000 UTC m=+1878.501862727" Oct 31 13:03:48.573748 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:48.573715 5693 patch_prober.go:29] interesting pod/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 31 13:03:48.574119 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:48.573765 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" podUID="ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 31 13:03:48.708679 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:48.708564 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c347ff-86d57944cc-sf94h" Oct 31 13:03:48.710993 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:03:48.710952 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:03:48.811789 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:48.811755 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-758f48-c94487788-czlnw" Oct 31 13:03:49.573999 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:49.573966 5693 patch_prober.go:29] interesting pod/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 31 13:03:49.574378 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:49.574018 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" podUID="ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 31 13:03:50.578053 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:50.577968 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" Oct 31 13:03:50.578620 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:50.578371 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" Oct 31 13:03:50.579093 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:50.579075 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" Oct 31 13:03:57.167676 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:57.166325 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf"] Oct 31 13:03:57.206924 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:57.206881 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf"] Oct 31 13:03:57.207244 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:57.207223 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf" Oct 31 13:03:57.214420 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:57.214380 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-fa2079-serving-cert\"" Oct 31 13:03:57.241499 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:57.241426 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a53ba38a-cc4c-4bd2-8241-f2772ac54e85-maas-api-tls\") pod \"maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf\" (UID: \"a53ba38a-cc4c-4bd2-8241-f2772ac54e85\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf" Oct 31 13:03:57.241692 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:57.241550 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a53ba38a-cc4c-4bd2-8241-f2772ac54e85-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf\" (UID: \"a53ba38a-cc4c-4bd2-8241-f2772ac54e85\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf" Oct 31 13:03:57.241692 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:57.241618 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a53ba38a-cc4c-4bd2-8241-f2772ac54e85-metrics-certs\") pod \"maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf\" (UID: \"a53ba38a-cc4c-4bd2-8241-f2772ac54e85\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf" Oct 31 13:03:57.241823 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:57.241737 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrq2d\" (UniqueName: \"kubernetes.io/projected/a53ba38a-cc4c-4bd2-8241-f2772ac54e85-kube-api-access-nrq2d\") pod \"maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf\" (UID: \"a53ba38a-cc4c-4bd2-8241-f2772ac54e85\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf" Oct 31 13:03:57.363036 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:57.362995 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a53ba38a-cc4c-4bd2-8241-f2772ac54e85-maas-api-tls\") pod \"maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf\" (UID: \"a53ba38a-cc4c-4bd2-8241-f2772ac54e85\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf" Oct 31 13:03:57.363408 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:57.363383 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a53ba38a-cc4c-4bd2-8241-f2772ac54e85-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf\" (UID: \"a53ba38a-cc4c-4bd2-8241-f2772ac54e85\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf" Oct 31 13:03:57.365575 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:57.364142 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a53ba38a-cc4c-4bd2-8241-f2772ac54e85-metrics-certs\") pod \"maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf\" (UID: \"a53ba38a-cc4c-4bd2-8241-f2772ac54e85\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf" Oct 31 13:03:57.365905 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:57.365882 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrq2d\" (UniqueName: \"kubernetes.io/projected/a53ba38a-cc4c-4bd2-8241-f2772ac54e85-kube-api-access-nrq2d\") pod \"maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf\" (UID: \"a53ba38a-cc4c-4bd2-8241-f2772ac54e85\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf" Oct 31 13:03:57.367289 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:57.367264 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a53ba38a-cc4c-4bd2-8241-f2772ac54e85-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf\" (UID: \"a53ba38a-cc4c-4bd2-8241-f2772ac54e85\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf" Oct 31 13:03:57.369160 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:57.368549 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a53ba38a-cc4c-4bd2-8241-f2772ac54e85-maas-api-tls\") pod \"maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf\" (UID: \"a53ba38a-cc4c-4bd2-8241-f2772ac54e85\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf" Oct 31 13:03:57.387318 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:57.385749 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a53ba38a-cc4c-4bd2-8241-f2772ac54e85-metrics-certs\") pod \"maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf\" (UID: \"a53ba38a-cc4c-4bd2-8241-f2772ac54e85\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf" Oct 31 13:03:57.448501 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:57.448422 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrq2d\" (UniqueName: \"kubernetes.io/projected/a53ba38a-cc4c-4bd2-8241-f2772ac54e85-kube-api-access-nrq2d\") pod \"maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf\" (UID: \"a53ba38a-cc4c-4bd2-8241-f2772ac54e85\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf" Oct 31 13:03:57.545864 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:57.545832 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf" Oct 31 13:03:57.870508 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:03:57.870469 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53ba38a_cc4c_4bd2_8241_f2772ac54e85.slice/crio-7e665779d39a42e36b9811ecb4d6509d273ca805576e47f48acfcf0a7c274889 WatchSource:0}: Error finding container 7e665779d39a42e36b9811ecb4d6509d273ca805576e47f48acfcf0a7c274889: Status 404 returned error can't find the container with id 7e665779d39a42e36b9811ecb4d6509d273ca805576e47f48acfcf0a7c274889 Oct 31 13:03:57.871388 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:57.871366 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf"] Oct 31 13:03:57.975879 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:57.975843 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf" event={"ID":"a53ba38a-cc4c-4bd2-8241-f2772ac54e85","Type":"ContainerStarted","Data":"7e665779d39a42e36b9811ecb4d6509d273ca805576e47f48acfcf0a7c274889"} Oct 31 13:03:58.983712 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:58.983670 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf" event={"ID":"a53ba38a-cc4c-4bd2-8241-f2772ac54e85","Type":"ContainerStarted","Data":"33da5d1292e0c7325f96e159f6c29e63c92de8a30a3d1497569ca89b70e4a634"} Oct 31 13:03:58.984150 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:58.983898 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf" Oct 31 13:03:59.058409 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:03:59.058357 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf" podStartSLOduration=1.592666925 podStartE2EDuration="2.058340749s" totalImagesPullingTime="465.673824ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:59.054057877 +0000 UTC m=+1889.761644172" watchObservedRunningTime="2026-10-31 13:03:59.058340749 +0000 UTC m=+1889.765927093" Oct 31 13:04:04.999122 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:04.999088 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf" Oct 31 13:04:05.368532 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:05.368453 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e4420f-667564665f-vjmfn"] Oct 31 13:04:05.374283 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:05.374262 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e4420f-667564665f-vjmfn" Oct 31 13:04:05.388481 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:05.388459 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e4420f-667564665f-vjmfn"] Oct 31 13:04:05.398880 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:05.398856 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-e4420f\"" Oct 31 13:04:05.399016 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:05.398977 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-e4420f-dockercfg-cg562\"" Oct 31 13:04:05.482524 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:05.475627 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsg8d\" (UniqueName: \"kubernetes.io/projected/d9389331-8d19-426f-ba40-edc6324d7463-kube-api-access-gsg8d\") pod \"payload-pre-processing-e2e-praxis-b-e4420f-667564665f-vjmfn\" (UID: \"d9389331-8d19-426f-ba40-edc6324d7463\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e4420f-667564665f-vjmfn" Oct 31 13:04:05.482524 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:05.475795 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9389331-8d19-426f-ba40-edc6324d7463-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e4420f-667564665f-vjmfn\" (UID: \"d9389331-8d19-426f-ba40-edc6324d7463\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e4420f-667564665f-vjmfn" Oct 31 13:04:05.579665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:05.577825 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsg8d\" (UniqueName: \"kubernetes.io/projected/d9389331-8d19-426f-ba40-edc6324d7463-kube-api-access-gsg8d\") pod \"payload-pre-processing-e2e-praxis-b-e4420f-667564665f-vjmfn\" (UID: \"d9389331-8d19-426f-ba40-edc6324d7463\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e4420f-667564665f-vjmfn" Oct 31 13:04:05.579665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:05.577949 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9389331-8d19-426f-ba40-edc6324d7463-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e4420f-667564665f-vjmfn\" (UID: \"d9389331-8d19-426f-ba40-edc6324d7463\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e4420f-667564665f-vjmfn" Oct 31 13:04:05.579665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:05.579078 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9389331-8d19-426f-ba40-edc6324d7463-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e4420f-667564665f-vjmfn\" (UID: \"d9389331-8d19-426f-ba40-edc6324d7463\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e4420f-667564665f-vjmfn" Oct 31 13:04:05.675670 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:05.663310 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsg8d\" (UniqueName: \"kubernetes.io/projected/d9389331-8d19-426f-ba40-edc6324d7463-kube-api-access-gsg8d\") pod \"payload-pre-processing-e2e-praxis-b-e4420f-667564665f-vjmfn\" (UID: \"d9389331-8d19-426f-ba40-edc6324d7463\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e4420f-667564665f-vjmfn" Oct 31 13:04:05.702879 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:05.702845 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e4420f-667564665f-vjmfn" Oct 31 13:04:06.006516 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:06.006487 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e4420f-667564665f-vjmfn"] Oct 31 13:04:06.007670 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:04:06.007616 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9389331_8d19_426f_ba40_edc6324d7463.slice/crio-1a33324579c8da5393aad398bcbe62f3c8148e6598a36b9df20bd76eb8aa9eea WatchSource:0}: Error finding container 1a33324579c8da5393aad398bcbe62f3c8148e6598a36b9df20bd76eb8aa9eea: Status 404 returned error can't find the container with id 1a33324579c8da5393aad398bcbe62f3c8148e6598a36b9df20bd76eb8aa9eea Oct 31 13:04:06.034059 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:06.034029 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e4420f-667564665f-vjmfn" event={"ID":"d9389331-8d19-426f-ba40-edc6324d7463","Type":"ContainerStarted","Data":"1a33324579c8da5393aad398bcbe62f3c8148e6598a36b9df20bd76eb8aa9eea"} Oct 31 13:04:07.042013 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:07.041977 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e4420f-667564665f-vjmfn" event={"ID":"d9389331-8d19-426f-ba40-edc6324d7463","Type":"ContainerStarted","Data":"283a9761a31b1abf2e295b9e7ee154c7a8d4c1b22ff83eade706a3deed3708e5"} Oct 31 13:04:07.042471 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:07.042127 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e4420f-667564665f-vjmfn" Oct 31 13:04:07.065730 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:07.065680 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e4420f-667564665f-vjmfn" podStartSLOduration=2.065635148 podStartE2EDuration="2.065635148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:04:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:04:07.064393095 +0000 UTC m=+1897.771979402" watchObservedRunningTime="2026-10-31 13:04:07.065635148 +0000 UTC m=+1897.773221443" Oct 31 13:04:10.553557 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:10.553472 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c347ff-86d57944cc-sf94h"] Oct 31 13:04:15.557667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:15.556721 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c347ff-86d57944cc-sf94h" podUID="5c5d33c5-5bd9-4386-a5d0-f905d505dc9a" containerName="payload-processing" containerID="cri-o://12aa97db67f73eacf3a258cd934b46fe4ef53a726f666ec16382d68c7e89991e" gracePeriod=25 Oct 31 13:04:15.928454 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:15.928426 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c347ff-86d57944cc-sf94h" Oct 31 13:04:16.048537 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:16.048501 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c5d33c5-5bd9-4386-a5d0-f905d505dc9a-kube-api-access-g2fwm\" (UniqueName: \"kubernetes.io/projected/5c5d33c5-5bd9-4386-a5d0-f905d505dc9a-kube-api-access-g2fwm\") pod \"5c5d33c5-5bd9-4386-a5d0-f905d505dc9a\" (UID: \"5c5d33c5-5bd9-4386-a5d0-f905d505dc9a\") " Oct 31 13:04:16.048744 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:16.048622 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c5d33c5-5bd9-4386-a5d0-f905d505dc9a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c5d33c5-5bd9-4386-a5d0-f905d505dc9a-plugins-config-volume\") pod \"5c5d33c5-5bd9-4386-a5d0-f905d505dc9a\" (UID: \"5c5d33c5-5bd9-4386-a5d0-f905d505dc9a\") " Oct 31 13:04:16.050218 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:16.050177 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c5d33c5-5bd9-4386-a5d0-f905d505dc9a-plugins-config-volume" pod "5c5d33c5-5bd9-4386-a5d0-f905d505dc9a" (UID: "5c5d33c5-5bd9-4386-a5d0-f905d505dc9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:04:16.051356 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:16.051327 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c5d33c5-5bd9-4386-a5d0-f905d505dc9a-kube-api-access-g2fwm" pod "5c5d33c5-5bd9-4386-a5d0-f905d505dc9a" (UID: "5c5d33c5-5bd9-4386-a5d0-f905d505dc9a"). InnerVolumeSpecName "kube-api-access-g2fwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:04:16.128942 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:16.128906 5693 generic.go:356] "Generic (PLEG): container finished" podID="5c5d33c5-5bd9-4386-a5d0-f905d505dc9a" containerID="12aa97db67f73eacf3a258cd934b46fe4ef53a726f666ec16382d68c7e89991e" exitCode=0 Oct 31 13:04:16.129133 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:16.128965 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c347ff-86d57944cc-sf94h" Oct 31 13:04:16.129133 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:16.129023 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c347ff-86d57944cc-sf94h" event={"ID":"5c5d33c5-5bd9-4386-a5d0-f905d505dc9a","Type":"ContainerDied","Data":"12aa97db67f73eacf3a258cd934b46fe4ef53a726f666ec16382d68c7e89991e"} Oct 31 13:04:16.129133 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:16.129080 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c347ff-86d57944cc-sf94h" event={"ID":"5c5d33c5-5bd9-4386-a5d0-f905d505dc9a","Type":"ContainerDied","Data":"c8a86249f7e1535a3189b7702e00cbbbb08d70e70bdace3c89de2fa5f9e8cd7e"} Oct 31 13:04:16.129133 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:16.129104 5693 scope.go:122] "RemoveContainer" containerID="12aa97db67f73eacf3a258cd934b46fe4ef53a726f666ec16382d68c7e89991e" Oct 31 13:04:16.147084 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:16.147022 5693 scope.go:122] "RemoveContainer" containerID="12aa97db67f73eacf3a258cd934b46fe4ef53a726f666ec16382d68c7e89991e" Oct 31 13:04:16.147527 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:04:16.147494 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12aa97db67f73eacf3a258cd934b46fe4ef53a726f666ec16382d68c7e89991e\": container with ID starting with 12aa97db67f73eacf3a258cd934b46fe4ef53a726f666ec16382d68c7e89991e not found: ID does not exist" containerID="12aa97db67f73eacf3a258cd934b46fe4ef53a726f666ec16382d68c7e89991e" Oct 31 13:04:16.147681 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:16.147537 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12aa97db67f73eacf3a258cd934b46fe4ef53a726f666ec16382d68c7e89991e"} err="failed to get container status \"12aa97db67f73eacf3a258cd934b46fe4ef53a726f666ec16382d68c7e89991e\": rpc error: code = NotFound desc = could not find container \"12aa97db67f73eacf3a258cd934b46fe4ef53a726f666ec16382d68c7e89991e\": container with ID starting with 12aa97db67f73eacf3a258cd934b46fe4ef53a726f666ec16382d68c7e89991e not found: ID does not exist" Oct 31 13:04:16.150570 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:16.150549 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2fwm\" (UniqueName: \"kubernetes.io/projected/5c5d33c5-5bd9-4386-a5d0-f905d505dc9a-kube-api-access-g2fwm\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:16.150625 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:16.150573 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c5d33c5-5bd9-4386-a5d0-f905d505dc9a-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:16.161004 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:16.160968 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c347ff-86d57944cc-sf94h"] Oct 31 13:04:16.171937 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:16.171896 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c347ff-86d57944cc-sf94h"] Oct 31 13:04:17.409956 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:17.409919 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c5d33c5-5bd9-4386-a5d0-f905d505dc9a" path="/var/lib/kubelet/pods/5c5d33c5-5bd9-4386-a5d0-f905d505dc9a/volumes" Oct 31 13:04:18.052885 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:18.052853 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e4420f-667564665f-vjmfn" Oct 31 13:04:18.747708 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:04:18.745797 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:04:20.310138 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:20.294935 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-49bffb-7678b64f8f-zdp9v"] Oct 31 13:04:20.316429 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:20.312684 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s"] Oct 31 13:04:20.316429 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:20.312924 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" podUID="99c723db-b336-48a6-a52b-df5174b4ab3d" containerName="istio-proxy" containerID="cri-o://f9d68877ba05896005c9b4fba27ef2270e2203345979dcc1614586007df04487" gracePeriod=30 Oct 31 13:04:20.326669 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:20.323475 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs"] Oct 31 13:04:20.326669 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:20.323766 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" podUID="5780a4a9-4b92-4663-8211-d831e6cfad35" containerName="istio-proxy" containerID="cri-o://c9b9d5baac20d34656141c2b23e95d059ddda8e5bbe706f8e10ebe8617e260b9" gracePeriod=30 Oct 31 13:04:20.335660 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:20.330440 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-49bffb-74bccc9cbc-6mlwp"] Oct 31 13:04:21.211203 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:21.211164 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-7b5b64f855-rpv5d"] Oct 31 13:04:21.211843 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:21.211828 5693 cpu_manager.go:414] "containerMap: removing container" podUID="5c5d33c5-5bd9-4386-a5d0-f905d505dc9a" containerName="payload-processing" Oct 31 13:04:21.211906 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:21.211844 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c5d33c5-5bd9-4386-a5d0-f905d505dc9a" containerName="payload-processing" Oct 31 13:04:21.211989 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:21.211979 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c5d33c5-5bd9-4386-a5d0-f905d505dc9a" containerName="payload-processing" Oct 31 13:04:21.215542 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:21.215524 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-7b5b64f855-rpv5d" Oct 31 13:04:21.222906 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:21.222879 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c347ff\"" Oct 31 13:04:21.223344 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:21.223078 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c347ff-dockercfg-j899d\"" Oct 31 13:04:21.228730 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:21.225152 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3894d4a0-ca36-4b76-8a2e-e1120d621cc4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c347ff-7b5b64f855-rpv5d\" (UID: \"3894d4a0-ca36-4b76-8a2e-e1120d621cc4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-7b5b64f855-rpv5d" Oct 31 13:04:21.228730 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:21.225305 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv7v7\" (UniqueName: \"kubernetes.io/projected/3894d4a0-ca36-4b76-8a2e-e1120d621cc4-kube-api-access-qv7v7\") pod \"payload-pre-processing-e2e-ipp-b-c347ff-7b5b64f855-rpv5d\" (UID: \"3894d4a0-ca36-4b76-8a2e-e1120d621cc4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-7b5b64f855-rpv5d" Oct 31 13:04:21.228730 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:21.228701 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-7b5b64f855-rpv5d"] Oct 31 13:04:21.334671 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:21.333264 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv7v7\" (UniqueName: \"kubernetes.io/projected/3894d4a0-ca36-4b76-8a2e-e1120d621cc4-kube-api-access-qv7v7\") pod \"payload-pre-processing-e2e-ipp-b-c347ff-7b5b64f855-rpv5d\" (UID: \"3894d4a0-ca36-4b76-8a2e-e1120d621cc4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-7b5b64f855-rpv5d" Oct 31 13:04:21.334671 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:21.333443 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3894d4a0-ca36-4b76-8a2e-e1120d621cc4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c347ff-7b5b64f855-rpv5d\" (UID: \"3894d4a0-ca36-4b76-8a2e-e1120d621cc4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-7b5b64f855-rpv5d" Oct 31 13:04:21.380664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:21.369524 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3894d4a0-ca36-4b76-8a2e-e1120d621cc4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c347ff-7b5b64f855-rpv5d\" (UID: \"3894d4a0-ca36-4b76-8a2e-e1120d621cc4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-7b5b64f855-rpv5d" Oct 31 13:04:21.380664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:21.374704 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c347ff-6c968cc48b-dxhqv"] Oct 31 13:04:21.396737 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:21.385901 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv7v7\" (UniqueName: \"kubernetes.io/projected/3894d4a0-ca36-4b76-8a2e-e1120d621cc4-kube-api-access-qv7v7\") pod \"payload-pre-processing-e2e-ipp-b-c347ff-7b5b64f855-rpv5d\" (UID: \"3894d4a0-ca36-4b76-8a2e-e1120d621cc4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-7b5b64f855-rpv5d" Oct 31 13:04:21.517839 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:21.479855 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c347ff-6c968cc48b-dxhqv" Oct 31 13:04:21.573473 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:21.563130 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-7b5b64f855-rpv5d" Oct 31 13:04:21.594074 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:21.594036 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/237288a4-fc4b-4333-bab3-0898a34fc166-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c347ff-6c968cc48b-dxhqv\" (UID: \"237288a4-fc4b-4333-bab3-0898a34fc166\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c347ff-6c968cc48b-dxhqv" Oct 31 13:04:21.594379 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:21.594354 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzvm5\" (UniqueName: \"kubernetes.io/projected/237288a4-fc4b-4333-bab3-0898a34fc166-kube-api-access-bzvm5\") pod \"payload-processing-e2e-ipp-b-c347ff-6c968cc48b-dxhqv\" (UID: \"237288a4-fc4b-4333-bab3-0898a34fc166\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c347ff-6c968cc48b-dxhqv" Oct 31 13:04:21.602694 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:21.598983 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c347ff-6c968cc48b-dxhqv"] Oct 31 13:04:21.713389 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:21.712926 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/237288a4-fc4b-4333-bab3-0898a34fc166-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c347ff-6c968cc48b-dxhqv\" (UID: \"237288a4-fc4b-4333-bab3-0898a34fc166\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c347ff-6c968cc48b-dxhqv" Oct 31 13:04:21.713389 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:21.712995 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzvm5\" (UniqueName: \"kubernetes.io/projected/237288a4-fc4b-4333-bab3-0898a34fc166-kube-api-access-bzvm5\") pod \"payload-processing-e2e-ipp-b-c347ff-6c968cc48b-dxhqv\" (UID: \"237288a4-fc4b-4333-bab3-0898a34fc166\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c347ff-6c968cc48b-dxhqv" Oct 31 13:04:21.713880 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:21.713853 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/237288a4-fc4b-4333-bab3-0898a34fc166-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c347ff-6c968cc48b-dxhqv\" (UID: \"237288a4-fc4b-4333-bab3-0898a34fc166\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c347ff-6c968cc48b-dxhqv" Oct 31 13:04:21.725568 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:21.725291 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzvm5\" (UniqueName: \"kubernetes.io/projected/237288a4-fc4b-4333-bab3-0898a34fc166-kube-api-access-bzvm5\") pod \"payload-processing-e2e-ipp-b-c347ff-6c968cc48b-dxhqv\" (UID: \"237288a4-fc4b-4333-bab3-0898a34fc166\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c347ff-6c968cc48b-dxhqv" Oct 31 13:04:21.892283 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:21.892252 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-7b5b64f855-rpv5d"] Oct 31 13:04:21.892891 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:04:21.892862 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3894d4a0_ca36_4b76_8a2e_e1120d621cc4.slice/crio-e4a5f52a3e34ece2e837cb0d638487be529655f188d1fc7473ac8fe0318d889a WatchSource:0}: Error finding container e4a5f52a3e34ece2e837cb0d638487be529655f188d1fc7473ac8fe0318d889a: Status 404 returned error can't find the container with id e4a5f52a3e34ece2e837cb0d638487be529655f188d1fc7473ac8fe0318d889a Oct 31 13:04:21.904710 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:21.904414 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c347ff-6c968cc48b-dxhqv" Oct 31 13:04:22.214942 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:22.214902 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-7b5b64f855-rpv5d" event={"ID":"3894d4a0-ca36-4b76-8a2e-e1120d621cc4","Type":"ContainerStarted","Data":"8cd94251ecc65e40b969273b03983233094242121bec999f15ca7d29ecf8af49"} Oct 31 13:04:22.215126 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:22.214968 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-7b5b64f855-rpv5d" event={"ID":"3894d4a0-ca36-4b76-8a2e-e1120d621cc4","Type":"ContainerStarted","Data":"e4a5f52a3e34ece2e837cb0d638487be529655f188d1fc7473ac8fe0318d889a"} Oct 31 13:04:22.215126 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:22.214999 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-7b5b64f855-rpv5d" Oct 31 13:04:22.240414 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:22.239262 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-7b5b64f855-rpv5d" podStartSLOduration=1.239237815 podStartE2EDuration="1.239237815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:04:22.237815135 +0000 UTC m=+1912.945401442" watchObservedRunningTime="2026-10-31 13:04:22.239237815 +0000 UTC m=+1912.946824103" Oct 31 13:04:22.245073 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:04:22.245046 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod237288a4_fc4b_4333_bab3_0898a34fc166.slice/crio-aba89d6dd338b7bb88c260dcddb2f7fd6bff8dcdc08622083389ba6010ba2676 WatchSource:0}: Error finding container aba89d6dd338b7bb88c260dcddb2f7fd6bff8dcdc08622083389ba6010ba2676: Status 404 returned error can't find the container with id aba89d6dd338b7bb88c260dcddb2f7fd6bff8dcdc08622083389ba6010ba2676 Oct 31 13:04:22.247609 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:22.247585 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c347ff-6c968cc48b-dxhqv"] Oct 31 13:04:23.226429 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:23.226017 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c347ff-6c968cc48b-dxhqv" event={"ID":"237288a4-fc4b-4333-bab3-0898a34fc166","Type":"ContainerStarted","Data":"4bc25eb31926e7cfa0954e884cddaf4cefa501ec3c1e3d1d36b58edc1c5a94c9"} Oct 31 13:04:23.226429 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:23.226092 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c347ff-6c968cc48b-dxhqv" event={"ID":"237288a4-fc4b-4333-bab3-0898a34fc166","Type":"ContainerStarted","Data":"aba89d6dd338b7bb88c260dcddb2f7fd6bff8dcdc08622083389ba6010ba2676"} Oct 31 13:04:23.226429 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:23.226371 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c347ff-6c968cc48b-dxhqv" Oct 31 13:04:23.250921 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:23.250857 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c347ff-6c968cc48b-dxhqv" podStartSLOduration=2.250837073 podStartE2EDuration="2.250837073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:04:23.247443023 +0000 UTC m=+1913.955029322" watchObservedRunningTime="2026-10-31 13:04:23.250837073 +0000 UTC m=+1913.958423368" Oct 31 13:04:25.295647 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:25.295595 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-49bffb-7678b64f8f-zdp9v" podUID="084961ae-b1f7-4f97-aca7-21e9cc5ae882" containerName="payload-processing" containerID="cri-o://43ff6dabfbbc97921a16be1f9337dd09c242f324b39f5f498d7bdf17ffafa72e" gracePeriod=25 Oct 31 13:04:25.331109 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:25.331070 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-49bffb-74bccc9cbc-6mlwp" podUID="39047437-bef9-4407-b5e6-2781f6811546" containerName="payload-pre-processing" containerID="cri-o://694d53bae5e1409ceaeebd0b031b0cae66a9c1d57cba4ac86138305a3cf54925" gracePeriod=25 Oct 31 13:04:25.853015 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:25.852947 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" Oct 31 13:04:25.932366 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:25.932171 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99c723db-b336-48a6-a52b-df5174b4ab3d-istio-token\" (UniqueName: \"kubernetes.io/projected/99c723db-b336-48a6-a52b-df5174b4ab3d-istio-token\") pod \"99c723db-b336-48a6-a52b-df5174b4ab3d\" (UID: \"99c723db-b336-48a6-a52b-df5174b4ab3d\") " Oct 31 13:04:25.932366 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:25.932222 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c723db-b336-48a6-a52b-df5174b4ab3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c723db-b336-48a6-a52b-df5174b4ab3d-workload-socket\") pod \"99c723db-b336-48a6-a52b-df5174b4ab3d\" (UID: \"99c723db-b336-48a6-a52b-df5174b4ab3d\") " Oct 31 13:04:25.932366 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:25.932275 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c723db-b336-48a6-a52b-df5174b4ab3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c723db-b336-48a6-a52b-df5174b4ab3d-credential-socket\") pod \"99c723db-b336-48a6-a52b-df5174b4ab3d\" (UID: \"99c723db-b336-48a6-a52b-df5174b4ab3d\") " Oct 31 13:04:25.932366 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:25.932331 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c723db-b336-48a6-a52b-df5174b4ab3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99c723db-b336-48a6-a52b-df5174b4ab3d-istio-data\") pod \"99c723db-b336-48a6-a52b-df5174b4ab3d\" (UID: \"99c723db-b336-48a6-a52b-df5174b4ab3d\") " Oct 31 13:04:25.932612 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:25.932427 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99c723db-b336-48a6-a52b-df5174b4ab3d-kube-api-access-z7jpc\" (UniqueName: \"kubernetes.io/projected/99c723db-b336-48a6-a52b-df5174b4ab3d-kube-api-access-z7jpc\") pod \"99c723db-b336-48a6-a52b-df5174b4ab3d\" (UID: \"99c723db-b336-48a6-a52b-df5174b4ab3d\") " Oct 31 13:04:25.932612 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:25.932475 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c723db-b336-48a6-a52b-df5174b4ab3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99c723db-b336-48a6-a52b-df5174b4ab3d-workload-certs\") pod \"99c723db-b336-48a6-a52b-df5174b4ab3d\" (UID: \"99c723db-b336-48a6-a52b-df5174b4ab3d\") " Oct 31 13:04:25.932612 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:25.932512 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99c723db-b336-48a6-a52b-df5174b4ab3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99c723db-b336-48a6-a52b-df5174b4ab3d-istiod-ca-cert\") pod \"99c723db-b336-48a6-a52b-df5174b4ab3d\" (UID: \"99c723db-b336-48a6-a52b-df5174b4ab3d\") " Oct 31 13:04:25.932612 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:25.932545 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c723db-b336-48a6-a52b-df5174b4ab3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99c723db-b336-48a6-a52b-df5174b4ab3d-istio-envoy\") pod \"99c723db-b336-48a6-a52b-df5174b4ab3d\" (UID: \"99c723db-b336-48a6-a52b-df5174b4ab3d\") " Oct 31 13:04:25.932844 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:25.932634 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99c723db-b336-48a6-a52b-df5174b4ab3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99c723db-b336-48a6-a52b-df5174b4ab3d-istio-podinfo\") pod \"99c723db-b336-48a6-a52b-df5174b4ab3d\" (UID: \"99c723db-b336-48a6-a52b-df5174b4ab3d\") " Oct 31 13:04:25.934169 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:25.934070 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c723db-b336-48a6-a52b-df5174b4ab3d-istio-data" pod "99c723db-b336-48a6-a52b-df5174b4ab3d" (UID: "99c723db-b336-48a6-a52b-df5174b4ab3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:04:25.934169 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:25.934084 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c723db-b336-48a6-a52b-df5174b4ab3d-workload-socket" pod "99c723db-b336-48a6-a52b-df5174b4ab3d" (UID: "99c723db-b336-48a6-a52b-df5174b4ab3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:04:25.934429 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:25.934392 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c723db-b336-48a6-a52b-df5174b4ab3d-credential-socket" pod "99c723db-b336-48a6-a52b-df5174b4ab3d" (UID: "99c723db-b336-48a6-a52b-df5174b4ab3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:04:25.951000 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:25.935885 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99c723db-b336-48a6-a52b-df5174b4ab3d-istiod-ca-cert" pod "99c723db-b336-48a6-a52b-df5174b4ab3d" (UID: "99c723db-b336-48a6-a52b-df5174b4ab3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:04:25.951000 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:25.936340 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c723db-b336-48a6-a52b-df5174b4ab3d-workload-certs" pod "99c723db-b336-48a6-a52b-df5174b4ab3d" (UID: "99c723db-b336-48a6-a52b-df5174b4ab3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:04:25.951000 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:25.941349 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99c723db-b336-48a6-a52b-df5174b4ab3d-kube-api-access-z7jpc" pod "99c723db-b336-48a6-a52b-df5174b4ab3d" (UID: "99c723db-b336-48a6-a52b-df5174b4ab3d"). InnerVolumeSpecName "kube-api-access-z7jpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:04:25.951000 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:25.944961 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99c723db-b336-48a6-a52b-df5174b4ab3d-istio-podinfo" pod "99c723db-b336-48a6-a52b-df5174b4ab3d" (UID: "99c723db-b336-48a6-a52b-df5174b4ab3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:04:25.951000 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:25.945102 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c723db-b336-48a6-a52b-df5174b4ab3d-istio-envoy" pod "99c723db-b336-48a6-a52b-df5174b4ab3d" (UID: "99c723db-b336-48a6-a52b-df5174b4ab3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:04:25.951000 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:25.946910 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99c723db-b336-48a6-a52b-df5174b4ab3d-istio-token" pod "99c723db-b336-48a6-a52b-df5174b4ab3d" (UID: "99c723db-b336-48a6-a52b-df5174b4ab3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:04:25.967161 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:25.966777 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-49bffb-74bccc9cbc-6mlwp" Oct 31 13:04:26.012379 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.012352 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-49bffb-7678b64f8f-zdp9v" Oct 31 13:04:26.035948 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.035912 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39047437-bef9-4407-b5e6-2781f6811546-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39047437-bef9-4407-b5e6-2781f6811546-plugins-config-volume\") pod \"39047437-bef9-4407-b5e6-2781f6811546\" (UID: \"39047437-bef9-4407-b5e6-2781f6811546\") " Oct 31 13:04:26.036219 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.036188 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39047437-bef9-4407-b5e6-2781f6811546-kube-api-access-ldvrr\" (UniqueName: \"kubernetes.io/projected/39047437-bef9-4407-b5e6-2781f6811546-kube-api-access-ldvrr\") pod \"39047437-bef9-4407-b5e6-2781f6811546\" (UID: \"39047437-bef9-4407-b5e6-2781f6811546\") " Oct 31 13:04:26.036845 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.036821 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99c723db-b336-48a6-a52b-df5174b4ab3d-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:26.037133 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.036849 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c723db-b336-48a6-a52b-df5174b4ab3d-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:26.037133 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.036866 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c723db-b336-48a6-a52b-df5174b4ab3d-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:26.037133 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.036880 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99c723db-b336-48a6-a52b-df5174b4ab3d-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:26.037133 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.036894 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7jpc\" (UniqueName: \"kubernetes.io/projected/99c723db-b336-48a6-a52b-df5174b4ab3d-kube-api-access-z7jpc\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:26.037133 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.036910 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99c723db-b336-48a6-a52b-df5174b4ab3d-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:26.037133 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.036924 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99c723db-b336-48a6-a52b-df5174b4ab3d-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:26.037133 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.036938 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99c723db-b336-48a6-a52b-df5174b4ab3d-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:26.037133 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.036953 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99c723db-b336-48a6-a52b-df5174b4ab3d-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:26.039766 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.039740 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39047437-bef9-4407-b5e6-2781f6811546-plugins-config-volume" pod "39047437-bef9-4407-b5e6-2781f6811546" (UID: "39047437-bef9-4407-b5e6-2781f6811546"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:04:26.042825 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.042796 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" Oct 31 13:04:26.046768 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.046734 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39047437-bef9-4407-b5e6-2781f6811546-kube-api-access-ldvrr" pod "39047437-bef9-4407-b5e6-2781f6811546" (UID: "39047437-bef9-4407-b5e6-2781f6811546"). InnerVolumeSpecName "kube-api-access-ldvrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:04:26.138340 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.138305 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/084961ae-b1f7-4f97-aca7-21e9cc5ae882-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/084961ae-b1f7-4f97-aca7-21e9cc5ae882-plugins-config-volume\") pod \"084961ae-b1f7-4f97-aca7-21e9cc5ae882\" (UID: \"084961ae-b1f7-4f97-aca7-21e9cc5ae882\") " Oct 31 13:04:26.139611 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.138418 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5780a4a9-4b92-4663-8211-d831e6cfad35-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5780a4a9-4b92-4663-8211-d831e6cfad35-istio-podinfo\") pod \"5780a4a9-4b92-4663-8211-d831e6cfad35\" (UID: \"5780a4a9-4b92-4663-8211-d831e6cfad35\") " Oct 31 13:04:26.139611 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.138452 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5780a4a9-4b92-4663-8211-d831e6cfad35-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5780a4a9-4b92-4663-8211-d831e6cfad35-workload-socket\") pod \"5780a4a9-4b92-4663-8211-d831e6cfad35\" (UID: \"5780a4a9-4b92-4663-8211-d831e6cfad35\") " Oct 31 13:04:26.139611 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.138494 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/084961ae-b1f7-4f97-aca7-21e9cc5ae882-kube-api-access-2tt7b\" (UniqueName: \"kubernetes.io/projected/084961ae-b1f7-4f97-aca7-21e9cc5ae882-kube-api-access-2tt7b\") pod \"084961ae-b1f7-4f97-aca7-21e9cc5ae882\" (UID: \"084961ae-b1f7-4f97-aca7-21e9cc5ae882\") " Oct 31 13:04:26.139611 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.138545 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5780a4a9-4b92-4663-8211-d831e6cfad35-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5780a4a9-4b92-4663-8211-d831e6cfad35-credential-socket\") pod \"5780a4a9-4b92-4663-8211-d831e6cfad35\" (UID: \"5780a4a9-4b92-4663-8211-d831e6cfad35\") " Oct 31 13:04:26.139611 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.138610 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5780a4a9-4b92-4663-8211-d831e6cfad35-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5780a4a9-4b92-4663-8211-d831e6cfad35-istiod-ca-cert\") pod \"5780a4a9-4b92-4663-8211-d831e6cfad35\" (UID: \"5780a4a9-4b92-4663-8211-d831e6cfad35\") " Oct 31 13:04:26.139611 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.138758 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/084961ae-b1f7-4f97-aca7-21e9cc5ae882-plugins-config-volume" pod "084961ae-b1f7-4f97-aca7-21e9cc5ae882" (UID: "084961ae-b1f7-4f97-aca7-21e9cc5ae882"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:04:26.139611 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.138781 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5780a4a9-4b92-4663-8211-d831e6cfad35-workload-socket" pod "5780a4a9-4b92-4663-8211-d831e6cfad35" (UID: "5780a4a9-4b92-4663-8211-d831e6cfad35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:04:26.139611 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.138958 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5780a4a9-4b92-4663-8211-d831e6cfad35-credential-socket" pod "5780a4a9-4b92-4663-8211-d831e6cfad35" (UID: "5780a4a9-4b92-4663-8211-d831e6cfad35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:04:26.139611 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.139132 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5780a4a9-4b92-4663-8211-d831e6cfad35-istiod-ca-cert" pod "5780a4a9-4b92-4663-8211-d831e6cfad35" (UID: "5780a4a9-4b92-4663-8211-d831e6cfad35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:04:26.139611 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.139183 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5780a4a9-4b92-4663-8211-d831e6cfad35-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5780a4a9-4b92-4663-8211-d831e6cfad35-istio-envoy\") pod \"5780a4a9-4b92-4663-8211-d831e6cfad35\" (UID: \"5780a4a9-4b92-4663-8211-d831e6cfad35\") " Oct 31 13:04:26.139611 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.139250 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5780a4a9-4b92-4663-8211-d831e6cfad35-kube-api-access-td26m\" (UniqueName: \"kubernetes.io/projected/5780a4a9-4b92-4663-8211-d831e6cfad35-kube-api-access-td26m\") pod \"5780a4a9-4b92-4663-8211-d831e6cfad35\" (UID: \"5780a4a9-4b92-4663-8211-d831e6cfad35\") " Oct 31 13:04:26.139611 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.139329 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5780a4a9-4b92-4663-8211-d831e6cfad35-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5780a4a9-4b92-4663-8211-d831e6cfad35-workload-certs\") pod \"5780a4a9-4b92-4663-8211-d831e6cfad35\" (UID: \"5780a4a9-4b92-4663-8211-d831e6cfad35\") " Oct 31 13:04:26.139611 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.139389 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5780a4a9-4b92-4663-8211-d831e6cfad35-istio-token\" (UniqueName: \"kubernetes.io/projected/5780a4a9-4b92-4663-8211-d831e6cfad35-istio-token\") pod \"5780a4a9-4b92-4663-8211-d831e6cfad35\" (UID: \"5780a4a9-4b92-4663-8211-d831e6cfad35\") " Oct 31 13:04:26.139611 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.139462 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5780a4a9-4b92-4663-8211-d831e6cfad35-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5780a4a9-4b92-4663-8211-d831e6cfad35-istio-data\") pod \"5780a4a9-4b92-4663-8211-d831e6cfad35\" (UID: \"5780a4a9-4b92-4663-8211-d831e6cfad35\") " Oct 31 13:04:26.139611 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.139601 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5780a4a9-4b92-4663-8211-d831e6cfad35-workload-certs" pod "5780a4a9-4b92-4663-8211-d831e6cfad35" (UID: "5780a4a9-4b92-4663-8211-d831e6cfad35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:04:26.140466 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.139874 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5780a4a9-4b92-4663-8211-d831e6cfad35-istio-data" pod "5780a4a9-4b92-4663-8211-d831e6cfad35" (UID: "5780a4a9-4b92-4663-8211-d831e6cfad35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:04:26.140466 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.140339 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5780a4a9-4b92-4663-8211-d831e6cfad35-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:26.140466 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.140359 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39047437-bef9-4407-b5e6-2781f6811546-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:26.140466 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.140374 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5780a4a9-4b92-4663-8211-d831e6cfad35-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:26.140466 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.140389 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5780a4a9-4b92-4663-8211-d831e6cfad35-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:26.140466 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.140402 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldvrr\" (UniqueName: \"kubernetes.io/projected/39047437-bef9-4407-b5e6-2781f6811546-kube-api-access-ldvrr\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:26.140466 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.140415 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5780a4a9-4b92-4663-8211-d831e6cfad35-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:26.140466 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.140428 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5780a4a9-4b92-4663-8211-d831e6cfad35-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:26.140466 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.140441 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/084961ae-b1f7-4f97-aca7-21e9cc5ae882-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:26.141157 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.141130 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5780a4a9-4b92-4663-8211-d831e6cfad35-istio-podinfo" pod "5780a4a9-4b92-4663-8211-d831e6cfad35" (UID: "5780a4a9-4b92-4663-8211-d831e6cfad35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:04:26.141371 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.141349 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/084961ae-b1f7-4f97-aca7-21e9cc5ae882-kube-api-access-2tt7b" pod "084961ae-b1f7-4f97-aca7-21e9cc5ae882" (UID: "084961ae-b1f7-4f97-aca7-21e9cc5ae882"). InnerVolumeSpecName "kube-api-access-2tt7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:04:26.142014 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.141996 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5780a4a9-4b92-4663-8211-d831e6cfad35-istio-envoy" pod "5780a4a9-4b92-4663-8211-d831e6cfad35" (UID: "5780a4a9-4b92-4663-8211-d831e6cfad35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:04:26.142139 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.142115 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5780a4a9-4b92-4663-8211-d831e6cfad35-kube-api-access-td26m" pod "5780a4a9-4b92-4663-8211-d831e6cfad35" (UID: "5780a4a9-4b92-4663-8211-d831e6cfad35"). InnerVolumeSpecName "kube-api-access-td26m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:04:26.142251 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.142218 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5780a4a9-4b92-4663-8211-d831e6cfad35-istio-token" pod "5780a4a9-4b92-4663-8211-d831e6cfad35" (UID: "5780a4a9-4b92-4663-8211-d831e6cfad35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:04:26.242303 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.242271 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5780a4a9-4b92-4663-8211-d831e6cfad35-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:26.242303 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.242300 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tt7b\" (UniqueName: \"kubernetes.io/projected/084961ae-b1f7-4f97-aca7-21e9cc5ae882-kube-api-access-2tt7b\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:26.242482 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.242311 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5780a4a9-4b92-4663-8211-d831e6cfad35-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:26.242482 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.242320 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td26m\" (UniqueName: \"kubernetes.io/projected/5780a4a9-4b92-4663-8211-d831e6cfad35-kube-api-access-td26m\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:26.242482 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.242328 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5780a4a9-4b92-4663-8211-d831e6cfad35-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:26.260446 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.260416 5693 generic.go:356] "Generic (PLEG): container finished" podID="5780a4a9-4b92-4663-8211-d831e6cfad35" containerID="c9b9d5baac20d34656141c2b23e95d059ddda8e5bbe706f8e10ebe8617e260b9" exitCode=0 Oct 31 13:04:26.260586 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.260494 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" event={"ID":"5780a4a9-4b92-4663-8211-d831e6cfad35","Type":"ContainerDied","Data":"c9b9d5baac20d34656141c2b23e95d059ddda8e5bbe706f8e10ebe8617e260b9"} Oct 31 13:04:26.260586 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.260530 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" Oct 31 13:04:26.260586 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.260558 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs" event={"ID":"5780a4a9-4b92-4663-8211-d831e6cfad35","Type":"ContainerDied","Data":"5b38f39845cf8870c99116b099167825b5b9465519f41cc6901f607fc89cb3eb"} Oct 31 13:04:26.260731 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.260591 5693 scope.go:122] "RemoveContainer" containerID="c9b9d5baac20d34656141c2b23e95d059ddda8e5bbe706f8e10ebe8617e260b9" Oct 31 13:04:26.262370 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.262349 5693 generic.go:356] "Generic (PLEG): container finished" podID="99c723db-b336-48a6-a52b-df5174b4ab3d" containerID="f9d68877ba05896005c9b4fba27ef2270e2203345979dcc1614586007df04487" exitCode=0 Oct 31 13:04:26.262485 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.262397 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" Oct 31 13:04:26.262485 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.262431 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" event={"ID":"99c723db-b336-48a6-a52b-df5174b4ab3d","Type":"ContainerDied","Data":"f9d68877ba05896005c9b4fba27ef2270e2203345979dcc1614586007df04487"} Oct 31 13:04:26.262485 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.262475 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s" event={"ID":"99c723db-b336-48a6-a52b-df5174b4ab3d","Type":"ContainerDied","Data":"a9f8ad8229848efc7cc98ee3092ab432a60cac1b5d383c6cdc1604b11548df6f"} Oct 31 13:04:26.264023 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.264003 5693 generic.go:356] "Generic (PLEG): container finished" podID="084961ae-b1f7-4f97-aca7-21e9cc5ae882" containerID="43ff6dabfbbc97921a16be1f9337dd09c242f324b39f5f498d7bdf17ffafa72e" exitCode=0 Oct 31 13:04:26.264153 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.264135 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-49bffb-7678b64f8f-zdp9v" event={"ID":"084961ae-b1f7-4f97-aca7-21e9cc5ae882","Type":"ContainerDied","Data":"43ff6dabfbbc97921a16be1f9337dd09c242f324b39f5f498d7bdf17ffafa72e"} Oct 31 13:04:26.264225 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.264181 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-49bffb-7678b64f8f-zdp9v" event={"ID":"084961ae-b1f7-4f97-aca7-21e9cc5ae882","Type":"ContainerDied","Data":"85560b811426a8f05907132e478d04414a47c1981c370320710d873c71a56398"} Oct 31 13:04:26.264285 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.264248 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-49bffb-7678b64f8f-zdp9v" Oct 31 13:04:26.268294 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.268270 5693 generic.go:356] "Generic (PLEG): container finished" podID="39047437-bef9-4407-b5e6-2781f6811546" containerID="694d53bae5e1409ceaeebd0b031b0cae66a9c1d57cba4ac86138305a3cf54925" exitCode=0 Oct 31 13:04:26.268418 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.268302 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-49bffb-74bccc9cbc-6mlwp" Oct 31 13:04:26.268418 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.268356 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-49bffb-74bccc9cbc-6mlwp" event={"ID":"39047437-bef9-4407-b5e6-2781f6811546","Type":"ContainerDied","Data":"694d53bae5e1409ceaeebd0b031b0cae66a9c1d57cba4ac86138305a3cf54925"} Oct 31 13:04:26.268418 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.268404 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-49bffb-74bccc9cbc-6mlwp" event={"ID":"39047437-bef9-4407-b5e6-2781f6811546","Type":"ContainerDied","Data":"e16f5d81df5c90e1221b29fcfdfc364fcd739cefae99a8800e90dbb740d6bc2d"} Oct 31 13:04:26.284248 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.284230 5693 scope.go:122] "RemoveContainer" containerID="c9b9d5baac20d34656141c2b23e95d059ddda8e5bbe706f8e10ebe8617e260b9" Oct 31 13:04:26.284548 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:04:26.284531 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9b9d5baac20d34656141c2b23e95d059ddda8e5bbe706f8e10ebe8617e260b9\": container with ID starting with c9b9d5baac20d34656141c2b23e95d059ddda8e5bbe706f8e10ebe8617e260b9 not found: ID does not exist" containerID="c9b9d5baac20d34656141c2b23e95d059ddda8e5bbe706f8e10ebe8617e260b9" Oct 31 13:04:26.284626 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.284562 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9b9d5baac20d34656141c2b23e95d059ddda8e5bbe706f8e10ebe8617e260b9"} err="failed to get container status \"c9b9d5baac20d34656141c2b23e95d059ddda8e5bbe706f8e10ebe8617e260b9\": rpc error: code = NotFound desc = could not find container \"c9b9d5baac20d34656141c2b23e95d059ddda8e5bbe706f8e10ebe8617e260b9\": container with ID starting with c9b9d5baac20d34656141c2b23e95d059ddda8e5bbe706f8e10ebe8617e260b9 not found: ID does not exist" Oct 31 13:04:26.284626 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.284584 5693 scope.go:122] "RemoveContainer" containerID="f9d68877ba05896005c9b4fba27ef2270e2203345979dcc1614586007df04487" Oct 31 13:04:26.295094 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.295077 5693 scope.go:122] "RemoveContainer" containerID="f9d68877ba05896005c9b4fba27ef2270e2203345979dcc1614586007df04487" Oct 31 13:04:26.295375 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:04:26.295357 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9d68877ba05896005c9b4fba27ef2270e2203345979dcc1614586007df04487\": container with ID starting with f9d68877ba05896005c9b4fba27ef2270e2203345979dcc1614586007df04487 not found: ID does not exist" containerID="f9d68877ba05896005c9b4fba27ef2270e2203345979dcc1614586007df04487" Oct 31 13:04:26.295441 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.295388 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9d68877ba05896005c9b4fba27ef2270e2203345979dcc1614586007df04487"} err="failed to get container status \"f9d68877ba05896005c9b4fba27ef2270e2203345979dcc1614586007df04487\": rpc error: code = NotFound desc = could not find container \"f9d68877ba05896005c9b4fba27ef2270e2203345979dcc1614586007df04487\": container with ID starting with f9d68877ba05896005c9b4fba27ef2270e2203345979dcc1614586007df04487 not found: ID does not exist" Oct 31 13:04:26.295441 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.295411 5693 scope.go:122] "RemoveContainer" containerID="43ff6dabfbbc97921a16be1f9337dd09c242f324b39f5f498d7bdf17ffafa72e" Oct 31 13:04:26.301831 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.301808 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-49bffb-7678b64f8f-zdp9v"] Oct 31 13:04:26.305301 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.305281 5693 scope.go:122] "RemoveContainer" containerID="43ff6dabfbbc97921a16be1f9337dd09c242f324b39f5f498d7bdf17ffafa72e" Oct 31 13:04:26.305652 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:04:26.305615 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43ff6dabfbbc97921a16be1f9337dd09c242f324b39f5f498d7bdf17ffafa72e\": container with ID starting with 43ff6dabfbbc97921a16be1f9337dd09c242f324b39f5f498d7bdf17ffafa72e not found: ID does not exist" containerID="43ff6dabfbbc97921a16be1f9337dd09c242f324b39f5f498d7bdf17ffafa72e" Oct 31 13:04:26.305746 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.305658 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43ff6dabfbbc97921a16be1f9337dd09c242f324b39f5f498d7bdf17ffafa72e"} err="failed to get container status \"43ff6dabfbbc97921a16be1f9337dd09c242f324b39f5f498d7bdf17ffafa72e\": rpc error: code = NotFound desc = could not find container \"43ff6dabfbbc97921a16be1f9337dd09c242f324b39f5f498d7bdf17ffafa72e\": container with ID starting with 43ff6dabfbbc97921a16be1f9337dd09c242f324b39f5f498d7bdf17ffafa72e not found: ID does not exist" Oct 31 13:04:26.305746 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.305682 5693 scope.go:122] "RemoveContainer" containerID="694d53bae5e1409ceaeebd0b031b0cae66a9c1d57cba4ac86138305a3cf54925" Oct 31 13:04:26.309834 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.309811 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-49bffb-7678b64f8f-zdp9v"] Oct 31 13:04:26.326022 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.325998 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs"] Oct 31 13:04:26.329786 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.329765 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-hv5zs"] Oct 31 13:04:26.341669 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.341623 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s"] Oct 31 13:04:26.348210 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.348189 5693 scope.go:122] "RemoveContainer" containerID="694d53bae5e1409ceaeebd0b031b0cae66a9c1d57cba4ac86138305a3cf54925" Oct 31 13:04:26.349109 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:04:26.349080 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"694d53bae5e1409ceaeebd0b031b0cae66a9c1d57cba4ac86138305a3cf54925\": container with ID starting with 694d53bae5e1409ceaeebd0b031b0cae66a9c1d57cba4ac86138305a3cf54925 not found: ID does not exist" containerID="694d53bae5e1409ceaeebd0b031b0cae66a9c1d57cba4ac86138305a3cf54925" Oct 31 13:04:26.349228 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.349119 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"694d53bae5e1409ceaeebd0b031b0cae66a9c1d57cba4ac86138305a3cf54925"} err="failed to get container status \"694d53bae5e1409ceaeebd0b031b0cae66a9c1d57cba4ac86138305a3cf54925\": rpc error: code = NotFound desc = could not find container \"694d53bae5e1409ceaeebd0b031b0cae66a9c1d57cba4ac86138305a3cf54925\": container with ID starting with 694d53bae5e1409ceaeebd0b031b0cae66a9c1d57cba4ac86138305a3cf54925 not found: ID does not exist" Oct 31 13:04:26.351725 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.351706 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-6tl2s"] Oct 31 13:04:26.360270 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.360232 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-49bffb-74bccc9cbc-6mlwp"] Oct 31 13:04:26.363475 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:26.363452 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-49bffb-74bccc9cbc-6mlwp"] Oct 31 13:04:27.278378 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:27.278347 5693 generic.go:356] "Generic (PLEG): container finished" podID="0f339605-e98a-4957-88d8-59afe07f2a7b" containerID="4903c0f767d66841eb4600396088e4dabad0bb51de52a827da3d27f1d333a42e" exitCode=0 Oct 31 13:04:27.278586 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:27.278423 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" event={"ID":"0f339605-e98a-4957-88d8-59afe07f2a7b","Type":"ContainerDied","Data":"4903c0f767d66841eb4600396088e4dabad0bb51de52a827da3d27f1d333a42e"} Oct 31 13:04:27.278957 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:27.278939 5693 scope.go:122] "RemoveContainer" containerID="4903c0f767d66841eb4600396088e4dabad0bb51de52a827da3d27f1d333a42e" Oct 31 13:04:27.280902 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:27.280874 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:04:27.280979 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:27.280949 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:04:27.281106 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:27.280992 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:04:27.424543 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:27.424508 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="084961ae-b1f7-4f97-aca7-21e9cc5ae882" path="/var/lib/kubelet/pods/084961ae-b1f7-4f97-aca7-21e9cc5ae882/volumes" Oct 31 13:04:27.425185 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:27.425163 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39047437-bef9-4407-b5e6-2781f6811546" path="/var/lib/kubelet/pods/39047437-bef9-4407-b5e6-2781f6811546/volumes" Oct 31 13:04:27.425682 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:27.425661 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5780a4a9-4b92-4663-8211-d831e6cfad35" path="/var/lib/kubelet/pods/5780a4a9-4b92-4663-8211-d831e6cfad35/volumes" Oct 31 13:04:27.426414 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:27.426399 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99c723db-b336-48a6-a52b-df5174b4ab3d" path="/var/lib/kubelet/pods/99c723db-b336-48a6-a52b-df5174b4ab3d/volumes" Oct 31 13:04:28.296741 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:28.294798 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" event={"ID":"0f339605-e98a-4957-88d8-59afe07f2a7b","Type":"ContainerStarted","Data":"4baa80937fb85e303325a4fcc0623522d35d980dcf5f81bd593a4892e7fa6911"} Oct 31 13:04:28.466172 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:28.466140 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" Oct 31 13:04:28.467336 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:28.467306 5693 patch_prober.go:29] interesting pod/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 31 13:04:28.467476 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:28.467362 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" podUID="0f339605-e98a-4957-88d8-59afe07f2a7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 31 13:04:29.253662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:29.251252 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf"] Oct 31 13:04:29.253662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:29.251494 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" podUID="ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758" containerName="istio-proxy" containerID="cri-o://f35347918fc0ed154824bf23d1ab19e548ff352e8fb09680571cf4fba939104f" gracePeriod=30 Oct 31 13:04:29.481803 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:29.481035 5693 patch_prober.go:29] interesting pod/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 31 13:04:29.481803 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:29.481092 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" podUID="0f339605-e98a-4957-88d8-59afe07f2a7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 31 13:04:30.466101 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:30.466061 5693 patch_prober.go:29] interesting pod/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 31 13:04:30.466281 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:30.466112 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" podUID="0f339605-e98a-4957-88d8-59afe07f2a7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 31 13:04:31.467018 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:31.466983 5693 patch_prober.go:29] interesting pod/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 31 13:04:31.467585 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:31.467552 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" podUID="0f339605-e98a-4957-88d8-59afe07f2a7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 31 13:04:31.647805 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:31.647763 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" Oct 31 13:04:32.466236 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:32.466200 5693 patch_prober.go:29] interesting pod/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 31 13:04:32.466408 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:32.466282 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" podUID="0f339605-e98a-4957-88d8-59afe07f2a7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 31 13:04:33.229365 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:33.229336 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-7b5b64f855-rpv5d" Oct 31 13:04:33.468386 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:33.468356 5693 patch_prober.go:29] interesting pod/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 31 13:04:33.468669 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:33.468629 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" podUID="0f339605-e98a-4957-88d8-59afe07f2a7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 31 13:04:34.237483 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:34.237447 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c347ff-6c968cc48b-dxhqv" Oct 31 13:04:34.432779 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:34.432743 5693 generic.go:356] "Generic (PLEG): container finished" podID="ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758" containerID="f35347918fc0ed154824bf23d1ab19e548ff352e8fb09680571cf4fba939104f" exitCode=0 Oct 31 13:04:34.432971 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:34.432862 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" event={"ID":"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758","Type":"ContainerDied","Data":"f35347918fc0ed154824bf23d1ab19e548ff352e8fb09680571cf4fba939104f"} Oct 31 13:04:34.478002 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:34.477296 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" Oct 31 13:04:34.479377 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:34.479118 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" Oct 31 13:04:34.596860 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:34.596830 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" Oct 31 13:04:34.740978 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:34.740944 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-workload-certs\") pod \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\" (UID: \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\") " Oct 31 13:04:34.741168 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:34.741022 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-credential-socket\") pod \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\" (UID: \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\") " Oct 31 13:04:34.741168 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:34.741135 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-workload-socket\") pod \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\" (UID: \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\") " Oct 31 13:04:34.741294 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:34.741186 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-istio-token\" (UniqueName: \"kubernetes.io/projected/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-istio-token\") pod \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\" (UID: \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\") " Oct 31 13:04:34.741294 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:34.741205 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-workload-certs" pod "ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758" (UID: "ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:04:34.741294 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:34.741239 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-istio-podinfo\") pod \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\" (UID: \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\") " Oct 31 13:04:34.741453 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:34.741329 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-credential-socket" pod "ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758" (UID: "ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:04:34.741453 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:34.741340 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-istiod-ca-cert\") pod \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\" (UID: \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\") " Oct 31 13:04:34.741453 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:34.741388 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-istio-data\") pod \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\" (UID: \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\") " Oct 31 13:04:34.741453 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:34.741430 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-workload-socket" pod "ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758" (UID: "ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:04:34.744299 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:34.742006 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-istio-data" pod "ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758" (UID: "ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:04:34.744299 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:34.742085 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-istio-envoy\") pod \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\" (UID: \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\") " Oct 31 13:04:34.744299 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:34.742164 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-kube-api-access-m25tl\" (UniqueName: \"kubernetes.io/projected/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-kube-api-access-m25tl\") pod \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\" (UID: \"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758\") " Oct 31 13:04:34.744299 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:34.742175 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-istiod-ca-cert" pod "ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758" (UID: "ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:04:34.744299 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:34.743908 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:34.744299 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:34.743932 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:34.744299 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:34.743948 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:34.744299 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:34.743964 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:34.744299 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:34.743985 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:34.745827 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:34.745041 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-kube-api-access-m25tl" pod "ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758" (UID: "ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758"). InnerVolumeSpecName "kube-api-access-m25tl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:04:34.745966 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:34.745899 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-istio-podinfo" pod "ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758" (UID: "ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:04:34.818319 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:34.818226 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-istio-token" pod "ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758" (UID: "ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:04:34.824886 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:34.824735 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-istio-envoy" pod "ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758" (UID: "ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:04:34.845851 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:34.845817 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:34.845851 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:34.845854 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:34.846056 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:34.845870 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:34.846056 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:34.845885 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m25tl\" (UniqueName: \"kubernetes.io/projected/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758-kube-api-access-m25tl\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:35.440845 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:35.440818 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" Oct 31 13:04:35.441288 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:35.440818 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf" event={"ID":"ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758","Type":"ContainerDied","Data":"ef9ce215813d80cbec25579338b50a752c537e6745cbc7665e0868b68e16a5f1"} Oct 31 13:04:35.441288 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:35.440979 5693 scope.go:122] "RemoveContainer" containerID="f35347918fc0ed154824bf23d1ab19e548ff352e8fb09680571cf4fba939104f" Oct 31 13:04:35.478506 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:35.478477 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf"] Oct 31 13:04:35.482220 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:35.482198 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-lqlmf"] Oct 31 13:04:37.410225 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:37.410194 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758" path="/var/lib/kubelet/pods/ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758/volumes" Oct 31 13:04:40.028747 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:40.028713 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-w7mmf_6485629e-cfde-468c-93b0-9f68ca0ded66/payload-processing/0.log" Oct 31 13:04:40.245779 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:40.245749 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-w7mmf_6485629e-cfde-468c-93b0-9f68ca0ded66/payload-processing/0.log" Oct 31 13:04:40.430361 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:40.430336 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-c347ff-6c968cc48b-dxhqv_237288a4-fc4b-4333-bab3-0898a34fc166/payload-processing/0.log" Oct 31 13:04:41.393066 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:41.393018 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-204e6c-789f8c8f7c-tzjf8"] Oct 31 13:04:41.440382 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:41.440350 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-204e6c-76ffbdd775-7b6th"] Oct 31 13:04:46.393786 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:46.393744 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-204e6c-789f8c8f7c-tzjf8" podUID="5ce01e69-4d72-471e-a60a-ec028a9b2f87" containerName="payload-processing" containerID="cri-o://52fc0bea6825729f7529ad6cb75710667332eaa3c31f0678c4cc316a34d0ac8a" gracePeriod=25 Oct 31 13:04:46.440947 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:46.440892 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-204e6c-76ffbdd775-7b6th" podUID="75dac63c-3308-45cc-84b0-be41aa27da1a" containerName="payload-pre-processing" containerID="cri-o://0cb826a4474b229708cf2af686f7c537e75c2b208af851af0eca36902d0698bf" gracePeriod=25 Oct 31 13:04:46.518067 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:46.518036 5693 generic.go:356] "Generic (PLEG): container finished" podID="5ce01e69-4d72-471e-a60a-ec028a9b2f87" containerID="52fc0bea6825729f7529ad6cb75710667332eaa3c31f0678c4cc316a34d0ac8a" exitCode=0 Oct 31 13:04:46.518194 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:46.518105 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-204e6c-789f8c8f7c-tzjf8" event={"ID":"5ce01e69-4d72-471e-a60a-ec028a9b2f87","Type":"ContainerDied","Data":"52fc0bea6825729f7529ad6cb75710667332eaa3c31f0678c4cc316a34d0ac8a"} Oct 31 13:04:46.744369 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:46.744344 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-204e6c-789f8c8f7c-tzjf8" Oct 31 13:04:46.776915 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:46.776878 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-204e6c-76ffbdd775-7b6th" Oct 31 13:04:46.842138 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:46.842098 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ce01e69-4d72-471e-a60a-ec028a9b2f87-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce01e69-4d72-471e-a60a-ec028a9b2f87-plugins-config-volume\") pod \"5ce01e69-4d72-471e-a60a-ec028a9b2f87\" (UID: \"5ce01e69-4d72-471e-a60a-ec028a9b2f87\") " Oct 31 13:04:46.842287 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:46.842197 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75dac63c-3308-45cc-84b0-be41aa27da1a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75dac63c-3308-45cc-84b0-be41aa27da1a-plugins-config-volume\") pod \"75dac63c-3308-45cc-84b0-be41aa27da1a\" (UID: \"75dac63c-3308-45cc-84b0-be41aa27da1a\") " Oct 31 13:04:46.842388 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:46.842369 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75dac63c-3308-45cc-84b0-be41aa27da1a-kube-api-access-dcp7q\" (UniqueName: \"kubernetes.io/projected/75dac63c-3308-45cc-84b0-be41aa27da1a-kube-api-access-dcp7q\") pod \"75dac63c-3308-45cc-84b0-be41aa27da1a\" (UID: \"75dac63c-3308-45cc-84b0-be41aa27da1a\") " Oct 31 13:04:46.842472 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:46.842457 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ce01e69-4d72-471e-a60a-ec028a9b2f87-kube-api-access-zf87z\" (UniqueName: \"kubernetes.io/projected/5ce01e69-4d72-471e-a60a-ec028a9b2f87-kube-api-access-zf87z\") pod \"5ce01e69-4d72-471e-a60a-ec028a9b2f87\" (UID: \"5ce01e69-4d72-471e-a60a-ec028a9b2f87\") " Oct 31 13:04:46.842541 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:46.842465 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ce01e69-4d72-471e-a60a-ec028a9b2f87-plugins-config-volume" pod "5ce01e69-4d72-471e-a60a-ec028a9b2f87" (UID: "5ce01e69-4d72-471e-a60a-ec028a9b2f87"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:04:46.842541 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:46.842531 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75dac63c-3308-45cc-84b0-be41aa27da1a-plugins-config-volume" pod "75dac63c-3308-45cc-84b0-be41aa27da1a" (UID: "75dac63c-3308-45cc-84b0-be41aa27da1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:04:46.843168 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:46.843086 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75dac63c-3308-45cc-84b0-be41aa27da1a-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:46.843168 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:46.843114 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce01e69-4d72-471e-a60a-ec028a9b2f87-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:46.844699 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:46.844671 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ce01e69-4d72-471e-a60a-ec028a9b2f87-kube-api-access-zf87z" pod "5ce01e69-4d72-471e-a60a-ec028a9b2f87" (UID: "5ce01e69-4d72-471e-a60a-ec028a9b2f87"). InnerVolumeSpecName "kube-api-access-zf87z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:04:46.844699 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:46.844692 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75dac63c-3308-45cc-84b0-be41aa27da1a-kube-api-access-dcp7q" pod "75dac63c-3308-45cc-84b0-be41aa27da1a" (UID: "75dac63c-3308-45cc-84b0-be41aa27da1a"). InnerVolumeSpecName "kube-api-access-dcp7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:04:46.944269 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:46.944189 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf87z\" (UniqueName: \"kubernetes.io/projected/5ce01e69-4d72-471e-a60a-ec028a9b2f87-kube-api-access-zf87z\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:46.944269 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:46.944219 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcp7q\" (UniqueName: \"kubernetes.io/projected/75dac63c-3308-45cc-84b0-be41aa27da1a-kube-api-access-dcp7q\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:47.525688 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:47.525625 5693 generic.go:356] "Generic (PLEG): container finished" podID="75dac63c-3308-45cc-84b0-be41aa27da1a" containerID="0cb826a4474b229708cf2af686f7c537e75c2b208af851af0eca36902d0698bf" exitCode=0 Oct 31 13:04:47.526130 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:47.525708 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-204e6c-76ffbdd775-7b6th" Oct 31 13:04:47.526130 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:47.525709 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-204e6c-76ffbdd775-7b6th" event={"ID":"75dac63c-3308-45cc-84b0-be41aa27da1a","Type":"ContainerDied","Data":"0cb826a4474b229708cf2af686f7c537e75c2b208af851af0eca36902d0698bf"} Oct 31 13:04:47.526130 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:47.525768 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-204e6c-76ffbdd775-7b6th" event={"ID":"75dac63c-3308-45cc-84b0-be41aa27da1a","Type":"ContainerDied","Data":"90f1551315046677d13cd3242d21169a424f74c195ec874f23431ae6e1ce3208"} Oct 31 13:04:47.526130 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:47.525800 5693 scope.go:122] "RemoveContainer" containerID="0cb826a4474b229708cf2af686f7c537e75c2b208af851af0eca36902d0698bf" Oct 31 13:04:47.527297 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:47.527243 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-204e6c-789f8c8f7c-tzjf8" event={"ID":"5ce01e69-4d72-471e-a60a-ec028a9b2f87","Type":"ContainerDied","Data":"10f611e4205a5ba49dd6fc5a6e4a92b50fcbbcb79f943e680cb16e295185543b"} Oct 31 13:04:47.527297 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:47.527283 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-204e6c-789f8c8f7c-tzjf8" Oct 31 13:04:47.536535 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:47.536518 5693 scope.go:122] "RemoveContainer" containerID="0cb826a4474b229708cf2af686f7c537e75c2b208af851af0eca36902d0698bf" Oct 31 13:04:47.536839 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:04:47.536815 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cb826a4474b229708cf2af686f7c537e75c2b208af851af0eca36902d0698bf\": container with ID starting with 0cb826a4474b229708cf2af686f7c537e75c2b208af851af0eca36902d0698bf not found: ID does not exist" containerID="0cb826a4474b229708cf2af686f7c537e75c2b208af851af0eca36902d0698bf" Oct 31 13:04:47.536920 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:47.536848 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cb826a4474b229708cf2af686f7c537e75c2b208af851af0eca36902d0698bf"} err="failed to get container status \"0cb826a4474b229708cf2af686f7c537e75c2b208af851af0eca36902d0698bf\": rpc error: code = NotFound desc = could not find container \"0cb826a4474b229708cf2af686f7c537e75c2b208af851af0eca36902d0698bf\": container with ID starting with 0cb826a4474b229708cf2af686f7c537e75c2b208af851af0eca36902d0698bf not found: ID does not exist" Oct 31 13:04:47.536920 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:47.536869 5693 scope.go:122] "RemoveContainer" containerID="52fc0bea6825729f7529ad6cb75710667332eaa3c31f0678c4cc316a34d0ac8a" Oct 31 13:04:47.544761 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:47.544737 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-204e6c-76ffbdd775-7b6th"] Oct 31 13:04:47.550132 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:47.550109 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-204e6c-76ffbdd775-7b6th"] Oct 31 13:04:47.567430 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:47.567407 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-204e6c-789f8c8f7c-tzjf8"] Oct 31 13:04:47.571851 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:47.571824 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-204e6c-789f8c8f7c-tzjf8"] Oct 31 13:04:48.689012 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:04:48.688973 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:04:49.443509 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:49.443475 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ce01e69-4d72-471e-a60a-ec028a9b2f87" path="/var/lib/kubelet/pods/5ce01e69-4d72-471e-a60a-ec028a9b2f87/volumes" Oct 31 13:04:49.444431 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:49.444402 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75dac63c-3308-45cc-84b0-be41aa27da1a" path="/var/lib/kubelet/pods/75dac63c-3308-45cc-84b0-be41aa27da1a/volumes" Oct 31 13:04:51.627468 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:51.627436 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t"] Oct 31 13:04:51.627919 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:51.627677 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" podUID="0f339605-e98a-4957-88d8-59afe07f2a7b" containerName="istio-proxy" containerID="cri-o://4baa80937fb85e303325a4fcc0623522d35d980dcf5f81bd593a4892e7fa6911" gracePeriod=30 Oct 31 13:04:53.251557 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:53.251523 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf"] Oct 31 13:04:53.252335 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:53.251826 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf" podUID="a53ba38a-cc4c-4bd2-8241-f2772ac54e85" containerName="maas-api" containerID="cri-o://33da5d1292e0c7325f96e159f6c29e63c92de8a30a3d1497569ca89b70e4a634" gracePeriod=30 Oct 31 13:04:53.647383 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:53.647353 5693 generic.go:356] "Generic (PLEG): container finished" podID="a53ba38a-cc4c-4bd2-8241-f2772ac54e85" containerID="33da5d1292e0c7325f96e159f6c29e63c92de8a30a3d1497569ca89b70e4a634" exitCode=0 Oct 31 13:04:53.647573 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:53.647429 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf" event={"ID":"a53ba38a-cc4c-4bd2-8241-f2772ac54e85","Type":"ContainerDied","Data":"33da5d1292e0c7325f96e159f6c29e63c92de8a30a3d1497569ca89b70e4a634"} Oct 31 13:04:53.685118 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:53.685085 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf" Oct 31 13:04:53.760073 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:53.760038 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a53ba38a-cc4c-4bd2-8241-f2772ac54e85-kube-api-access-nrq2d\" (UniqueName: \"kubernetes.io/projected/a53ba38a-cc4c-4bd2-8241-f2772ac54e85-kube-api-access-nrq2d\") pod \"a53ba38a-cc4c-4bd2-8241-f2772ac54e85\" (UID: \"a53ba38a-cc4c-4bd2-8241-f2772ac54e85\") " Oct 31 13:04:53.761868 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:53.760090 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a53ba38a-cc4c-4bd2-8241-f2772ac54e85-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a53ba38a-cc4c-4bd2-8241-f2772ac54e85-maas-api-tls\") pod \"a53ba38a-cc4c-4bd2-8241-f2772ac54e85\" (UID: \"a53ba38a-cc4c-4bd2-8241-f2772ac54e85\") " Oct 31 13:04:53.761868 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:53.760332 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a53ba38a-cc4c-4bd2-8241-f2772ac54e85-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a53ba38a-cc4c-4bd2-8241-f2772ac54e85-openshift-service-ca\") pod \"a53ba38a-cc4c-4bd2-8241-f2772ac54e85\" (UID: \"a53ba38a-cc4c-4bd2-8241-f2772ac54e85\") " Oct 31 13:04:53.761868 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:53.760493 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a53ba38a-cc4c-4bd2-8241-f2772ac54e85-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a53ba38a-cc4c-4bd2-8241-f2772ac54e85-metrics-certs\") pod \"a53ba38a-cc4c-4bd2-8241-f2772ac54e85\" (UID: \"a53ba38a-cc4c-4bd2-8241-f2772ac54e85\") " Oct 31 13:04:53.761868 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:53.761148 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a53ba38a-cc4c-4bd2-8241-f2772ac54e85-openshift-service-ca" pod "a53ba38a-cc4c-4bd2-8241-f2772ac54e85" (UID: "a53ba38a-cc4c-4bd2-8241-f2772ac54e85"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:04:53.761868 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:53.761424 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a53ba38a-cc4c-4bd2-8241-f2772ac54e85-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:53.769751 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:53.765773 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a53ba38a-cc4c-4bd2-8241-f2772ac54e85-metrics-certs" pod "a53ba38a-cc4c-4bd2-8241-f2772ac54e85" (UID: "a53ba38a-cc4c-4bd2-8241-f2772ac54e85"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:04:53.770318 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:53.770285 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a53ba38a-cc4c-4bd2-8241-f2772ac54e85-kube-api-access-nrq2d" pod "a53ba38a-cc4c-4bd2-8241-f2772ac54e85" (UID: "a53ba38a-cc4c-4bd2-8241-f2772ac54e85"). InnerVolumeSpecName "kube-api-access-nrq2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:04:53.772797 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:53.772775 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a53ba38a-cc4c-4bd2-8241-f2772ac54e85-maas-api-tls" pod "a53ba38a-cc4c-4bd2-8241-f2772ac54e85" (UID: "a53ba38a-cc4c-4bd2-8241-f2772ac54e85"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:04:53.873517 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:53.873485 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a53ba38a-cc4c-4bd2-8241-f2772ac54e85-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:53.873517 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:53.873521 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrq2d\" (UniqueName: \"kubernetes.io/projected/a53ba38a-cc4c-4bd2-8241-f2772ac54e85-kube-api-access-nrq2d\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:53.873822 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:53.873537 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a53ba38a-cc4c-4bd2-8241-f2772ac54e85-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:54.667085 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:54.667053 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf" Oct 31 13:04:54.667085 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:54.667077 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf" event={"ID":"a53ba38a-cc4c-4bd2-8241-f2772ac54e85","Type":"ContainerDied","Data":"7e665779d39a42e36b9811ecb4d6509d273ca805576e47f48acfcf0a7c274889"} Oct 31 13:04:54.668376 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:54.667141 5693 scope.go:122] "RemoveContainer" containerID="33da5d1292e0c7325f96e159f6c29e63c92de8a30a3d1497569ca89b70e4a634" Oct 31 13:04:54.698840 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:54.698809 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf"] Oct 31 13:04:54.705106 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:54.705042 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fa2079-cb7944c86-jl4cf"] Oct 31 13:04:55.409331 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:55.409298 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a53ba38a-cc4c-4bd2-8241-f2772ac54e85" path="/var/lib/kubelet/pods/a53ba38a-cc4c-4bd2-8241-f2772ac54e85/volumes" Oct 31 13:04:56.930843 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:56.930814 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" Oct 31 13:04:57.014010 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:57.013965 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f339605-e98a-4957-88d8-59afe07f2a7b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f339605-e98a-4957-88d8-59afe07f2a7b-istio-data\") pod \"0f339605-e98a-4957-88d8-59afe07f2a7b\" (UID: \"0f339605-e98a-4957-88d8-59afe07f2a7b\") " Oct 31 13:04:57.014189 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:57.014027 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f339605-e98a-4957-88d8-59afe07f2a7b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f339605-e98a-4957-88d8-59afe07f2a7b-workload-certs\") pod \"0f339605-e98a-4957-88d8-59afe07f2a7b\" (UID: \"0f339605-e98a-4957-88d8-59afe07f2a7b\") " Oct 31 13:04:57.014189 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:57.014179 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f339605-e98a-4957-88d8-59afe07f2a7b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f339605-e98a-4957-88d8-59afe07f2a7b-istio-envoy\") pod \"0f339605-e98a-4957-88d8-59afe07f2a7b\" (UID: \"0f339605-e98a-4957-88d8-59afe07f2a7b\") " Oct 31 13:04:57.014318 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:57.014211 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f339605-e98a-4957-88d8-59afe07f2a7b-istio-data" pod "0f339605-e98a-4957-88d8-59afe07f2a7b" (UID: "0f339605-e98a-4957-88d8-59afe07f2a7b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:04:57.014318 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:57.014272 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f339605-e98a-4957-88d8-59afe07f2a7b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f339605-e98a-4957-88d8-59afe07f2a7b-istiod-ca-cert\") pod \"0f339605-e98a-4957-88d8-59afe07f2a7b\" (UID: \"0f339605-e98a-4957-88d8-59afe07f2a7b\") " Oct 31 13:04:57.014318 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:57.014301 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f339605-e98a-4957-88d8-59afe07f2a7b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f339605-e98a-4957-88d8-59afe07f2a7b-credential-socket\") pod \"0f339605-e98a-4957-88d8-59afe07f2a7b\" (UID: \"0f339605-e98a-4957-88d8-59afe07f2a7b\") " Oct 31 13:04:57.014472 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:57.014321 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f339605-e98a-4957-88d8-59afe07f2a7b-workload-certs" pod "0f339605-e98a-4957-88d8-59afe07f2a7b" (UID: "0f339605-e98a-4957-88d8-59afe07f2a7b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:04:57.014472 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:57.014337 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f339605-e98a-4957-88d8-59afe07f2a7b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f339605-e98a-4957-88d8-59afe07f2a7b-istio-podinfo\") pod \"0f339605-e98a-4957-88d8-59afe07f2a7b\" (UID: \"0f339605-e98a-4957-88d8-59afe07f2a7b\") " Oct 31 13:04:57.014472 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:57.014372 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f339605-e98a-4957-88d8-59afe07f2a7b-kube-api-access-wknz6\" (UniqueName: \"kubernetes.io/projected/0f339605-e98a-4957-88d8-59afe07f2a7b-kube-api-access-wknz6\") pod \"0f339605-e98a-4957-88d8-59afe07f2a7b\" (UID: \"0f339605-e98a-4957-88d8-59afe07f2a7b\") " Oct 31 13:04:57.014472 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:57.014443 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f339605-e98a-4957-88d8-59afe07f2a7b-istio-token\" (UniqueName: \"kubernetes.io/projected/0f339605-e98a-4957-88d8-59afe07f2a7b-istio-token\") pod \"0f339605-e98a-4957-88d8-59afe07f2a7b\" (UID: \"0f339605-e98a-4957-88d8-59afe07f2a7b\") " Oct 31 13:04:57.014701 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:57.014476 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f339605-e98a-4957-88d8-59afe07f2a7b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f339605-e98a-4957-88d8-59afe07f2a7b-workload-socket\") pod \"0f339605-e98a-4957-88d8-59afe07f2a7b\" (UID: \"0f339605-e98a-4957-88d8-59afe07f2a7b\") " Oct 31 13:04:57.014701 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:57.014667 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f339605-e98a-4957-88d8-59afe07f2a7b-istiod-ca-cert" pod "0f339605-e98a-4957-88d8-59afe07f2a7b" (UID: "0f339605-e98a-4957-88d8-59afe07f2a7b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:04:57.014968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:57.014946 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f339605-e98a-4957-88d8-59afe07f2a7b-credential-socket" pod "0f339605-e98a-4957-88d8-59afe07f2a7b" (UID: "0f339605-e98a-4957-88d8-59afe07f2a7b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:04:57.015038 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:57.014991 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f339605-e98a-4957-88d8-59afe07f2a7b-workload-socket" pod "0f339605-e98a-4957-88d8-59afe07f2a7b" (UID: "0f339605-e98a-4957-88d8-59afe07f2a7b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:04:57.015475 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:57.015311 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f339605-e98a-4957-88d8-59afe07f2a7b-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:57.015475 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:57.015408 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f339605-e98a-4957-88d8-59afe07f2a7b-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:57.015475 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:57.015424 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f339605-e98a-4957-88d8-59afe07f2a7b-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:57.015475 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:57.015438 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f339605-e98a-4957-88d8-59afe07f2a7b-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:57.015475 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:57.015451 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f339605-e98a-4957-88d8-59afe07f2a7b-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:57.018159 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:57.018136 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f339605-e98a-4957-88d8-59afe07f2a7b-istio-envoy" pod "0f339605-e98a-4957-88d8-59afe07f2a7b" (UID: "0f339605-e98a-4957-88d8-59afe07f2a7b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:04:57.019227 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:57.019206 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f339605-e98a-4957-88d8-59afe07f2a7b-kube-api-access-wknz6" pod "0f339605-e98a-4957-88d8-59afe07f2a7b" (UID: "0f339605-e98a-4957-88d8-59afe07f2a7b"). InnerVolumeSpecName "kube-api-access-wknz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:04:57.019442 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:57.019411 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f339605-e98a-4957-88d8-59afe07f2a7b-istio-token" pod "0f339605-e98a-4957-88d8-59afe07f2a7b" (UID: "0f339605-e98a-4957-88d8-59afe07f2a7b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:04:57.020212 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:57.020187 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f339605-e98a-4957-88d8-59afe07f2a7b-istio-podinfo" pod "0f339605-e98a-4957-88d8-59afe07f2a7b" (UID: "0f339605-e98a-4957-88d8-59afe07f2a7b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:04:57.140310 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:57.140272 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f339605-e98a-4957-88d8-59afe07f2a7b-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:57.140310 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:57.140310 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f339605-e98a-4957-88d8-59afe07f2a7b-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:57.140573 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:57.140326 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wknz6\" (UniqueName: \"kubernetes.io/projected/0f339605-e98a-4957-88d8-59afe07f2a7b-kube-api-access-wknz6\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:57.140573 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:57.140343 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f339605-e98a-4957-88d8-59afe07f2a7b-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:57.691332 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:57.691239 5693 generic.go:356] "Generic (PLEG): container finished" podID="0f339605-e98a-4957-88d8-59afe07f2a7b" containerID="4baa80937fb85e303325a4fcc0623522d35d980dcf5f81bd593a4892e7fa6911" exitCode=0 Oct 31 13:04:57.691332 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:57.691287 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" Oct 31 13:04:57.691583 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:57.691324 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" event={"ID":"0f339605-e98a-4957-88d8-59afe07f2a7b","Type":"ContainerDied","Data":"4baa80937fb85e303325a4fcc0623522d35d980dcf5f81bd593a4892e7fa6911"} Oct 31 13:04:57.691583 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:57.691394 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t" event={"ID":"0f339605-e98a-4957-88d8-59afe07f2a7b","Type":"ContainerDied","Data":"5dff25b60f6d732180b5c9ebe9a2fe23c973c0dfe8602644ccba341c5c45c396"} Oct 31 13:04:57.691583 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:57.691426 5693 scope.go:122] "RemoveContainer" containerID="4baa80937fb85e303325a4fcc0623522d35d980dcf5f81bd593a4892e7fa6911" Oct 31 13:04:57.704836 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:57.704814 5693 scope.go:122] "RemoveContainer" containerID="4903c0f767d66841eb4600396088e4dabad0bb51de52a827da3d27f1d333a42e" Oct 31 13:04:57.716954 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:57.716938 5693 scope.go:122] "RemoveContainer" containerID="4baa80937fb85e303325a4fcc0623522d35d980dcf5f81bd593a4892e7fa6911" Oct 31 13:04:57.717259 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:04:57.717231 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4baa80937fb85e303325a4fcc0623522d35d980dcf5f81bd593a4892e7fa6911\": container with ID starting with 4baa80937fb85e303325a4fcc0623522d35d980dcf5f81bd593a4892e7fa6911 not found: ID does not exist" containerID="4baa80937fb85e303325a4fcc0623522d35d980dcf5f81bd593a4892e7fa6911" Oct 31 13:04:57.717324 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:57.717260 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4baa80937fb85e303325a4fcc0623522d35d980dcf5f81bd593a4892e7fa6911"} err="failed to get container status \"4baa80937fb85e303325a4fcc0623522d35d980dcf5f81bd593a4892e7fa6911\": rpc error: code = NotFound desc = could not find container \"4baa80937fb85e303325a4fcc0623522d35d980dcf5f81bd593a4892e7fa6911\": container with ID starting with 4baa80937fb85e303325a4fcc0623522d35d980dcf5f81bd593a4892e7fa6911 not found: ID does not exist" Oct 31 13:04:57.717324 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:57.717282 5693 scope.go:122] "RemoveContainer" containerID="4903c0f767d66841eb4600396088e4dabad0bb51de52a827da3d27f1d333a42e" Oct 31 13:04:57.717539 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:04:57.717524 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4903c0f767d66841eb4600396088e4dabad0bb51de52a827da3d27f1d333a42e\": container with ID starting with 4903c0f767d66841eb4600396088e4dabad0bb51de52a827da3d27f1d333a42e not found: ID does not exist" containerID="4903c0f767d66841eb4600396088e4dabad0bb51de52a827da3d27f1d333a42e" Oct 31 13:04:57.717594 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:57.717543 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4903c0f767d66841eb4600396088e4dabad0bb51de52a827da3d27f1d333a42e"} err="failed to get container status \"4903c0f767d66841eb4600396088e4dabad0bb51de52a827da3d27f1d333a42e\": rpc error: code = NotFound desc = could not find container \"4903c0f767d66841eb4600396088e4dabad0bb51de52a827da3d27f1d333a42e\": container with ID starting with 4903c0f767d66841eb4600396088e4dabad0bb51de52a827da3d27f1d333a42e not found: ID does not exist" Oct 31 13:04:57.756788 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:57.756752 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t"] Oct 31 13:04:57.768746 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:57.768714 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-204e6c-openshift-default-6b47d8c649-5gg5t"] Oct 31 13:04:59.408578 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:04:59.408542 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f339605-e98a-4957-88d8-59afe07f2a7b" path="/var/lib/kubelet/pods/0f339605-e98a-4957-88d8-59afe07f2a7b/volumes" Oct 31 13:05:08.730878 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:08.730844 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c347ff-6c968cc48b-dxhqv"] Oct 31 13:05:08.731464 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:08.731080 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c347ff-6c968cc48b-dxhqv" podUID="237288a4-fc4b-4333-bab3-0898a34fc166" containerName="payload-processing" containerID="cri-o://4bc25eb31926e7cfa0954e884cddaf4cefa501ec3c1e3d1d36b58edc1c5a94c9" gracePeriod=30 Oct 31 13:05:08.770839 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:08.767801 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-7b5b64f855-rpv5d"] Oct 31 13:05:08.770839 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:08.768116 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-7b5b64f855-rpv5d" podUID="3894d4a0-ca36-4b76-8a2e-e1120d621cc4" containerName="payload-pre-processing" containerID="cri-o://8cd94251ecc65e40b969273b03983233094242121bec999f15ca7d29ecf8af49" gracePeriod=30 Oct 31 13:05:09.265332 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:09.265304 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c347ff-6c968cc48b-dxhqv" Oct 31 13:05:09.306798 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:09.306744 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-7b5b64f855-rpv5d" Oct 31 13:05:09.398795 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:09.398761 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/237288a4-fc4b-4333-bab3-0898a34fc166-kube-api-access-bzvm5\" (UniqueName: \"kubernetes.io/projected/237288a4-fc4b-4333-bab3-0898a34fc166-kube-api-access-bzvm5\") pod \"237288a4-fc4b-4333-bab3-0898a34fc166\" (UID: \"237288a4-fc4b-4333-bab3-0898a34fc166\") " Oct 31 13:05:09.398983 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:09.398860 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3894d4a0-ca36-4b76-8a2e-e1120d621cc4-kube-api-access-qv7v7\" (UniqueName: \"kubernetes.io/projected/3894d4a0-ca36-4b76-8a2e-e1120d621cc4-kube-api-access-qv7v7\") pod \"3894d4a0-ca36-4b76-8a2e-e1120d621cc4\" (UID: \"3894d4a0-ca36-4b76-8a2e-e1120d621cc4\") " Oct 31 13:05:09.398983 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:09.398920 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3894d4a0-ca36-4b76-8a2e-e1120d621cc4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3894d4a0-ca36-4b76-8a2e-e1120d621cc4-plugins-config-volume\") pod \"3894d4a0-ca36-4b76-8a2e-e1120d621cc4\" (UID: \"3894d4a0-ca36-4b76-8a2e-e1120d621cc4\") " Oct 31 13:05:09.399177 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:09.399112 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/237288a4-fc4b-4333-bab3-0898a34fc166-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/237288a4-fc4b-4333-bab3-0898a34fc166-plugins-config-volume\") pod \"237288a4-fc4b-4333-bab3-0898a34fc166\" (UID: \"237288a4-fc4b-4333-bab3-0898a34fc166\") " Oct 31 13:05:09.399391 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:09.399363 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3894d4a0-ca36-4b76-8a2e-e1120d621cc4-plugins-config-volume" pod "3894d4a0-ca36-4b76-8a2e-e1120d621cc4" (UID: "3894d4a0-ca36-4b76-8a2e-e1120d621cc4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:05:09.399512 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:09.399486 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/237288a4-fc4b-4333-bab3-0898a34fc166-plugins-config-volume" pod "237288a4-fc4b-4333-bab3-0898a34fc166" (UID: "237288a4-fc4b-4333-bab3-0898a34fc166"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:05:09.400149 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:09.400125 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/237288a4-fc4b-4333-bab3-0898a34fc166-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:09.400241 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:09.400155 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3894d4a0-ca36-4b76-8a2e-e1120d621cc4-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:09.401256 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:09.401232 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3894d4a0-ca36-4b76-8a2e-e1120d621cc4-kube-api-access-qv7v7" pod "3894d4a0-ca36-4b76-8a2e-e1120d621cc4" (UID: "3894d4a0-ca36-4b76-8a2e-e1120d621cc4"). InnerVolumeSpecName "kube-api-access-qv7v7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:05:09.401375 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:09.401361 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/237288a4-fc4b-4333-bab3-0898a34fc166-kube-api-access-bzvm5" pod "237288a4-fc4b-4333-bab3-0898a34fc166" (UID: "237288a4-fc4b-4333-bab3-0898a34fc166"). InnerVolumeSpecName "kube-api-access-bzvm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:05:09.501087 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:09.501052 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv7v7\" (UniqueName: \"kubernetes.io/projected/3894d4a0-ca36-4b76-8a2e-e1120d621cc4-kube-api-access-qv7v7\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:09.501087 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:09.501079 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzvm5\" (UniqueName: \"kubernetes.io/projected/237288a4-fc4b-4333-bab3-0898a34fc166-kube-api-access-bzvm5\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:09.786366 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:09.786332 5693 generic.go:356] "Generic (PLEG): container finished" podID="237288a4-fc4b-4333-bab3-0898a34fc166" containerID="4bc25eb31926e7cfa0954e884cddaf4cefa501ec3c1e3d1d36b58edc1c5a94c9" exitCode=0 Oct 31 13:05:09.786863 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:09.786380 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c347ff-6c968cc48b-dxhqv" Oct 31 13:05:09.786863 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:09.786415 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c347ff-6c968cc48b-dxhqv" event={"ID":"237288a4-fc4b-4333-bab3-0898a34fc166","Type":"ContainerDied","Data":"4bc25eb31926e7cfa0954e884cddaf4cefa501ec3c1e3d1d36b58edc1c5a94c9"} Oct 31 13:05:09.786863 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:09.786477 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c347ff-6c968cc48b-dxhqv" event={"ID":"237288a4-fc4b-4333-bab3-0898a34fc166","Type":"ContainerDied","Data":"aba89d6dd338b7bb88c260dcddb2f7fd6bff8dcdc08622083389ba6010ba2676"} Oct 31 13:05:09.786863 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:09.786507 5693 scope.go:122] "RemoveContainer" containerID="4bc25eb31926e7cfa0954e884cddaf4cefa501ec3c1e3d1d36b58edc1c5a94c9" Oct 31 13:05:09.788614 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:09.788522 5693 generic.go:356] "Generic (PLEG): container finished" podID="3894d4a0-ca36-4b76-8a2e-e1120d621cc4" containerID="8cd94251ecc65e40b969273b03983233094242121bec999f15ca7d29ecf8af49" exitCode=0 Oct 31 13:05:09.788614 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:09.788597 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-7b5b64f855-rpv5d" event={"ID":"3894d4a0-ca36-4b76-8a2e-e1120d621cc4","Type":"ContainerDied","Data":"8cd94251ecc65e40b969273b03983233094242121bec999f15ca7d29ecf8af49"} Oct 31 13:05:09.788829 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:09.788631 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-7b5b64f855-rpv5d" event={"ID":"3894d4a0-ca36-4b76-8a2e-e1120d621cc4","Type":"ContainerDied","Data":"e4a5f52a3e34ece2e837cb0d638487be529655f188d1fc7473ac8fe0318d889a"} Oct 31 13:05:09.788829 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:09.788727 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-7b5b64f855-rpv5d" Oct 31 13:05:09.802118 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:09.802097 5693 scope.go:122] "RemoveContainer" containerID="4bc25eb31926e7cfa0954e884cddaf4cefa501ec3c1e3d1d36b58edc1c5a94c9" Oct 31 13:05:09.802471 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:05:09.802447 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bc25eb31926e7cfa0954e884cddaf4cefa501ec3c1e3d1d36b58edc1c5a94c9\": container with ID starting with 4bc25eb31926e7cfa0954e884cddaf4cefa501ec3c1e3d1d36b58edc1c5a94c9 not found: ID does not exist" containerID="4bc25eb31926e7cfa0954e884cddaf4cefa501ec3c1e3d1d36b58edc1c5a94c9" Oct 31 13:05:09.802627 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:09.802510 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bc25eb31926e7cfa0954e884cddaf4cefa501ec3c1e3d1d36b58edc1c5a94c9"} err="failed to get container status \"4bc25eb31926e7cfa0954e884cddaf4cefa501ec3c1e3d1d36b58edc1c5a94c9\": rpc error: code = NotFound desc = could not find container \"4bc25eb31926e7cfa0954e884cddaf4cefa501ec3c1e3d1d36b58edc1c5a94c9\": container with ID starting with 4bc25eb31926e7cfa0954e884cddaf4cefa501ec3c1e3d1d36b58edc1c5a94c9 not found: ID does not exist" Oct 31 13:05:09.802627 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:09.802536 5693 scope.go:122] "RemoveContainer" containerID="8cd94251ecc65e40b969273b03983233094242121bec999f15ca7d29ecf8af49" Oct 31 13:05:09.813262 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:09.813232 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-7b5b64f855-rpv5d"] Oct 31 13:05:09.819062 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:09.819036 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-7b5b64f855-rpv5d"] Oct 31 13:05:09.820322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:09.820242 5693 scope.go:122] "RemoveContainer" containerID="8cd94251ecc65e40b969273b03983233094242121bec999f15ca7d29ecf8af49" Oct 31 13:05:09.820894 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:05:09.820825 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cd94251ecc65e40b969273b03983233094242121bec999f15ca7d29ecf8af49\": container with ID starting with 8cd94251ecc65e40b969273b03983233094242121bec999f15ca7d29ecf8af49 not found: ID does not exist" containerID="8cd94251ecc65e40b969273b03983233094242121bec999f15ca7d29ecf8af49" Oct 31 13:05:09.820894 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:09.820860 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cd94251ecc65e40b969273b03983233094242121bec999f15ca7d29ecf8af49"} err="failed to get container status \"8cd94251ecc65e40b969273b03983233094242121bec999f15ca7d29ecf8af49\": rpc error: code = NotFound desc = could not find container \"8cd94251ecc65e40b969273b03983233094242121bec999f15ca7d29ecf8af49\": container with ID starting with 8cd94251ecc65e40b969273b03983233094242121bec999f15ca7d29ecf8af49 not found: ID does not exist" Oct 31 13:05:09.834024 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:09.834001 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c347ff-6c968cc48b-dxhqv"] Oct 31 13:05:09.836817 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:09.836799 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c347ff-6c968cc48b-dxhqv"] Oct 31 13:05:11.410302 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:11.410271 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="237288a4-fc4b-4333-bab3-0898a34fc166" path="/var/lib/kubelet/pods/237288a4-fc4b-4333-bab3-0898a34fc166/volumes" Oct 31 13:05:11.410741 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:11.410727 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3894d4a0-ca36-4b76-8a2e-e1120d621cc4" path="/var/lib/kubelet/pods/3894d4a0-ca36-4b76-8a2e-e1120d621cc4/volumes" Oct 31 13:05:18.690271 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:05:18.690225 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:05:20.959383 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:20.959348 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7e08dc-7b446bdff9-vvsqh"] Oct 31 13:05:20.959945 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:20.959924 5693 cpu_manager.go:414] "containerMap: removing container" podUID="084961ae-b1f7-4f97-aca7-21e9cc5ae882" containerName="payload-processing" Oct 31 13:05:20.959945 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:20.959942 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="084961ae-b1f7-4f97-aca7-21e9cc5ae882" containerName="payload-processing" Oct 31 13:05:20.960107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:20.959962 5693 cpu_manager.go:414] "containerMap: removing container" podUID="5780a4a9-4b92-4663-8211-d831e6cfad35" containerName="istio-proxy" Oct 31 13:05:20.960107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:20.959967 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5780a4a9-4b92-4663-8211-d831e6cfad35" containerName="istio-proxy" Oct 31 13:05:20.960107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:20.959974 5693 cpu_manager.go:414] "containerMap: removing container" podUID="237288a4-fc4b-4333-bab3-0898a34fc166" containerName="payload-processing" Oct 31 13:05:20.960107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:20.959979 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="237288a4-fc4b-4333-bab3-0898a34fc166" containerName="payload-processing" Oct 31 13:05:20.960107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:20.959993 5693 cpu_manager.go:414] "containerMap: removing container" podUID="39047437-bef9-4407-b5e6-2781f6811546" containerName="payload-pre-processing" Oct 31 13:05:20.960107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:20.959998 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39047437-bef9-4407-b5e6-2781f6811546" containerName="payload-pre-processing" Oct 31 13:05:20.960107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:20.960009 5693 cpu_manager.go:414] "containerMap: removing container" podUID="3894d4a0-ca36-4b76-8a2e-e1120d621cc4" containerName="payload-pre-processing" Oct 31 13:05:20.960107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:20.960014 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3894d4a0-ca36-4b76-8a2e-e1120d621cc4" containerName="payload-pre-processing" Oct 31 13:05:20.960107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:20.960029 5693 cpu_manager.go:414] "containerMap: removing container" podUID="75dac63c-3308-45cc-84b0-be41aa27da1a" containerName="payload-pre-processing" Oct 31 13:05:20.960107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:20.960033 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75dac63c-3308-45cc-84b0-be41aa27da1a" containerName="payload-pre-processing" Oct 31 13:05:20.960107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:20.960041 5693 cpu_manager.go:414] "containerMap: removing container" podUID="ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758" containerName="istio-proxy" Oct 31 13:05:20.960107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:20.960045 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758" containerName="istio-proxy" Oct 31 13:05:20.960107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:20.960056 5693 cpu_manager.go:414] "containerMap: removing container" podUID="a53ba38a-cc4c-4bd2-8241-f2772ac54e85" containerName="maas-api" Oct 31 13:05:20.960107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:20.960060 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a53ba38a-cc4c-4bd2-8241-f2772ac54e85" containerName="maas-api" Oct 31 13:05:20.960107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:20.960073 5693 cpu_manager.go:414] "containerMap: removing container" podUID="5ce01e69-4d72-471e-a60a-ec028a9b2f87" containerName="payload-processing" Oct 31 13:05:20.960107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:20.960077 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ce01e69-4d72-471e-a60a-ec028a9b2f87" containerName="payload-processing" Oct 31 13:05:20.960107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:20.960083 5693 cpu_manager.go:414] "containerMap: removing container" podUID="0f339605-e98a-4957-88d8-59afe07f2a7b" containerName="istio-proxy" Oct 31 13:05:20.960107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:20.960088 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f339605-e98a-4957-88d8-59afe07f2a7b" containerName="istio-proxy" Oct 31 13:05:20.960107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:20.960095 5693 cpu_manager.go:414] "containerMap: removing container" podUID="99c723db-b336-48a6-a52b-df5174b4ab3d" containerName="istio-proxy" Oct 31 13:05:20.960107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:20.960099 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99c723db-b336-48a6-a52b-df5174b4ab3d" containerName="istio-proxy" Oct 31 13:05:20.960107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:20.960105 5693 cpu_manager.go:414] "containerMap: removing container" podUID="0f339605-e98a-4957-88d8-59afe07f2a7b" containerName="istio-proxy" Oct 31 13:05:20.960107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:20.960110 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f339605-e98a-4957-88d8-59afe07f2a7b" containerName="istio-proxy" Oct 31 13:05:20.960796 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:20.960190 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="5780a4a9-4b92-4663-8211-d831e6cfad35" containerName="istio-proxy" Oct 31 13:05:20.960796 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:20.960197 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="237288a4-fc4b-4333-bab3-0898a34fc166" containerName="payload-processing" Oct 31 13:05:20.960796 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:20.960204 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="39047437-bef9-4407-b5e6-2781f6811546" containerName="payload-pre-processing" Oct 31 13:05:20.960796 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:20.960211 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="3894d4a0-ca36-4b76-8a2e-e1120d621cc4" containerName="payload-pre-processing" Oct 31 13:05:20.960796 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:20.960220 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="75dac63c-3308-45cc-84b0-be41aa27da1a" containerName="payload-pre-processing" Oct 31 13:05:20.960796 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:20.960225 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebbd6fa9-5e1a-4244-8c0e-5e6f4c8df758" containerName="istio-proxy" Oct 31 13:05:20.960796 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:20.960231 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="a53ba38a-cc4c-4bd2-8241-f2772ac54e85" containerName="maas-api" Oct 31 13:05:20.960796 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:20.960236 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ce01e69-4d72-471e-a60a-ec028a9b2f87" containerName="payload-processing" Oct 31 13:05:20.960796 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:20.960241 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f339605-e98a-4957-88d8-59afe07f2a7b" containerName="istio-proxy" Oct 31 13:05:20.960796 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:20.960247 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f339605-e98a-4957-88d8-59afe07f2a7b" containerName="istio-proxy" Oct 31 13:05:20.960796 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:20.960251 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="99c723db-b336-48a6-a52b-df5174b4ab3d" containerName="istio-proxy" Oct 31 13:05:20.960796 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:20.960257 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="084961ae-b1f7-4f97-aca7-21e9cc5ae882" containerName="payload-processing" Oct 31 13:05:20.963629 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:20.963613 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7e08dc-7b446bdff9-vvsqh" Oct 31 13:05:20.984393 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:20.984369 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-7e08dc\"" Oct 31 13:05:21.005554 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:21.005525 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7e08dc-7b446bdff9-vvsqh"] Oct 31 13:05:21.022187 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:21.022154 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7e08dc-6cf96fcf89-pzv8w"] Oct 31 13:05:21.026329 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:21.026309 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-7e08dc-6cf96fcf89-pzv8w" Oct 31 13:05:21.046784 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:21.046749 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7e08dc-6cf96fcf89-pzv8w"] Oct 31 13:05:21.055960 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:21.055930 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd465\" (UniqueName: \"kubernetes.io/projected/c1b76053-3fdc-49c0-97e1-cc142093b6b7-kube-api-access-nd465\") pod \"payload-pre-processing-e2e-praxis-a-7e08dc-7b446bdff9-vvsqh\" (UID: \"c1b76053-3fdc-49c0-97e1-cc142093b6b7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7e08dc-7b446bdff9-vvsqh" Oct 31 13:05:21.056101 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:21.056077 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bf930ee-79a6-4b2e-a18e-77b4948aba78-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-7e08dc-6cf96fcf89-pzv8w\" (UID: \"2bf930ee-79a6-4b2e-a18e-77b4948aba78\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7e08dc-6cf96fcf89-pzv8w" Oct 31 13:05:21.056151 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:21.056131 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b76053-3fdc-49c0-97e1-cc142093b6b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-7e08dc-7b446bdff9-vvsqh\" (UID: \"c1b76053-3fdc-49c0-97e1-cc142093b6b7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7e08dc-7b446bdff9-vvsqh" Oct 31 13:05:21.056350 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:21.056322 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szqsg\" (UniqueName: \"kubernetes.io/projected/2bf930ee-79a6-4b2e-a18e-77b4948aba78-kube-api-access-szqsg\") pod \"payload-processing-e2e-praxis-a-7e08dc-6cf96fcf89-pzv8w\" (UID: \"2bf930ee-79a6-4b2e-a18e-77b4948aba78\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7e08dc-6cf96fcf89-pzv8w" Oct 31 13:05:21.068425 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:21.068391 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-7e08dc-dockercfg-pbg9s\"" Oct 31 13:05:21.165291 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:21.160194 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szqsg\" (UniqueName: \"kubernetes.io/projected/2bf930ee-79a6-4b2e-a18e-77b4948aba78-kube-api-access-szqsg\") pod \"payload-processing-e2e-praxis-a-7e08dc-6cf96fcf89-pzv8w\" (UID: \"2bf930ee-79a6-4b2e-a18e-77b4948aba78\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7e08dc-6cf96fcf89-pzv8w" Oct 31 13:05:21.165291 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:21.160335 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd465\" (UniqueName: \"kubernetes.io/projected/c1b76053-3fdc-49c0-97e1-cc142093b6b7-kube-api-access-nd465\") pod \"payload-pre-processing-e2e-praxis-a-7e08dc-7b446bdff9-vvsqh\" (UID: \"c1b76053-3fdc-49c0-97e1-cc142093b6b7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7e08dc-7b446bdff9-vvsqh" Oct 31 13:05:21.165291 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:21.160459 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bf930ee-79a6-4b2e-a18e-77b4948aba78-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-7e08dc-6cf96fcf89-pzv8w\" (UID: \"2bf930ee-79a6-4b2e-a18e-77b4948aba78\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7e08dc-6cf96fcf89-pzv8w" Oct 31 13:05:21.165291 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:21.160508 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b76053-3fdc-49c0-97e1-cc142093b6b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-7e08dc-7b446bdff9-vvsqh\" (UID: \"c1b76053-3fdc-49c0-97e1-cc142093b6b7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7e08dc-7b446bdff9-vvsqh" Oct 31 13:05:21.189665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:21.181315 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bf930ee-79a6-4b2e-a18e-77b4948aba78-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-7e08dc-6cf96fcf89-pzv8w\" (UID: \"2bf930ee-79a6-4b2e-a18e-77b4948aba78\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7e08dc-6cf96fcf89-pzv8w" Oct 31 13:05:21.205663 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:21.191525 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b76053-3fdc-49c0-97e1-cc142093b6b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-7e08dc-7b446bdff9-vvsqh\" (UID: \"c1b76053-3fdc-49c0-97e1-cc142093b6b7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7e08dc-7b446bdff9-vvsqh" Oct 31 13:05:21.251389 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:21.249305 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szqsg\" (UniqueName: \"kubernetes.io/projected/2bf930ee-79a6-4b2e-a18e-77b4948aba78-kube-api-access-szqsg\") pod \"payload-processing-e2e-praxis-a-7e08dc-6cf96fcf89-pzv8w\" (UID: \"2bf930ee-79a6-4b2e-a18e-77b4948aba78\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7e08dc-6cf96fcf89-pzv8w" Oct 31 13:05:21.272666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:21.265516 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd465\" (UniqueName: \"kubernetes.io/projected/c1b76053-3fdc-49c0-97e1-cc142093b6b7-kube-api-access-nd465\") pod \"payload-pre-processing-e2e-praxis-a-7e08dc-7b446bdff9-vvsqh\" (UID: \"c1b76053-3fdc-49c0-97e1-cc142093b6b7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7e08dc-7b446bdff9-vvsqh" Oct 31 13:05:21.285086 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:21.283895 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7e08dc-7b446bdff9-vvsqh" Oct 31 13:05:21.350782 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:21.350748 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-7e08dc-6cf96fcf89-pzv8w" Oct 31 13:05:21.602828 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:21.602800 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7e08dc-7b446bdff9-vvsqh"] Oct 31 13:05:21.603215 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:05:21.603196 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1b76053_3fdc_49c0_97e1_cc142093b6b7.slice/crio-d2fdb7b7147e6b78eb4ad7a1c791d98e2ee3271461d0bed08b030e2a7d0bd9e6 WatchSource:0}: Error finding container d2fdb7b7147e6b78eb4ad7a1c791d98e2ee3271461d0bed08b030e2a7d0bd9e6: Status 404 returned error can't find the container with id d2fdb7b7147e6b78eb4ad7a1c791d98e2ee3271461d0bed08b030e2a7d0bd9e6 Oct 31 13:05:21.687348 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:05:21.687307 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bf930ee_79a6_4b2e_a18e_77b4948aba78.slice/crio-ee120b7c80ab9f77f26d3f60733891ea5f7e42988d2a59fe26f83dc78d126276 WatchSource:0}: Error finding container ee120b7c80ab9f77f26d3f60733891ea5f7e42988d2a59fe26f83dc78d126276: Status 404 returned error can't find the container with id ee120b7c80ab9f77f26d3f60733891ea5f7e42988d2a59fe26f83dc78d126276 Oct 31 13:05:21.688434 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:21.688405 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7e08dc-6cf96fcf89-pzv8w"] Oct 31 13:05:21.961248 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:21.961212 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-7e08dc-6cf96fcf89-pzv8w" event={"ID":"2bf930ee-79a6-4b2e-a18e-77b4948aba78","Type":"ContainerStarted","Data":"36ab8df3c04988a7464ac23f6b6b0eca69c10361abc7c5ea22ec9510151ffce6"} Oct 31 13:05:21.961739 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:21.961274 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-7e08dc-6cf96fcf89-pzv8w" event={"ID":"2bf930ee-79a6-4b2e-a18e-77b4948aba78","Type":"ContainerStarted","Data":"ee120b7c80ab9f77f26d3f60733891ea5f7e42988d2a59fe26f83dc78d126276"} Oct 31 13:05:21.961739 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:21.961366 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-7e08dc-6cf96fcf89-pzv8w" Oct 31 13:05:21.962860 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:21.962837 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7e08dc-7b446bdff9-vvsqh" event={"ID":"c1b76053-3fdc-49c0-97e1-cc142093b6b7","Type":"ContainerStarted","Data":"b2a399a55280496253eddcae956d0c6ccfc8501326c03df050106013707d7a56"} Oct 31 13:05:21.962977 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:21.962884 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7e08dc-7b446bdff9-vvsqh" Oct 31 13:05:21.962977 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:21.962905 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7e08dc-7b446bdff9-vvsqh" event={"ID":"c1b76053-3fdc-49c0-97e1-cc142093b6b7","Type":"ContainerStarted","Data":"d2fdb7b7147e6b78eb4ad7a1c791d98e2ee3271461d0bed08b030e2a7d0bd9e6"} Oct 31 13:05:22.029951 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:22.029904 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-7e08dc-6cf96fcf89-pzv8w" podStartSLOduration=2.029889316 podStartE2EDuration="2.029889316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:05:22.026390229 +0000 UTC m=+1972.733976524" watchObservedRunningTime="2026-10-31 13:05:22.029889316 +0000 UTC m=+1972.737475611" Oct 31 13:05:22.060014 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:22.059968 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7e08dc-7b446bdff9-vvsqh" podStartSLOduration=2.059956297 podStartE2EDuration="2.059956297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:05:22.058289928 +0000 UTC m=+1972.765876219" watchObservedRunningTime="2026-10-31 13:05:22.059956297 +0000 UTC m=+1972.767542625" Oct 31 13:05:32.973181 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:32.973153 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7e08dc-7b446bdff9-vvsqh" Oct 31 13:05:32.973576 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:32.973225 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-7e08dc-6cf96fcf89-pzv8w" Oct 31 13:05:38.367666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:38.367422 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" podUID="f2b3766f-058c-40e7-9a2d-a7f5028e2b1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:05:43.463357 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:43.463324 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-w7mmf_6485629e-cfde-468c-93b0-9f68ca0ded66/payload-processing/0.log" Oct 31 13:05:43.762088 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:43.762002 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-w7mmf_6485629e-cfde-468c-93b0-9f68ca0ded66/payload-processing/0.log" Oct 31 13:05:46.549344 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:46.549305 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h"] Oct 31 13:05:46.555792 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:46.555763 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" Oct 31 13:05:46.563898 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:46.563876 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-3a3000-openshift-default-dockercfg-8nc9m\"" Oct 31 13:05:46.595148 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:46.595107 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h"] Oct 31 13:05:46.638391 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:46.638358 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-istio-podinfo\") pod \"e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h\" (UID: \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\") " pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" Oct 31 13:05:46.638755 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:46.638735 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-istiod-ca-cert\") pod \"e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h\" (UID: \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\") " pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" Oct 31 13:05:46.638936 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:46.638918 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-workload-certs\") pod \"e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h\" (UID: \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\") " pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" Oct 31 13:05:46.639100 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:46.639079 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-istio-token\") pod \"e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h\" (UID: \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\") " pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" Oct 31 13:05:46.639305 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:46.639291 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-workload-socket\") pod \"e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h\" (UID: \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\") " pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" Oct 31 13:05:46.639457 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:46.639442 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-istio-data\") pod \"e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h\" (UID: \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\") " pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" Oct 31 13:05:46.639634 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:46.639621 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-istio-envoy\") pod \"e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h\" (UID: \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\") " pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" Oct 31 13:05:46.639870 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:46.639853 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f29vp\" (UniqueName: \"kubernetes.io/projected/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-kube-api-access-f29vp\") pod \"e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h\" (UID: \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\") " pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" Oct 31 13:05:46.640113 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:46.640097 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-credential-socket\") pod \"e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h\" (UID: \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\") " pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" Oct 31 13:05:46.741761 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:46.741730 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-istio-token\") pod \"e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h\" (UID: \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\") " pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" Oct 31 13:05:46.741974 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:46.741805 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-workload-socket\") pod \"e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h\" (UID: \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\") " pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" Oct 31 13:05:46.741974 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:46.741849 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-istio-data\") pod \"e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h\" (UID: \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\") " pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" Oct 31 13:05:46.741974 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:46.741916 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-istio-envoy\") pod \"e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h\" (UID: \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\") " pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" Oct 31 13:05:46.742159 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:46.742000 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f29vp\" (UniqueName: \"kubernetes.io/projected/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-kube-api-access-f29vp\") pod \"e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h\" (UID: \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\") " pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" Oct 31 13:05:46.742159 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:46.742031 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-credential-socket\") pod \"e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h\" (UID: \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\") " pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" Oct 31 13:05:46.742719 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:46.742689 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-istio-podinfo\") pod \"e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h\" (UID: \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\") " pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" Oct 31 13:05:46.742930 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:46.742908 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-istiod-ca-cert\") pod \"e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h\" (UID: \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\") " pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" Oct 31 13:05:46.743032 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:46.742954 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-workload-certs\") pod \"e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h\" (UID: \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\") " pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" Oct 31 13:05:46.743032 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:46.743009 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-credential-socket\") pod \"e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h\" (UID: \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\") " pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" Oct 31 13:05:46.743148 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:46.742833 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-workload-socket\") pod \"e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h\" (UID: \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\") " pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" Oct 31 13:05:46.743297 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:46.743277 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-workload-certs\") pod \"e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h\" (UID: \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\") " pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" Oct 31 13:05:46.743676 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:46.743632 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-istiod-ca-cert\") pod \"e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h\" (UID: \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\") " pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" Oct 31 13:05:46.744114 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:46.744092 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-istio-data\") pod \"e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h\" (UID: \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\") " pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" Oct 31 13:05:46.746341 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:46.746319 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-istio-envoy\") pod \"e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h\" (UID: \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\") " pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" Oct 31 13:05:46.746543 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:46.746522 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-istio-podinfo\") pod \"e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h\" (UID: \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\") " pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" Oct 31 13:05:46.758876 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:46.758844 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-istio-token\") pod \"e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h\" (UID: \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\") " pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" Oct 31 13:05:46.760334 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:46.759861 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f29vp\" (UniqueName: \"kubernetes.io/projected/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-kube-api-access-f29vp\") pod \"e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h\" (UID: \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\") " pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" Oct 31 13:05:46.869888 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:46.869796 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" Oct 31 13:05:47.249125 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:05:47.249098 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d0d805e_ce8f_4cd0_91c1_7f00fa21e672.slice/crio-c6e18dd38f3c4b4905cf17d20c718b3d836c0fbc82566f98b13c1bd45a91d671 WatchSource:0}: Error finding container c6e18dd38f3c4b4905cf17d20c718b3d836c0fbc82566f98b13c1bd45a91d671: Status 404 returned error can't find the container with id c6e18dd38f3c4b4905cf17d20c718b3d836c0fbc82566f98b13c1bd45a91d671 Oct 31 13:05:47.250952 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:47.250928 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h"] Oct 31 13:05:47.252366 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:47.252304 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:05:47.252463 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:47.252419 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:05:47.252504 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:47.252468 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:05:48.181762 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:48.181722 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" event={"ID":"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672","Type":"ContainerStarted","Data":"6544623a77af546e84f159923833b0a89c99167c9683d4a2872354068bc62f83"} Oct 31 13:05:48.182142 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:48.181815 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" event={"ID":"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672","Type":"ContainerStarted","Data":"c6e18dd38f3c4b4905cf17d20c718b3d836c0fbc82566f98b13c1bd45a91d671"} Oct 31 13:05:48.212243 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:48.212183 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" podStartSLOduration=2.21216416 podStartE2EDuration="2.21216416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:05:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:05:48.211057544 +0000 UTC m=+1998.918643842" watchObservedRunningTime="2026-10-31 13:05:48.21216416 +0000 UTC m=+1998.919750458" Oct 31 13:05:48.698399 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:05:48.698365 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:05:48.874674 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:48.871385 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" Oct 31 13:05:48.890898 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:48.890861 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" Oct 31 13:05:49.189101 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:49.189069 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" Oct 31 13:05:49.190192 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:49.190168 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" Oct 31 13:05:51.899828 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:51.899790 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3a3000-6c8fc86499-5rmb9"] Oct 31 13:05:51.904239 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:51.904215 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3a3000-6c8fc86499-5rmb9" Oct 31 13:05:51.914536 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:51.914504 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-3a3000\"" Oct 31 13:05:51.919703 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:51.919677 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3a3000-6c8fc86499-5rmb9"] Oct 31 13:05:51.926760 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:51.926735 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3a3000-589587b7c5-nf4bj"] Oct 31 13:05:51.931711 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:51.931689 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3a3000-589587b7c5-nf4bj" Oct 31 13:05:51.932738 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:51.932552 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc77g\" (UniqueName: \"kubernetes.io/projected/818a6fca-5402-45b0-9851-f31157caddc7-kube-api-access-hc77g\") pod \"payload-pre-processing-e2e-praxis-b-3a3000-6c8fc86499-5rmb9\" (UID: \"818a6fca-5402-45b0-9851-f31157caddc7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3a3000-6c8fc86499-5rmb9" Oct 31 13:05:51.932738 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:51.932732 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/818a6fca-5402-45b0-9851-f31157caddc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-3a3000-6c8fc86499-5rmb9\" (UID: \"818a6fca-5402-45b0-9851-f31157caddc7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3a3000-6c8fc86499-5rmb9" Oct 31 13:05:51.946701 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:51.945819 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-3a3000-dockercfg-9f967\"" Oct 31 13:05:51.950497 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:51.950470 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3a3000-589587b7c5-nf4bj"] Oct 31 13:05:52.047823 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:52.034114 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/818a6fca-5402-45b0-9851-f31157caddc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-3a3000-6c8fc86499-5rmb9\" (UID: \"818a6fca-5402-45b0-9851-f31157caddc7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3a3000-6c8fc86499-5rmb9" Oct 31 13:05:52.047823 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:52.035449 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc77g\" (UniqueName: \"kubernetes.io/projected/818a6fca-5402-45b0-9851-f31157caddc7-kube-api-access-hc77g\") pod \"payload-pre-processing-e2e-praxis-b-3a3000-6c8fc86499-5rmb9\" (UID: \"818a6fca-5402-45b0-9851-f31157caddc7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3a3000-6c8fc86499-5rmb9" Oct 31 13:05:52.047823 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:52.037507 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2z8d\" (UniqueName: \"kubernetes.io/projected/df10a6fe-ce6b-4515-9264-79d420a3389a-kube-api-access-v2z8d\") pod \"payload-processing-e2e-praxis-b-3a3000-589587b7c5-nf4bj\" (UID: \"df10a6fe-ce6b-4515-9264-79d420a3389a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3a3000-589587b7c5-nf4bj" Oct 31 13:05:52.047823 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:52.040005 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df10a6fe-ce6b-4515-9264-79d420a3389a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-3a3000-589587b7c5-nf4bj\" (UID: \"df10a6fe-ce6b-4515-9264-79d420a3389a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3a3000-589587b7c5-nf4bj" Oct 31 13:05:52.047823 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:52.040170 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/818a6fca-5402-45b0-9851-f31157caddc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-3a3000-6c8fc86499-5rmb9\" (UID: \"818a6fca-5402-45b0-9851-f31157caddc7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3a3000-6c8fc86499-5rmb9" Oct 31 13:05:52.062372 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:52.062340 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc77g\" (UniqueName: \"kubernetes.io/projected/818a6fca-5402-45b0-9851-f31157caddc7-kube-api-access-hc77g\") pod \"payload-pre-processing-e2e-praxis-b-3a3000-6c8fc86499-5rmb9\" (UID: \"818a6fca-5402-45b0-9851-f31157caddc7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3a3000-6c8fc86499-5rmb9" Oct 31 13:05:52.161733 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:52.144727 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2z8d\" (UniqueName: \"kubernetes.io/projected/df10a6fe-ce6b-4515-9264-79d420a3389a-kube-api-access-v2z8d\") pod \"payload-processing-e2e-praxis-b-3a3000-589587b7c5-nf4bj\" (UID: \"df10a6fe-ce6b-4515-9264-79d420a3389a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3a3000-589587b7c5-nf4bj" Oct 31 13:05:52.161733 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:52.144803 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df10a6fe-ce6b-4515-9264-79d420a3389a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-3a3000-589587b7c5-nf4bj\" (UID: \"df10a6fe-ce6b-4515-9264-79d420a3389a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3a3000-589587b7c5-nf4bj" Oct 31 13:05:52.161733 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:52.145802 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df10a6fe-ce6b-4515-9264-79d420a3389a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-3a3000-589587b7c5-nf4bj\" (UID: \"df10a6fe-ce6b-4515-9264-79d420a3389a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3a3000-589587b7c5-nf4bj" Oct 31 13:05:52.196934 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:52.196458 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2z8d\" (UniqueName: \"kubernetes.io/projected/df10a6fe-ce6b-4515-9264-79d420a3389a-kube-api-access-v2z8d\") pod \"payload-processing-e2e-praxis-b-3a3000-589587b7c5-nf4bj\" (UID: \"df10a6fe-ce6b-4515-9264-79d420a3389a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3a3000-589587b7c5-nf4bj" Oct 31 13:05:52.219286 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:52.219252 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3a3000-6c8fc86499-5rmb9" Oct 31 13:05:52.258847 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:52.258813 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3a3000-589587b7c5-nf4bj" Oct 31 13:05:52.521218 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:52.521189 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3a3000-6c8fc86499-5rmb9"] Oct 31 13:05:52.522242 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:05:52.522218 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod818a6fca_5402_45b0_9851_f31157caddc7.slice/crio-ae22805c38d0689f8ce4ed65a8ded941c5983523ba77a6e0f6123afaf187bec9 WatchSource:0}: Error finding container ae22805c38d0689f8ce4ed65a8ded941c5983523ba77a6e0f6123afaf187bec9: Status 404 returned error can't find the container with id ae22805c38d0689f8ce4ed65a8ded941c5983523ba77a6e0f6123afaf187bec9 Oct 31 13:05:52.573555 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:52.573526 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3a3000-589587b7c5-nf4bj"] Oct 31 13:05:52.576589 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:05:52.576560 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf10a6fe_ce6b_4515_9264_79d420a3389a.slice/crio-383af2dcc6cb32b3d1a462d82119088d3dc4e19d2eeaec0dbbce2fd91865d3c2 WatchSource:0}: Error finding container 383af2dcc6cb32b3d1a462d82119088d3dc4e19d2eeaec0dbbce2fd91865d3c2: Status 404 returned error can't find the container with id 383af2dcc6cb32b3d1a462d82119088d3dc4e19d2eeaec0dbbce2fd91865d3c2 Oct 31 13:05:53.218755 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:53.218710 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3a3000-6c8fc86499-5rmb9" event={"ID":"818a6fca-5402-45b0-9851-f31157caddc7","Type":"ContainerStarted","Data":"c5cd4cad4387f34c36d4936a5ab84ccc13933e7c312910198f511c23b02272bc"} Oct 31 13:05:53.219232 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:53.218786 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3a3000-6c8fc86499-5rmb9" Oct 31 13:05:53.219232 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:53.218808 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3a3000-6c8fc86499-5rmb9" event={"ID":"818a6fca-5402-45b0-9851-f31157caddc7","Type":"ContainerStarted","Data":"ae22805c38d0689f8ce4ed65a8ded941c5983523ba77a6e0f6123afaf187bec9"} Oct 31 13:05:53.220387 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:53.220364 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3a3000-589587b7c5-nf4bj" event={"ID":"df10a6fe-ce6b-4515-9264-79d420a3389a","Type":"ContainerStarted","Data":"8de5297ab1c162e575bc8301212d5deb158bfcd294e91d47a146191fdb62b0fb"} Oct 31 13:05:53.220519 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:53.220402 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3a3000-589587b7c5-nf4bj" event={"ID":"df10a6fe-ce6b-4515-9264-79d420a3389a","Type":"ContainerStarted","Data":"383af2dcc6cb32b3d1a462d82119088d3dc4e19d2eeaec0dbbce2fd91865d3c2"} Oct 31 13:05:53.220519 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:53.220456 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-3a3000-589587b7c5-nf4bj" Oct 31 13:05:53.239966 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:53.239918 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3a3000-6c8fc86499-5rmb9" podStartSLOduration=2.239904844 podStartE2EDuration="2.239904844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:05:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:05:53.236820482 +0000 UTC m=+2003.944406778" watchObservedRunningTime="2026-10-31 13:05:53.239904844 +0000 UTC m=+2003.947491140" Oct 31 13:05:53.257699 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:05:53.257629 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-3a3000-589587b7c5-nf4bj" podStartSLOduration=2.257617208 podStartE2EDuration="2.257617208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:05:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:05:53.255508127 +0000 UTC m=+2003.963094449" watchObservedRunningTime="2026-10-31 13:05:53.257617208 +0000 UTC m=+2003.965203502" Oct 31 13:06:04.233677 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:04.233628 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3a3000-6c8fc86499-5rmb9" Oct 31 13:06:04.234145 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:04.233720 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-3a3000-589587b7c5-nf4bj" Oct 31 13:06:05.473629 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:05.473591 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz"] Oct 31 13:06:05.479076 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:05.479054 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" Oct 31 13:06:05.494464 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:05.494435 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz"] Oct 31 13:06:05.653783 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:05.653754 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbec326f-9f8a-4e82-82de-21436c94736e-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz\" (UID: \"dbec326f-9f8a-4e82-82de-21436c94736e\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" Oct 31 13:06:05.653950 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:05.653904 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqzxx\" (UniqueName: \"kubernetes.io/projected/dbec326f-9f8a-4e82-82de-21436c94736e-kube-api-access-mqzxx\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz\" (UID: \"dbec326f-9f8a-4e82-82de-21436c94736e\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" Oct 31 13:06:05.654015 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:05.653997 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbec326f-9f8a-4e82-82de-21436c94736e-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz\" (UID: \"dbec326f-9f8a-4e82-82de-21436c94736e\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" Oct 31 13:06:05.654053 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:05.654023 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbec326f-9f8a-4e82-82de-21436c94736e-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz\" (UID: \"dbec326f-9f8a-4e82-82de-21436c94736e\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" Oct 31 13:06:05.654106 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:05.654093 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbec326f-9f8a-4e82-82de-21436c94736e-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz\" (UID: \"dbec326f-9f8a-4e82-82de-21436c94736e\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" Oct 31 13:06:05.654180 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:05.654130 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbec326f-9f8a-4e82-82de-21436c94736e-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz\" (UID: \"dbec326f-9f8a-4e82-82de-21436c94736e\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" Oct 31 13:06:05.654180 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:05.654157 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbec326f-9f8a-4e82-82de-21436c94736e-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz\" (UID: \"dbec326f-9f8a-4e82-82de-21436c94736e\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" Oct 31 13:06:05.654248 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:05.654189 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbec326f-9f8a-4e82-82de-21436c94736e-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz\" (UID: \"dbec326f-9f8a-4e82-82de-21436c94736e\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" Oct 31 13:06:05.654373 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:05.654348 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbec326f-9f8a-4e82-82de-21436c94736e-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz\" (UID: \"dbec326f-9f8a-4e82-82de-21436c94736e\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" Oct 31 13:06:05.756144 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:05.756051 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqzxx\" (UniqueName: \"kubernetes.io/projected/dbec326f-9f8a-4e82-82de-21436c94736e-kube-api-access-mqzxx\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz\" (UID: \"dbec326f-9f8a-4e82-82de-21436c94736e\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" Oct 31 13:06:05.756330 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:05.756162 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbec326f-9f8a-4e82-82de-21436c94736e-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz\" (UID: \"dbec326f-9f8a-4e82-82de-21436c94736e\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" Oct 31 13:06:05.756330 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:05.756185 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbec326f-9f8a-4e82-82de-21436c94736e-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz\" (UID: \"dbec326f-9f8a-4e82-82de-21436c94736e\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" Oct 31 13:06:05.756330 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:05.756239 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbec326f-9f8a-4e82-82de-21436c94736e-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz\" (UID: \"dbec326f-9f8a-4e82-82de-21436c94736e\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" Oct 31 13:06:05.756330 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:05.756274 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbec326f-9f8a-4e82-82de-21436c94736e-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz\" (UID: \"dbec326f-9f8a-4e82-82de-21436c94736e\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" Oct 31 13:06:05.756330 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:05.756305 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbec326f-9f8a-4e82-82de-21436c94736e-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz\" (UID: \"dbec326f-9f8a-4e82-82de-21436c94736e\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" Oct 31 13:06:05.756589 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:05.756350 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbec326f-9f8a-4e82-82de-21436c94736e-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz\" (UID: \"dbec326f-9f8a-4e82-82de-21436c94736e\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" Oct 31 13:06:05.756589 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:05.756459 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbec326f-9f8a-4e82-82de-21436c94736e-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz\" (UID: \"dbec326f-9f8a-4e82-82de-21436c94736e\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" Oct 31 13:06:05.756589 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:05.756501 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbec326f-9f8a-4e82-82de-21436c94736e-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz\" (UID: \"dbec326f-9f8a-4e82-82de-21436c94736e\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" Oct 31 13:06:05.756824 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:05.756796 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbec326f-9f8a-4e82-82de-21436c94736e-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz\" (UID: \"dbec326f-9f8a-4e82-82de-21436c94736e\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" Oct 31 13:06:05.757175 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:05.756837 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbec326f-9f8a-4e82-82de-21436c94736e-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz\" (UID: \"dbec326f-9f8a-4e82-82de-21436c94736e\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" Oct 31 13:06:05.757279 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:05.757021 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbec326f-9f8a-4e82-82de-21436c94736e-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz\" (UID: \"dbec326f-9f8a-4e82-82de-21436c94736e\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" Oct 31 13:06:05.757279 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:05.757184 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbec326f-9f8a-4e82-82de-21436c94736e-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz\" (UID: \"dbec326f-9f8a-4e82-82de-21436c94736e\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" Oct 31 13:06:05.757445 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:05.757424 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbec326f-9f8a-4e82-82de-21436c94736e-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz\" (UID: \"dbec326f-9f8a-4e82-82de-21436c94736e\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" Oct 31 13:06:05.758917 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:05.758887 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbec326f-9f8a-4e82-82de-21436c94736e-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz\" (UID: \"dbec326f-9f8a-4e82-82de-21436c94736e\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" Oct 31 13:06:05.759259 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:05.759239 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbec326f-9f8a-4e82-82de-21436c94736e-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz\" (UID: \"dbec326f-9f8a-4e82-82de-21436c94736e\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" Oct 31 13:06:05.768503 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:05.768477 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbec326f-9f8a-4e82-82de-21436c94736e-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz\" (UID: \"dbec326f-9f8a-4e82-82de-21436c94736e\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" Oct 31 13:06:05.769216 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:05.769191 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqzxx\" (UniqueName: \"kubernetes.io/projected/dbec326f-9f8a-4e82-82de-21436c94736e-kube-api-access-mqzxx\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz\" (UID: \"dbec326f-9f8a-4e82-82de-21436c94736e\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" Oct 31 13:06:05.795096 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:05.795066 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" Oct 31 13:06:06.083198 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:06.083175 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz"] Oct 31 13:06:06.086104 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:06:06.086064 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbec326f_9f8a_4e82_82de_21436c94736e.slice/crio-74d2fba0fb7a72eca4bba245164133319c997a3b596042eda99e2ec1c01ca7ca WatchSource:0}: Error finding container 74d2fba0fb7a72eca4bba245164133319c997a3b596042eda99e2ec1c01ca7ca: Status 404 returned error can't find the container with id 74d2fba0fb7a72eca4bba245164133319c997a3b596042eda99e2ec1c01ca7ca Oct 31 13:06:06.089019 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:06.088982 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:06:06.089120 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:06.089052 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:06:06.089120 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:06.089082 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:06:06.360897 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:06.360803 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" event={"ID":"dbec326f-9f8a-4e82-82de-21436c94736e","Type":"ContainerStarted","Data":"399af08f4b98592ca3b7f4841bba2e6ef9106a2961e29adbb01422c42339a459"} Oct 31 13:06:06.360897 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:06.360871 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" event={"ID":"dbec326f-9f8a-4e82-82de-21436c94736e","Type":"ContainerStarted","Data":"74d2fba0fb7a72eca4bba245164133319c997a3b596042eda99e2ec1c01ca7ca"} Oct 31 13:06:06.394825 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:06.394773 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" podStartSLOduration=1.394757746 podStartE2EDuration="1.394757746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:06:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:06:06.391276612 +0000 UTC m=+2017.098862908" watchObservedRunningTime="2026-10-31 13:06:06.394757746 +0000 UTC m=+2017.102344040" Oct 31 13:06:06.796084 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:06.796053 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" Oct 31 13:06:07.800990 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:07.800956 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 31 13:06:07.801585 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:07.801550 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" podUID="dbec326f-9f8a-4e82-82de-21436c94736e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 31 13:06:08.805794 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:08.805213 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 31 13:06:08.805794 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:08.805279 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" podUID="dbec326f-9f8a-4e82-82de-21436c94736e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 31 13:06:09.799074 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:09.797252 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 31 13:06:09.799074 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:09.797313 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" podUID="dbec326f-9f8a-4e82-82de-21436c94736e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 31 13:06:10.797666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:10.797527 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 31 13:06:10.797666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:10.797586 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" podUID="dbec326f-9f8a-4e82-82de-21436c94736e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 31 13:06:11.801062 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:11.801025 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 31 13:06:11.801568 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:11.801081 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" podUID="dbec326f-9f8a-4e82-82de-21436c94736e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 31 13:06:12.800671 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:12.798466 5693 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 31 13:06:12.800671 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:12.798529 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" podUID="dbec326f-9f8a-4e82-82de-21436c94736e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 31 13:06:13.842285 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:13.842242 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" Oct 31 13:06:13.842964 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:13.842942 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" Oct 31 13:06:13.843830 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:13.843809 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" Oct 31 13:06:18.688076 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:06:18.688040 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:06:22.568855 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:22.568821 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7e08dc-6cf96fcf89-pzv8w"] Oct 31 13:06:22.586109 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:22.586077 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7e08dc-7b446bdff9-vvsqh"] Oct 31 13:06:24.152573 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:24.152513 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6"] Oct 31 13:06:24.177299 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:24.177273 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" Oct 31 13:06:24.183913 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:24.181846 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-758f48-openshift-default-dockercfg-9d6f8\"" Oct 31 13:06:24.192474 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:24.192239 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6"] Oct 31 13:06:24.248856 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:24.248823 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c28ec7e-5395-4112-af19-7ea652fa1dae-istio-data\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6\" (UID: \"2c28ec7e-5395-4112-af19-7ea652fa1dae\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" Oct 31 13:06:24.249134 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:24.249094 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c28ec7e-5395-4112-af19-7ea652fa1dae-istio-podinfo\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6\" (UID: \"2c28ec7e-5395-4112-af19-7ea652fa1dae\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" Oct 31 13:06:24.252102 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:24.249240 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c28ec7e-5395-4112-af19-7ea652fa1dae-workload-certs\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6\" (UID: \"2c28ec7e-5395-4112-af19-7ea652fa1dae\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" Oct 31 13:06:24.253048 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:24.252761 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c28ec7e-5395-4112-af19-7ea652fa1dae-istio-envoy\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6\" (UID: \"2c28ec7e-5395-4112-af19-7ea652fa1dae\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" Oct 31 13:06:24.254845 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:24.254821 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c28ec7e-5395-4112-af19-7ea652fa1dae-workload-socket\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6\" (UID: \"2c28ec7e-5395-4112-af19-7ea652fa1dae\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" Oct 31 13:06:24.255076 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:24.255054 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c28ec7e-5395-4112-af19-7ea652fa1dae-credential-socket\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6\" (UID: \"2c28ec7e-5395-4112-af19-7ea652fa1dae\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" Oct 31 13:06:24.255212 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:24.255195 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgrmn\" (UniqueName: \"kubernetes.io/projected/2c28ec7e-5395-4112-af19-7ea652fa1dae-kube-api-access-xgrmn\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6\" (UID: \"2c28ec7e-5395-4112-af19-7ea652fa1dae\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" Oct 31 13:06:24.255526 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:24.255506 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c28ec7e-5395-4112-af19-7ea652fa1dae-istio-token\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6\" (UID: \"2c28ec7e-5395-4112-af19-7ea652fa1dae\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" Oct 31 13:06:24.255826 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:24.255794 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c28ec7e-5395-4112-af19-7ea652fa1dae-istiod-ca-cert\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6\" (UID: \"2c28ec7e-5395-4112-af19-7ea652fa1dae\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" Oct 31 13:06:24.360664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:24.357465 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c28ec7e-5395-4112-af19-7ea652fa1dae-workload-socket\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6\" (UID: \"2c28ec7e-5395-4112-af19-7ea652fa1dae\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" Oct 31 13:06:24.360664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:24.357545 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c28ec7e-5395-4112-af19-7ea652fa1dae-credential-socket\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6\" (UID: \"2c28ec7e-5395-4112-af19-7ea652fa1dae\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" Oct 31 13:06:24.360664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:24.357586 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgrmn\" (UniqueName: \"kubernetes.io/projected/2c28ec7e-5395-4112-af19-7ea652fa1dae-kube-api-access-xgrmn\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6\" (UID: \"2c28ec7e-5395-4112-af19-7ea652fa1dae\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" Oct 31 13:06:24.360664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:24.357702 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c28ec7e-5395-4112-af19-7ea652fa1dae-istio-token\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6\" (UID: \"2c28ec7e-5395-4112-af19-7ea652fa1dae\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" Oct 31 13:06:24.360664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:24.357754 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c28ec7e-5395-4112-af19-7ea652fa1dae-istiod-ca-cert\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6\" (UID: \"2c28ec7e-5395-4112-af19-7ea652fa1dae\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" Oct 31 13:06:24.360664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:24.357801 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c28ec7e-5395-4112-af19-7ea652fa1dae-istio-data\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6\" (UID: \"2c28ec7e-5395-4112-af19-7ea652fa1dae\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" Oct 31 13:06:24.360664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:24.357837 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c28ec7e-5395-4112-af19-7ea652fa1dae-istio-podinfo\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6\" (UID: \"2c28ec7e-5395-4112-af19-7ea652fa1dae\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" Oct 31 13:06:24.360664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:24.357873 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c28ec7e-5395-4112-af19-7ea652fa1dae-workload-certs\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6\" (UID: \"2c28ec7e-5395-4112-af19-7ea652fa1dae\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" Oct 31 13:06:24.360664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:24.358005 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c28ec7e-5395-4112-af19-7ea652fa1dae-istio-envoy\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6\" (UID: \"2c28ec7e-5395-4112-af19-7ea652fa1dae\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" Oct 31 13:06:24.361298 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:24.361099 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c28ec7e-5395-4112-af19-7ea652fa1dae-workload-socket\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6\" (UID: \"2c28ec7e-5395-4112-af19-7ea652fa1dae\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" Oct 31 13:06:24.361361 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:24.361337 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c28ec7e-5395-4112-af19-7ea652fa1dae-credential-socket\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6\" (UID: \"2c28ec7e-5395-4112-af19-7ea652fa1dae\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" Oct 31 13:06:24.366714 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:24.361813 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c28ec7e-5395-4112-af19-7ea652fa1dae-istio-data\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6\" (UID: \"2c28ec7e-5395-4112-af19-7ea652fa1dae\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" Oct 31 13:06:24.366714 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:24.362612 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c28ec7e-5395-4112-af19-7ea652fa1dae-istiod-ca-cert\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6\" (UID: \"2c28ec7e-5395-4112-af19-7ea652fa1dae\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" Oct 31 13:06:24.374204 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:24.374179 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c28ec7e-5395-4112-af19-7ea652fa1dae-istio-envoy\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6\" (UID: \"2c28ec7e-5395-4112-af19-7ea652fa1dae\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" Oct 31 13:06:24.377398 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:24.377347 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgrmn\" (UniqueName: \"kubernetes.io/projected/2c28ec7e-5395-4112-af19-7ea652fa1dae-kube-api-access-xgrmn\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6\" (UID: \"2c28ec7e-5395-4112-af19-7ea652fa1dae\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" Oct 31 13:06:24.377574 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:24.377418 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c28ec7e-5395-4112-af19-7ea652fa1dae-workload-certs\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6\" (UID: \"2c28ec7e-5395-4112-af19-7ea652fa1dae\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" Oct 31 13:06:24.378294 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:24.378248 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c28ec7e-5395-4112-af19-7ea652fa1dae-istio-podinfo\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6\" (UID: \"2c28ec7e-5395-4112-af19-7ea652fa1dae\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" Oct 31 13:06:24.382187 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:24.381047 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c28ec7e-5395-4112-af19-7ea652fa1dae-istio-token\") pod \"e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6\" (UID: \"2c28ec7e-5395-4112-af19-7ea652fa1dae\") " pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" Oct 31 13:06:24.508286 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:24.505909 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" Oct 31 13:06:25.773669 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:25.765096 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" podUID="e5a40b79-9879-4981-ab9f-446e1750a013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:25.777902 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:25.777614 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" podUID="710a52fa-2bc4-4bb9-96be-4b08dc60de67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:25.831673 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:25.808375 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:06:25.831673 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:25.808480 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:06:25.831673 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:25.808523 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:06:25.831673 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:25.831107 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" podUID="66c6ec5a-c0da-4bd7-a314-0c90ccec2425" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:26.057715 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:26.049107 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" event={"ID":"2c28ec7e-5395-4112-af19-7ea652fa1dae","Type":"ContainerStarted","Data":"38f723732f4012be7f98a6cbdc0a0f41196219e44d323b3a4984cb2de0ccf1dd"} Oct 31 13:06:26.057715 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:26.049181 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6"] Oct 31 13:06:26.103678 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:26.102742 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" podUID="17d56bfd-8cb1-42bf-bb11-05810c46d469" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:26.719665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:26.717980 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" podUID="ff9a9909-6fc3-475b-a9fe-562ec150c76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:26.816668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:26.807342 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" podUID="e5a40b79-9879-4981-ab9f-446e1750a013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:26.816668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:26.808065 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" Oct 31 13:06:26.898664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:26.852022 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" podUID="66c6ec5a-c0da-4bd7-a314-0c90ccec2425" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 13:06:26.898664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:26.852501 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" Oct 31 13:06:26.950665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:26.949830 5693 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" podUID="17d56bfd-8cb1-42bf-bb11-05810c46d469" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded" Oct 31 13:06:27.105028 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:27.104935 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" podUID="1a2abd3f-fb4f-4990-9a32-5f945a588068" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:27.258666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:27.258408 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" event={"ID":"2c28ec7e-5395-4112-af19-7ea652fa1dae","Type":"ContainerStarted","Data":"df438f11c0baebb51552ad4df5e9b40e940937ce9422161ebd4a683a1d9b6741"} Oct 31 13:06:27.314897 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:27.296842 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" podStartSLOduration=3.296820779 podStartE2EDuration="3.296820779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:06:27.294980533 +0000 UTC m=+2038.002566829" watchObservedRunningTime="2026-10-31 13:06:27.296820779 +0000 UTC m=+2038.004407076" Oct 31 13:06:27.365721 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:27.364477 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" podUID="f2b3766f-058c-40e7-9a2d-a7f5028e2b1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:27.626846 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:27.626715 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7e08dc-7b446bdff9-vvsqh" podUID="c1b76053-3fdc-49c0-97e1-cc142093b6b7" containerName="payload-pre-processing" containerID="cri-o://b2a399a55280496253eddcae956d0c6ccfc8501326c03df050106013707d7a56" gracePeriod=25 Oct 31 13:06:27.702580 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:27.627058 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-7e08dc-6cf96fcf89-pzv8w" podUID="2bf930ee-79a6-4b2e-a18e-77b4948aba78" containerName="payload-processing" containerID="cri-o://36ab8df3c04988a7464ac23f6b6b0eca69c10361abc7c5ea22ec9510151ffce6" gracePeriod=25 Oct 31 13:06:27.704334 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:27.704291 5693 patch_prober.go:29] interesting pod/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 31 13:06:27.704526 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:27.704492 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" podUID="2c28ec7e-5395-4112-af19-7ea652fa1dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 31 13:06:27.738667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:27.723241 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" podUID="ff9a9909-6fc3-475b-a9fe-562ec150c76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 31 13:06:27.945666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:27.927074 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" podUID="e5a40b79-9879-4981-ab9f-446e1750a013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:28.077947 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:28.077888 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" podUID="17d56bfd-8cb1-42bf-bb11-05810c46d469" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:28.352764 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:28.078195 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" podUID="8fb66a94-c399-43fb-945c-610ca63f0d76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 13:06:28.470856 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:28.470712 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" podUID="f2b3766f-058c-40e7-9a2d-a7f5028e2b1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Oct 31 13:06:28.471722 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:28.078500 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" podUID="710a52fa-2bc4-4bb9-96be-4b08dc60de67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 31 13:06:28.498192 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:28.498141 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" podUID="66c6ec5a-c0da-4bd7-a314-0c90ccec2425" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:28.740594 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:28.674967 5693 patch_prober.go:29] interesting pod/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 31 13:06:28.740594 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:28.675027 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" podUID="2c28ec7e-5395-4112-af19-7ea652fa1dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 31 13:06:28.793668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:28.775885 5693 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" podUID="f2b3766f-058c-40e7-9a2d-a7f5028e2b1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:28.897620 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:28.897577 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" podUID="1a2abd3f-fb4f-4990-9a32-5f945a588068" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 31 13:06:28.928841 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:28.928742 5693 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" podUID="710a52fa-2bc4-4bb9-96be-4b08dc60de67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:28.930764 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:28.930721 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" podUID="e5a40b79-9879-4981-ab9f-446e1750a013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 13:06:29.043676 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:06:29.019826 5693 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.611s" Oct 31 13:06:29.048739 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:29.048709 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" Oct 31 13:06:29.050876 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:29.050842 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" Oct 31 13:06:29.056653 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:29.051364 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" Oct 31 13:06:29.056653 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:29.051490 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" Oct 31 13:06:29.056653 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:29.051537 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" Oct 31 13:06:29.056653 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:29.051570 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" Oct 31 13:06:29.079749 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:29.067234 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" Oct 31 13:06:29.079749 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:29.068519 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" Oct 31 13:06:29.079749 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:29.074737 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" Oct 31 13:06:29.567322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:29.566435 5693 patch_prober.go:29] interesting pod/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 31 13:06:29.567322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:29.566489 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" podUID="2c28ec7e-5395-4112-af19-7ea652fa1dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 31 13:06:29.756439 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:29.756361 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7e08dc-7b446bdff9-vvsqh" Oct 31 13:06:30.063531 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:30.063461 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1b76053-3fdc-49c0-97e1-cc142093b6b7-kube-api-access-nd465\" (UniqueName: \"kubernetes.io/projected/c1b76053-3fdc-49c0-97e1-cc142093b6b7-kube-api-access-nd465\") pod \"c1b76053-3fdc-49c0-97e1-cc142093b6b7\" (UID: \"c1b76053-3fdc-49c0-97e1-cc142093b6b7\") " Oct 31 13:06:30.064011 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:30.063992 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1b76053-3fdc-49c0-97e1-cc142093b6b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b76053-3fdc-49c0-97e1-cc142093b6b7-plugins-config-volume\") pod \"c1b76053-3fdc-49c0-97e1-cc142093b6b7\" (UID: \"c1b76053-3fdc-49c0-97e1-cc142093b6b7\") " Oct 31 13:06:30.064919 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:30.064894 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1b76053-3fdc-49c0-97e1-cc142093b6b7-plugins-config-volume" pod "c1b76053-3fdc-49c0-97e1-cc142093b6b7" (UID: "c1b76053-3fdc-49c0-97e1-cc142093b6b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:06:30.065478 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:30.065453 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b76053-3fdc-49c0-97e1-cc142093b6b7-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:30.073150 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:30.073021 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1b76053-3fdc-49c0-97e1-cc142093b6b7-kube-api-access-nd465" pod "c1b76053-3fdc-49c0-97e1-cc142093b6b7" (UID: "c1b76053-3fdc-49c0-97e1-cc142093b6b7"). InnerVolumeSpecName "kube-api-access-nd465". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:06:30.089724 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:30.089671 5693 generic.go:356] "Generic (PLEG): container finished" podID="2bf930ee-79a6-4b2e-a18e-77b4948aba78" containerID="36ab8df3c04988a7464ac23f6b6b0eca69c10361abc7c5ea22ec9510151ffce6" exitCode=0 Oct 31 13:06:30.117186 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:30.117153 5693 generic.go:356] "Generic (PLEG): container finished" podID="c1b76053-3fdc-49c0-97e1-cc142093b6b7" containerID="b2a399a55280496253eddcae956d0c6ccfc8501326c03df050106013707d7a56" exitCode=0 Oct 31 13:06:30.118783 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:30.118760 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7e08dc-7b446bdff9-vvsqh" Oct 31 13:06:30.142200 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:30.142161 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-7e08dc-6cf96fcf89-pzv8w" event={"ID":"2bf930ee-79a6-4b2e-a18e-77b4948aba78","Type":"ContainerDied","Data":"36ab8df3c04988a7464ac23f6b6b0eca69c10361abc7c5ea22ec9510151ffce6"} Oct 31 13:06:30.143185 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:30.143158 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-7e08dc-6cf96fcf89-pzv8w" event={"ID":"2bf930ee-79a6-4b2e-a18e-77b4948aba78","Type":"ContainerDied","Data":"ee120b7c80ab9f77f26d3f60733891ea5f7e42988d2a59fe26f83dc78d126276"} Oct 31 13:06:30.143357 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:30.143344 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee120b7c80ab9f77f26d3f60733891ea5f7e42988d2a59fe26f83dc78d126276" Oct 31 13:06:30.143454 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:30.143438 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7e08dc-7b446bdff9-vvsqh" event={"ID":"c1b76053-3fdc-49c0-97e1-cc142093b6b7","Type":"ContainerDied","Data":"b2a399a55280496253eddcae956d0c6ccfc8501326c03df050106013707d7a56"} Oct 31 13:06:30.143557 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:30.143545 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7e08dc-7b446bdff9-vvsqh" event={"ID":"c1b76053-3fdc-49c0-97e1-cc142093b6b7","Type":"ContainerDied","Data":"d2fdb7b7147e6b78eb4ad7a1c791d98e2ee3271461d0bed08b030e2a7d0bd9e6"} Oct 31 13:06:30.143725 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:30.143631 5693 scope.go:122] "RemoveContainer" containerID="b2a399a55280496253eddcae956d0c6ccfc8501326c03df050106013707d7a56" Oct 31 13:06:30.150482 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:30.150173 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-7e08dc-6cf96fcf89-pzv8w" Oct 31 13:06:30.164617 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:30.162677 5693 scope.go:122] "RemoveContainer" containerID="b2a399a55280496253eddcae956d0c6ccfc8501326c03df050106013707d7a56" Oct 31 13:06:30.164617 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:06:30.162970 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2a399a55280496253eddcae956d0c6ccfc8501326c03df050106013707d7a56\": container with ID starting with b2a399a55280496253eddcae956d0c6ccfc8501326c03df050106013707d7a56 not found: ID does not exist" containerID="b2a399a55280496253eddcae956d0c6ccfc8501326c03df050106013707d7a56" Oct 31 13:06:30.164617 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:30.163003 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2a399a55280496253eddcae956d0c6ccfc8501326c03df050106013707d7a56"} err="failed to get container status \"b2a399a55280496253eddcae956d0c6ccfc8501326c03df050106013707d7a56\": rpc error: code = NotFound desc = could not find container \"b2a399a55280496253eddcae956d0c6ccfc8501326c03df050106013707d7a56\": container with ID starting with b2a399a55280496253eddcae956d0c6ccfc8501326c03df050106013707d7a56 not found: ID does not exist" Oct 31 13:06:30.166347 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:30.166329 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bf930ee-79a6-4b2e-a18e-77b4948aba78-kube-api-access-szqsg\" (UniqueName: \"kubernetes.io/projected/2bf930ee-79a6-4b2e-a18e-77b4948aba78-kube-api-access-szqsg\") pod \"2bf930ee-79a6-4b2e-a18e-77b4948aba78\" (UID: \"2bf930ee-79a6-4b2e-a18e-77b4948aba78\") " Oct 31 13:06:30.166443 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:30.166369 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bf930ee-79a6-4b2e-a18e-77b4948aba78-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bf930ee-79a6-4b2e-a18e-77b4948aba78-plugins-config-volume\") pod \"2bf930ee-79a6-4b2e-a18e-77b4948aba78\" (UID: \"2bf930ee-79a6-4b2e-a18e-77b4948aba78\") " Oct 31 13:06:30.166826 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:30.166799 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bf930ee-79a6-4b2e-a18e-77b4948aba78-plugins-config-volume" pod "2bf930ee-79a6-4b2e-a18e-77b4948aba78" (UID: "2bf930ee-79a6-4b2e-a18e-77b4948aba78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:06:30.167376 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:30.167349 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bf930ee-79a6-4b2e-a18e-77b4948aba78-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:30.167476 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:30.167383 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd465\" (UniqueName: \"kubernetes.io/projected/c1b76053-3fdc-49c0-97e1-cc142093b6b7-kube-api-access-nd465\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:30.168599 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:30.168576 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bf930ee-79a6-4b2e-a18e-77b4948aba78-kube-api-access-szqsg" pod "2bf930ee-79a6-4b2e-a18e-77b4948aba78" (UID: "2bf930ee-79a6-4b2e-a18e-77b4948aba78"). InnerVolumeSpecName "kube-api-access-szqsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:06:30.234760 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:30.234723 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7e08dc-7b446bdff9-vvsqh"] Oct 31 13:06:30.239208 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:30.239173 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7e08dc-7b446bdff9-vvsqh"] Oct 31 13:06:30.268344 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:30.268314 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szqsg\" (UniqueName: \"kubernetes.io/projected/2bf930ee-79a6-4b2e-a18e-77b4948aba78-kube-api-access-szqsg\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:30.515665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:30.515042 5693 patch_prober.go:29] interesting pod/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 31 13:06:30.515665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:30.515139 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" podUID="2c28ec7e-5395-4112-af19-7ea652fa1dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 31 13:06:31.138546 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:31.138517 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-7e08dc-6cf96fcf89-pzv8w" Oct 31 13:06:31.163541 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:31.163515 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7e08dc-6cf96fcf89-pzv8w"] Oct 31 13:06:31.167298 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:31.167276 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7e08dc-6cf96fcf89-pzv8w"] Oct 31 13:06:31.437504 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:31.436935 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bf930ee-79a6-4b2e-a18e-77b4948aba78" path="/var/lib/kubelet/pods/2bf930ee-79a6-4b2e-a18e-77b4948aba78/volumes" Oct 31 13:06:31.437746 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:31.437633 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1b76053-3fdc-49c0-97e1-cc142093b6b7" path="/var/lib/kubelet/pods/c1b76053-3fdc-49c0-97e1-cc142093b6b7/volumes" Oct 31 13:06:31.510667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:31.507055 5693 patch_prober.go:29] interesting pod/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 31 13:06:31.510667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:31.507123 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" podUID="2c28ec7e-5395-4112-af19-7ea652fa1dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 31 13:06:32.518307 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:32.516655 5693 patch_prober.go:29] interesting pod/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 31 13:06:32.518307 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:32.516728 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" podUID="2c28ec7e-5395-4112-af19-7ea652fa1dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 31 13:06:33.506568 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:33.506532 5693 patch_prober.go:29] interesting pod/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 31 13:06:33.506886 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:33.506589 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" podUID="2c28ec7e-5395-4112-af19-7ea652fa1dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 31 13:06:34.511982 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:34.511931 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" Oct 31 13:06:34.512484 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:34.512328 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" Oct 31 13:06:34.513237 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:34.513216 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" Oct 31 13:06:35.238829 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:35.238796 5693 patch_prober.go:29] interesting pod/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:06:35.239021 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:35.238849 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" podUID="9d0d805e-ce8f-4cd0-91c1-7f00fa21e672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:37.234682 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:37.234626 5693 generic.go:356] "Generic (PLEG): container finished" podID="9d0d805e-ce8f-4cd0-91c1-7f00fa21e672" containerID="6544623a77af546e84f159923833b0a89c99167c9683d4a2872354068bc62f83" exitCode=0 Oct 31 13:06:37.235119 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:37.234696 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" event={"ID":"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672","Type":"ContainerDied","Data":"6544623a77af546e84f159923833b0a89c99167c9683d4a2872354068bc62f83"} Oct 31 13:06:37.235175 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:37.235160 5693 scope.go:122] "RemoveContainer" containerID="6544623a77af546e84f159923833b0a89c99167c9683d4a2872354068bc62f83" Oct 31 13:06:37.236764 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:37.236738 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:06:37.236848 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:37.236805 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:06:37.236848 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:37.236839 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:06:38.245738 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:38.245696 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" event={"ID":"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672","Type":"ContainerStarted","Data":"dc268220a5bc0dc0c97a46437a5734381bcd5cff5b7bb16da5a9a62bf3074416"} Oct 31 13:06:38.871025 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:38.870993 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" Oct 31 13:06:38.871226 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:38.871149 5693 patch_prober.go:29] interesting pod/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 31 13:06:38.871226 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:38.871205 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" podUID="9d0d805e-ce8f-4cd0-91c1-7f00fa21e672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 31 13:06:39.696667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:39.676128 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3a3000-589587b7c5-nf4bj"] Oct 31 13:06:39.709043 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:39.709002 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3a3000-6c8fc86499-5rmb9"] Oct 31 13:06:39.870999 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:39.870963 5693 patch_prober.go:29] interesting pod/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 31 13:06:39.871166 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:39.871031 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" podUID="9d0d805e-ce8f-4cd0-91c1-7f00fa21e672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 31 13:06:40.870851 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:40.870812 5693 patch_prober.go:29] interesting pod/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 31 13:06:40.871241 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:40.870876 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" podUID="9d0d805e-ce8f-4cd0-91c1-7f00fa21e672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 31 13:06:41.870985 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:41.870951 5693 patch_prober.go:29] interesting pod/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 31 13:06:41.871388 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:41.871002 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" podUID="9d0d805e-ce8f-4cd0-91c1-7f00fa21e672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 31 13:06:42.870956 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:42.870917 5693 patch_prober.go:29] interesting pod/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 31 13:06:42.871151 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:42.871000 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" podUID="9d0d805e-ce8f-4cd0-91c1-7f00fa21e672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 31 13:06:43.874010 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:43.873978 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" Oct 31 13:06:43.874389 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:43.874328 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" Oct 31 13:06:43.875013 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:43.874994 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" Oct 31 13:06:44.700662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:44.699621 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-3a3000-589587b7c5-nf4bj" podUID="df10a6fe-ce6b-4515-9264-79d420a3389a" containerName="payload-processing" containerID="cri-o://8de5297ab1c162e575bc8301212d5deb158bfcd294e91d47a146191fdb62b0fb" gracePeriod=25 Oct 31 13:06:44.710151 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:44.710085 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3a3000-6c8fc86499-5rmb9" podUID="818a6fca-5402-45b0-9851-f31157caddc7" containerName="payload-pre-processing" containerID="cri-o://c5cd4cad4387f34c36d4936a5ab84ccc13933e7c312910198f511c23b02272bc" gracePeriod=25 Oct 31 13:06:45.006830 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:45.006660 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3a3000-589587b7c5-nf4bj" Oct 31 13:06:45.051151 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:45.051122 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3a3000-6c8fc86499-5rmb9" Oct 31 13:06:45.086705 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:45.086670 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df10a6fe-ce6b-4515-9264-79d420a3389a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df10a6fe-ce6b-4515-9264-79d420a3389a-plugins-config-volume\") pod \"df10a6fe-ce6b-4515-9264-79d420a3389a\" (UID: \"df10a6fe-ce6b-4515-9264-79d420a3389a\") " Oct 31 13:06:45.086914 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:45.086894 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/818a6fca-5402-45b0-9851-f31157caddc7-kube-api-access-hc77g\" (UniqueName: \"kubernetes.io/projected/818a6fca-5402-45b0-9851-f31157caddc7-kube-api-access-hc77g\") pod \"818a6fca-5402-45b0-9851-f31157caddc7\" (UID: \"818a6fca-5402-45b0-9851-f31157caddc7\") " Oct 31 13:06:45.086986 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:45.086970 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/818a6fca-5402-45b0-9851-f31157caddc7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/818a6fca-5402-45b0-9851-f31157caddc7-plugins-config-volume\") pod \"818a6fca-5402-45b0-9851-f31157caddc7\" (UID: \"818a6fca-5402-45b0-9851-f31157caddc7\") " Oct 31 13:06:45.087062 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:45.087030 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df10a6fe-ce6b-4515-9264-79d420a3389a-kube-api-access-v2z8d\" (UniqueName: \"kubernetes.io/projected/df10a6fe-ce6b-4515-9264-79d420a3389a-kube-api-access-v2z8d\") pod \"df10a6fe-ce6b-4515-9264-79d420a3389a\" (UID: \"df10a6fe-ce6b-4515-9264-79d420a3389a\") " Oct 31 13:06:45.088571 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:45.088544 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df10a6fe-ce6b-4515-9264-79d420a3389a-plugins-config-volume" pod "df10a6fe-ce6b-4515-9264-79d420a3389a" (UID: "df10a6fe-ce6b-4515-9264-79d420a3389a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:06:45.088924 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:45.088902 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/818a6fca-5402-45b0-9851-f31157caddc7-plugins-config-volume" pod "818a6fca-5402-45b0-9851-f31157caddc7" (UID: "818a6fca-5402-45b0-9851-f31157caddc7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:06:45.090929 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:45.090903 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df10a6fe-ce6b-4515-9264-79d420a3389a-kube-api-access-v2z8d" pod "df10a6fe-ce6b-4515-9264-79d420a3389a" (UID: "df10a6fe-ce6b-4515-9264-79d420a3389a"). InnerVolumeSpecName "kube-api-access-v2z8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:06:45.092032 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:45.092009 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/818a6fca-5402-45b0-9851-f31157caddc7-kube-api-access-hc77g" pod "818a6fca-5402-45b0-9851-f31157caddc7" (UID: "818a6fca-5402-45b0-9851-f31157caddc7"). InnerVolumeSpecName "kube-api-access-hc77g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:06:45.189179 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:45.189144 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc77g\" (UniqueName: \"kubernetes.io/projected/818a6fca-5402-45b0-9851-f31157caddc7-kube-api-access-hc77g\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:45.189179 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:45.189174 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/818a6fca-5402-45b0-9851-f31157caddc7-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:45.189179 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:45.189184 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2z8d\" (UniqueName: \"kubernetes.io/projected/df10a6fe-ce6b-4515-9264-79d420a3389a-kube-api-access-v2z8d\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:45.189408 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:45.189193 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df10a6fe-ce6b-4515-9264-79d420a3389a-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:45.384301 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:45.384200 5693 generic.go:356] "Generic (PLEG): container finished" podID="818a6fca-5402-45b0-9851-f31157caddc7" containerID="c5cd4cad4387f34c36d4936a5ab84ccc13933e7c312910198f511c23b02272bc" exitCode=0 Oct 31 13:06:45.384492 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:45.384364 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3a3000-6c8fc86499-5rmb9" event={"ID":"818a6fca-5402-45b0-9851-f31157caddc7","Type":"ContainerDied","Data":"c5cd4cad4387f34c36d4936a5ab84ccc13933e7c312910198f511c23b02272bc"} Oct 31 13:06:45.384492 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:45.384419 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3a3000-6c8fc86499-5rmb9" event={"ID":"818a6fca-5402-45b0-9851-f31157caddc7","Type":"ContainerDied","Data":"ae22805c38d0689f8ce4ed65a8ded941c5983523ba77a6e0f6123afaf187bec9"} Oct 31 13:06:45.384492 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:45.384449 5693 scope.go:122] "RemoveContainer" containerID="c5cd4cad4387f34c36d4936a5ab84ccc13933e7c312910198f511c23b02272bc" Oct 31 13:06:45.384714 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:45.384610 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3a3000-6c8fc86499-5rmb9" Oct 31 13:06:45.412168 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:45.409028 5693 generic.go:356] "Generic (PLEG): container finished" podID="df10a6fe-ce6b-4515-9264-79d420a3389a" containerID="8de5297ab1c162e575bc8301212d5deb158bfcd294e91d47a146191fdb62b0fb" exitCode=0 Oct 31 13:06:45.414676 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:45.412708 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3a3000-589587b7c5-nf4bj" Oct 31 13:06:45.437891 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:45.437864 5693 scope.go:122] "RemoveContainer" containerID="c5cd4cad4387f34c36d4936a5ab84ccc13933e7c312910198f511c23b02272bc" Oct 31 13:06:45.439736 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:06:45.439704 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5cd4cad4387f34c36d4936a5ab84ccc13933e7c312910198f511c23b02272bc\": container with ID starting with c5cd4cad4387f34c36d4936a5ab84ccc13933e7c312910198f511c23b02272bc not found: ID does not exist" containerID="c5cd4cad4387f34c36d4936a5ab84ccc13933e7c312910198f511c23b02272bc" Oct 31 13:06:45.439849 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:45.439746 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5cd4cad4387f34c36d4936a5ab84ccc13933e7c312910198f511c23b02272bc"} err="failed to get container status \"c5cd4cad4387f34c36d4936a5ab84ccc13933e7c312910198f511c23b02272bc\": rpc error: code = NotFound desc = could not find container \"c5cd4cad4387f34c36d4936a5ab84ccc13933e7c312910198f511c23b02272bc\": container with ID starting with c5cd4cad4387f34c36d4936a5ab84ccc13933e7c312910198f511c23b02272bc not found: ID does not exist" Oct 31 13:06:45.446457 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:45.446421 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3a3000-589587b7c5-nf4bj" event={"ID":"df10a6fe-ce6b-4515-9264-79d420a3389a","Type":"ContainerDied","Data":"8de5297ab1c162e575bc8301212d5deb158bfcd294e91d47a146191fdb62b0fb"} Oct 31 13:06:45.446571 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:45.446458 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3a3000-6c8fc86499-5rmb9"] Oct 31 13:06:45.446571 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:45.446490 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3a3000-6c8fc86499-5rmb9"] Oct 31 13:06:45.446571 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:45.446512 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3a3000-589587b7c5-nf4bj" event={"ID":"df10a6fe-ce6b-4515-9264-79d420a3389a","Type":"ContainerDied","Data":"383af2dcc6cb32b3d1a462d82119088d3dc4e19d2eeaec0dbbce2fd91865d3c2"} Oct 31 13:06:45.446571 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:45.446532 5693 scope.go:122] "RemoveContainer" containerID="8de5297ab1c162e575bc8301212d5deb158bfcd294e91d47a146191fdb62b0fb" Oct 31 13:06:45.462553 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:45.461807 5693 scope.go:122] "RemoveContainer" containerID="8de5297ab1c162e575bc8301212d5deb158bfcd294e91d47a146191fdb62b0fb" Oct 31 13:06:45.467905 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:06:45.467850 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8de5297ab1c162e575bc8301212d5deb158bfcd294e91d47a146191fdb62b0fb\": container with ID starting with 8de5297ab1c162e575bc8301212d5deb158bfcd294e91d47a146191fdb62b0fb not found: ID does not exist" containerID="8de5297ab1c162e575bc8301212d5deb158bfcd294e91d47a146191fdb62b0fb" Oct 31 13:06:45.468073 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:45.467895 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8de5297ab1c162e575bc8301212d5deb158bfcd294e91d47a146191fdb62b0fb"} err="failed to get container status \"8de5297ab1c162e575bc8301212d5deb158bfcd294e91d47a146191fdb62b0fb\": rpc error: code = NotFound desc = could not find container \"8de5297ab1c162e575bc8301212d5deb158bfcd294e91d47a146191fdb62b0fb\": container with ID starting with 8de5297ab1c162e575bc8301212d5deb158bfcd294e91d47a146191fdb62b0fb not found: ID does not exist" Oct 31 13:06:45.484307 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:45.477985 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3a3000-589587b7c5-nf4bj"] Oct 31 13:06:45.517669 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:45.515761 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3a3000-589587b7c5-nf4bj"] Oct 31 13:06:46.449124 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:46.449088 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5"] Oct 31 13:06:46.449620 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:46.449319 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5" podUID="416c552c-b82d-472a-8f81-ac404b2c8a31" containerName="maas-api" containerID="cri-o://f6f456283f425d33aed2c8186a59668730838c2ddd9af1ca170505245de5cf3c" gracePeriod=30 Oct 31 13:06:46.743831 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:46.743803 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5" Oct 31 13:06:46.834802 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:46.830752 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/416c552c-b82d-472a-8f81-ac404b2c8a31-kube-api-access-26hhx\" (UniqueName: \"kubernetes.io/projected/416c552c-b82d-472a-8f81-ac404b2c8a31-kube-api-access-26hhx\") pod \"416c552c-b82d-472a-8f81-ac404b2c8a31\" (UID: \"416c552c-b82d-472a-8f81-ac404b2c8a31\") " Oct 31 13:06:46.834802 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:46.830878 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/416c552c-b82d-472a-8f81-ac404b2c8a31-metrics-certs\" (UniqueName: \"kubernetes.io/secret/416c552c-b82d-472a-8f81-ac404b2c8a31-metrics-certs\") pod \"416c552c-b82d-472a-8f81-ac404b2c8a31\" (UID: \"416c552c-b82d-472a-8f81-ac404b2c8a31\") " Oct 31 13:06:46.834802 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:46.830978 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/416c552c-b82d-472a-8f81-ac404b2c8a31-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/416c552c-b82d-472a-8f81-ac404b2c8a31-openshift-service-ca\") pod \"416c552c-b82d-472a-8f81-ac404b2c8a31\" (UID: \"416c552c-b82d-472a-8f81-ac404b2c8a31\") " Oct 31 13:06:46.834802 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:46.831027 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/416c552c-b82d-472a-8f81-ac404b2c8a31-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/416c552c-b82d-472a-8f81-ac404b2c8a31-maas-api-tls\") pod \"416c552c-b82d-472a-8f81-ac404b2c8a31\" (UID: \"416c552c-b82d-472a-8f81-ac404b2c8a31\") " Oct 31 13:06:46.867194 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:46.867151 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/416c552c-b82d-472a-8f81-ac404b2c8a31-openshift-service-ca" pod "416c552c-b82d-472a-8f81-ac404b2c8a31" (UID: "416c552c-b82d-472a-8f81-ac404b2c8a31"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:06:46.934662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:46.933443 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/416c552c-b82d-472a-8f81-ac404b2c8a31-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:46.956496 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:46.955991 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/416c552c-b82d-472a-8f81-ac404b2c8a31-kube-api-access-26hhx" pod "416c552c-b82d-472a-8f81-ac404b2c8a31" (UID: "416c552c-b82d-472a-8f81-ac404b2c8a31"). InnerVolumeSpecName "kube-api-access-26hhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:06:46.958009 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:46.957957 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/416c552c-b82d-472a-8f81-ac404b2c8a31-metrics-certs" pod "416c552c-b82d-472a-8f81-ac404b2c8a31" (UID: "416c552c-b82d-472a-8f81-ac404b2c8a31"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:06:46.985069 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:46.985035 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/416c552c-b82d-472a-8f81-ac404b2c8a31-maas-api-tls" pod "416c552c-b82d-472a-8f81-ac404b2c8a31" (UID: "416c552c-b82d-472a-8f81-ac404b2c8a31"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:06:47.071080 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:47.071042 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26hhx\" (UniqueName: \"kubernetes.io/projected/416c552c-b82d-472a-8f81-ac404b2c8a31-kube-api-access-26hhx\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:47.090725 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:47.090692 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/416c552c-b82d-472a-8f81-ac404b2c8a31-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:47.090969 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:47.090953 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/416c552c-b82d-472a-8f81-ac404b2c8a31-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:47.414482 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:47.414441 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="818a6fca-5402-45b0-9851-f31157caddc7" path="/var/lib/kubelet/pods/818a6fca-5402-45b0-9851-f31157caddc7/volumes" Oct 31 13:06:47.415149 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:47.415123 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df10a6fe-ce6b-4515-9264-79d420a3389a" path="/var/lib/kubelet/pods/df10a6fe-ce6b-4515-9264-79d420a3389a/volumes" Oct 31 13:06:47.431443 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:47.431403 5693 generic.go:356] "Generic (PLEG): container finished" podID="416c552c-b82d-472a-8f81-ac404b2c8a31" containerID="f6f456283f425d33aed2c8186a59668730838c2ddd9af1ca170505245de5cf3c" exitCode=0 Oct 31 13:06:47.431557 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:47.431469 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5" event={"ID":"416c552c-b82d-472a-8f81-ac404b2c8a31","Type":"ContainerDied","Data":"f6f456283f425d33aed2c8186a59668730838c2ddd9af1ca170505245de5cf3c"} Oct 31 13:06:47.431557 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:47.431521 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5" event={"ID":"416c552c-b82d-472a-8f81-ac404b2c8a31","Type":"ContainerDied","Data":"bbbcaa15efda2e4956a5483c49c4b4e6f19dbae419f9d07714d6b5eb2227c969"} Oct 31 13:06:47.431557 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:47.431552 5693 scope.go:122] "RemoveContainer" containerID="f6f456283f425d33aed2c8186a59668730838c2ddd9af1ca170505245de5cf3c" Oct 31 13:06:47.431852 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:47.431676 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5" Oct 31 13:06:47.468496 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:47.468310 5693 scope.go:122] "RemoveContainer" containerID="f6f456283f425d33aed2c8186a59668730838c2ddd9af1ca170505245de5cf3c" Oct 31 13:06:47.469578 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:06:47.469542 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6f456283f425d33aed2c8186a59668730838c2ddd9af1ca170505245de5cf3c\": container with ID starting with f6f456283f425d33aed2c8186a59668730838c2ddd9af1ca170505245de5cf3c not found: ID does not exist" containerID="f6f456283f425d33aed2c8186a59668730838c2ddd9af1ca170505245de5cf3c" Oct 31 13:06:47.470168 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:47.470137 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6f456283f425d33aed2c8186a59668730838c2ddd9af1ca170505245de5cf3c"} err="failed to get container status \"f6f456283f425d33aed2c8186a59668730838c2ddd9af1ca170505245de5cf3c\": rpc error: code = NotFound desc = could not find container \"f6f456283f425d33aed2c8186a59668730838c2ddd9af1ca170505245de5cf3c\": container with ID starting with f6f456283f425d33aed2c8186a59668730838c2ddd9af1ca170505245de5cf3c not found: ID does not exist" Oct 31 13:06:47.594668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:47.496694 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5"] Oct 31 13:06:47.594668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:47.501593 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4989f9-9c9bf9f4f-qxtw5"] Oct 31 13:06:48.521665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:48.517891 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" podUID="66c6ec5a-c0da-4bd7-a314-0c90ccec2425" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 13:06:48.572504 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:48.572460 5693 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" podUID="f2b3766f-058c-40e7-9a2d-a7f5028e2b1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": context deadline exceeded" Oct 31 13:06:48.793233 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:48.793130 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" podUID="8fb66a94-c399-43fb-945c-610ca63f0d76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:48.920667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:48.914740 5693 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm" podUID="66c6ec5a-c0da-4bd7-a314-0c90ccec2425" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:48.950678 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:06:48.949850 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:06:48.950678 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:48.950350 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" podUID="e5a40b79-9879-4981-ab9f-446e1750a013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:49.080816 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:49.080718 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" podUID="17d56bfd-8cb1-42bf-bb11-05810c46d469" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:49.082093 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:49.082064 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" podUID="1a2abd3f-fb4f-4990-9a32-5f945a588068" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:49.335715 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:49.333156 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" podUID="f2b3766f-058c-40e7-9a2d-a7f5028e2b1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:49.428194 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:49.427371 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="416c552c-b82d-472a-8f81-ac404b2c8a31" path="/var/lib/kubelet/pods/416c552c-b82d-472a-8f81-ac404b2c8a31/volumes" Oct 31 13:06:49.787671 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:49.786779 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h"] Oct 31 13:06:49.787671 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:49.787014 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" podUID="9d0d805e-ce8f-4cd0-91c1-7f00fa21e672" containerName="istio-proxy" containerID="cri-o://dc268220a5bc0dc0c97a46437a5734381bcd5cff5b7bb16da5a9a62bf3074416" gracePeriod=30 Oct 31 13:06:54.434234 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:54.434197 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-758f48-6456c959dc-665vf"] Oct 31 13:06:54.454963 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:54.454932 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-758f48-6785db86d7-j4phq"] Oct 31 13:06:55.100701 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:55.097718 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" Oct 31 13:06:55.246468 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:55.246431 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-workload-socket\") pod \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\" (UID: \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\") " Oct 31 13:06:55.246668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:55.246488 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-istio-data\") pod \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\" (UID: \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\") " Oct 31 13:06:55.246668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:55.246532 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-istio-envoy\") pod \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\" (UID: \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\") " Oct 31 13:06:55.246668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:55.246590 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-credential-socket\") pod \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\" (UID: \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\") " Oct 31 13:06:55.246668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:55.246635 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-istiod-ca-cert\") pod \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\" (UID: \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\") " Oct 31 13:06:55.246939 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:55.246679 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-kube-api-access-f29vp\" (UniqueName: \"kubernetes.io/projected/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-kube-api-access-f29vp\") pod \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\" (UID: \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\") " Oct 31 13:06:55.246939 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:55.246768 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-workload-socket" pod "9d0d805e-ce8f-4cd0-91c1-7f00fa21e672" (UID: "9d0d805e-ce8f-4cd0-91c1-7f00fa21e672"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:06:55.247144 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:55.247115 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-istio-token\" (UniqueName: \"kubernetes.io/projected/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-istio-token\") pod \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\" (UID: \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\") " Oct 31 13:06:55.247270 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:55.247167 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-workload-certs\") pod \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\" (UID: \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\") " Oct 31 13:06:55.247270 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:55.247200 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-istio-podinfo\") pod \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\" (UID: \"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672\") " Oct 31 13:06:55.247444 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:55.247370 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-istio-data" pod "9d0d805e-ce8f-4cd0-91c1-7f00fa21e672" (UID: "9d0d805e-ce8f-4cd0-91c1-7f00fa21e672"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:06:55.247589 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:55.247568 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-workload-certs" pod "9d0d805e-ce8f-4cd0-91c1-7f00fa21e672" (UID: "9d0d805e-ce8f-4cd0-91c1-7f00fa21e672"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:06:55.247860 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:55.247831 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-credential-socket" pod "9d0d805e-ce8f-4cd0-91c1-7f00fa21e672" (UID: "9d0d805e-ce8f-4cd0-91c1-7f00fa21e672"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:06:55.247991 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:55.247968 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-istiod-ca-cert" pod "9d0d805e-ce8f-4cd0-91c1-7f00fa21e672" (UID: "9d0d805e-ce8f-4cd0-91c1-7f00fa21e672"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:06:55.248121 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:55.248108 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:55.248160 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:55.248125 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:55.248160 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:55.248134 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:55.248160 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:55.248143 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:55.248160 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:55.248151 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:55.249186 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:55.249162 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-istio-envoy" pod "9d0d805e-ce8f-4cd0-91c1-7f00fa21e672" (UID: "9d0d805e-ce8f-4cd0-91c1-7f00fa21e672"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:06:55.249827 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:55.249800 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-kube-api-access-f29vp" pod "9d0d805e-ce8f-4cd0-91c1-7f00fa21e672" (UID: "9d0d805e-ce8f-4cd0-91c1-7f00fa21e672"). InnerVolumeSpecName "kube-api-access-f29vp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:06:55.249931 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:55.249900 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-istio-podinfo" pod "9d0d805e-ce8f-4cd0-91c1-7f00fa21e672" (UID: "9d0d805e-ce8f-4cd0-91c1-7f00fa21e672"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:06:55.250285 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:55.250262 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-istio-token" pod "9d0d805e-ce8f-4cd0-91c1-7f00fa21e672" (UID: "9d0d805e-ce8f-4cd0-91c1-7f00fa21e672"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:06:55.349902 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:55.349795 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:55.349902 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:55.349835 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:55.349902 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:55.349852 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:55.349902 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:55.349867 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f29vp\" (UniqueName: \"kubernetes.io/projected/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672-kube-api-access-f29vp\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:55.765685 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:55.765620 5693 generic.go:356] "Generic (PLEG): container finished" podID="9d0d805e-ce8f-4cd0-91c1-7f00fa21e672" containerID="dc268220a5bc0dc0c97a46437a5734381bcd5cff5b7bb16da5a9a62bf3074416" exitCode=0 Oct 31 13:06:55.766159 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:55.765710 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" Oct 31 13:06:55.766159 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:55.765707 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" event={"ID":"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672","Type":"ContainerDied","Data":"dc268220a5bc0dc0c97a46437a5734381bcd5cff5b7bb16da5a9a62bf3074416"} Oct 31 13:06:55.766159 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:55.765781 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h" event={"ID":"9d0d805e-ce8f-4cd0-91c1-7f00fa21e672","Type":"ContainerDied","Data":"c6e18dd38f3c4b4905cf17d20c718b3d836c0fbc82566f98b13c1bd45a91d671"} Oct 31 13:06:55.766159 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:55.765814 5693 scope.go:122] "RemoveContainer" containerID="dc268220a5bc0dc0c97a46437a5734381bcd5cff5b7bb16da5a9a62bf3074416" Oct 31 13:06:55.784942 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:55.784915 5693 scope.go:122] "RemoveContainer" containerID="6544623a77af546e84f159923833b0a89c99167c9683d4a2872354068bc62f83" Oct 31 13:06:55.793748 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:55.793719 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h"] Oct 31 13:06:55.800541 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:55.800519 5693 scope.go:122] "RemoveContainer" containerID="dc268220a5bc0dc0c97a46437a5734381bcd5cff5b7bb16da5a9a62bf3074416" Oct 31 13:06:55.800809 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:55.800791 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-3a3000-openshift-default-6d9c8fd4b5-ttt6h"] Oct 31 13:06:55.800920 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:06:55.800900 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc268220a5bc0dc0c97a46437a5734381bcd5cff5b7bb16da5a9a62bf3074416\": container with ID starting with dc268220a5bc0dc0c97a46437a5734381bcd5cff5b7bb16da5a9a62bf3074416 not found: ID does not exist" containerID="dc268220a5bc0dc0c97a46437a5734381bcd5cff5b7bb16da5a9a62bf3074416" Oct 31 13:06:55.800961 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:55.800930 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc268220a5bc0dc0c97a46437a5734381bcd5cff5b7bb16da5a9a62bf3074416"} err="failed to get container status \"dc268220a5bc0dc0c97a46437a5734381bcd5cff5b7bb16da5a9a62bf3074416\": rpc error: code = NotFound desc = could not find container \"dc268220a5bc0dc0c97a46437a5734381bcd5cff5b7bb16da5a9a62bf3074416\": container with ID starting with dc268220a5bc0dc0c97a46437a5734381bcd5cff5b7bb16da5a9a62bf3074416 not found: ID does not exist" Oct 31 13:06:55.800961 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:55.800950 5693 scope.go:122] "RemoveContainer" containerID="6544623a77af546e84f159923833b0a89c99167c9683d4a2872354068bc62f83" Oct 31 13:06:55.801212 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:06:55.801194 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6544623a77af546e84f159923833b0a89c99167c9683d4a2872354068bc62f83\": container with ID starting with 6544623a77af546e84f159923833b0a89c99167c9683d4a2872354068bc62f83 not found: ID does not exist" containerID="6544623a77af546e84f159923833b0a89c99167c9683d4a2872354068bc62f83" Oct 31 13:06:55.801253 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:55.801220 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6544623a77af546e84f159923833b0a89c99167c9683d4a2872354068bc62f83"} err="failed to get container status \"6544623a77af546e84f159923833b0a89c99167c9683d4a2872354068bc62f83\": rpc error: code = NotFound desc = could not find container \"6544623a77af546e84f159923833b0a89c99167c9683d4a2872354068bc62f83\": container with ID starting with 6544623a77af546e84f159923833b0a89c99167c9683d4a2872354068bc62f83 not found: ID does not exist" Oct 31 13:06:57.408510 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:57.408479 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d0d805e-ce8f-4cd0-91c1-7f00fa21e672" path="/var/lib/kubelet/pods/9d0d805e-ce8f-4cd0-91c1-7f00fa21e672/volumes" Oct 31 13:06:59.435239 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:59.435204 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-758f48-6456c959dc-665vf" podUID="106bba13-d6d7-4c17-802e-bcf194297900" containerName="payload-processing" containerID="cri-o://57fc39968384db080e7fa6aa3c1aaa2b285b85e955881527a412b143fa5dbb29" gracePeriod=25 Oct 31 13:06:59.455723 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:06:59.455685 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-758f48-6785db86d7-j4phq" podUID="5022d397-de96-44f9-a83b-f28678b75b84" containerName="payload-pre-processing" containerID="cri-o://f0669d25033fd1e26c6788894fc59cba73c04b2b1d71b921b2277cd33dd404ad" gracePeriod=25 Oct 31 13:07:02.655129 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:02.655093 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-758f48-6456c959dc-665vf" podUID="106bba13-d6d7-4c17-802e-bcf194297900" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:07:02.655129 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:02.655119 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-758f48-6785db86d7-j4phq" podUID="5022d397-de96-44f9-a83b-f28678b75b84" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:07:04.018324 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:04.018270 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4"] Oct 31 13:07:04.020975 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:04.020917 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" podUID="024bc3d1-116f-4d0d-b8e6-d37839429e0e" containerName="istio-proxy" containerID="cri-o://bfecb7008e5791b0b5c9d75229ff53f49bd5e673582da6fd1630bb7a7bb4ad58" gracePeriod=30 Oct 31 13:07:07.947388 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:07.947349 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-758f48-c94487788-czlnw"] Oct 31 13:07:07.948709 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:07.947590 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-758f48-c94487788-czlnw" podUID="964ebb98-f4fe-43c0-acda-95c3cc1631c1" containerName="maas-api" containerID="cri-o://15bdd43cdbe827b6d8f976d63163f5d026da4c0147a599e0392c26dcd0e2acae" gracePeriod=30 Oct 31 13:07:08.777754 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:08.777718 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-758f48-c94487788-czlnw" podUID="964ebb98-f4fe-43c0-acda-95c3cc1631c1" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.121:8443/health\": dial tcp 10.133.0.121:8443: connect: connection refused" Oct 31 13:07:08.919433 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:08.919395 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-758f48-c94487788-czlnw" Oct 31 13:07:08.980260 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:08.980220 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/964ebb98-f4fe-43c0-acda-95c3cc1631c1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/964ebb98-f4fe-43c0-acda-95c3cc1631c1-openshift-service-ca\") pod \"964ebb98-f4fe-43c0-acda-95c3cc1631c1\" (UID: \"964ebb98-f4fe-43c0-acda-95c3cc1631c1\") " Oct 31 13:07:08.980793 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:08.980361 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/964ebb98-f4fe-43c0-acda-95c3cc1631c1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/964ebb98-f4fe-43c0-acda-95c3cc1631c1-maas-api-tls\") pod \"964ebb98-f4fe-43c0-acda-95c3cc1631c1\" (UID: \"964ebb98-f4fe-43c0-acda-95c3cc1631c1\") " Oct 31 13:07:08.980793 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:08.980417 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/964ebb98-f4fe-43c0-acda-95c3cc1631c1-kube-api-access-cq4m2\" (UniqueName: \"kubernetes.io/projected/964ebb98-f4fe-43c0-acda-95c3cc1631c1-kube-api-access-cq4m2\") pod \"964ebb98-f4fe-43c0-acda-95c3cc1631c1\" (UID: \"964ebb98-f4fe-43c0-acda-95c3cc1631c1\") " Oct 31 13:07:08.980793 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:08.980515 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/964ebb98-f4fe-43c0-acda-95c3cc1631c1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/964ebb98-f4fe-43c0-acda-95c3cc1631c1-metrics-certs\") pod \"964ebb98-f4fe-43c0-acda-95c3cc1631c1\" (UID: \"964ebb98-f4fe-43c0-acda-95c3cc1631c1\") " Oct 31 13:07:08.980793 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:08.980692 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/964ebb98-f4fe-43c0-acda-95c3cc1631c1-openshift-service-ca" pod "964ebb98-f4fe-43c0-acda-95c3cc1631c1" (UID: "964ebb98-f4fe-43c0-acda-95c3cc1631c1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:07:08.981176 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:08.981151 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/964ebb98-f4fe-43c0-acda-95c3cc1631c1-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:08.983889 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:08.983797 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/964ebb98-f4fe-43c0-acda-95c3cc1631c1-kube-api-access-cq4m2" pod "964ebb98-f4fe-43c0-acda-95c3cc1631c1" (UID: "964ebb98-f4fe-43c0-acda-95c3cc1631c1"). InnerVolumeSpecName "kube-api-access-cq4m2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:07:08.984085 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:08.984052 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/964ebb98-f4fe-43c0-acda-95c3cc1631c1-metrics-certs" pod "964ebb98-f4fe-43c0-acda-95c3cc1631c1" (UID: "964ebb98-f4fe-43c0-acda-95c3cc1631c1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:07:08.984489 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:08.984458 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/964ebb98-f4fe-43c0-acda-95c3cc1631c1-maas-api-tls" pod "964ebb98-f4fe-43c0-acda-95c3cc1631c1" (UID: "964ebb98-f4fe-43c0-acda-95c3cc1631c1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:07:09.082449 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.082379 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/964ebb98-f4fe-43c0-acda-95c3cc1631c1-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:09.082449 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.082411 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq4m2\" (UniqueName: \"kubernetes.io/projected/964ebb98-f4fe-43c0-acda-95c3cc1631c1-kube-api-access-cq4m2\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:09.082449 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.082427 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/964ebb98-f4fe-43c0-acda-95c3cc1631c1-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:09.381010 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.380751 5693 generic.go:356] "Generic (PLEG): container finished" podID="024bc3d1-116f-4d0d-b8e6-d37839429e0e" containerID="bfecb7008e5791b0b5c9d75229ff53f49bd5e673582da6fd1630bb7a7bb4ad58" exitCode=0 Oct 31 13:07:09.381010 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.380910 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" event={"ID":"024bc3d1-116f-4d0d-b8e6-d37839429e0e","Type":"ContainerDied","Data":"bfecb7008e5791b0b5c9d75229ff53f49bd5e673582da6fd1630bb7a7bb4ad58"} Oct 31 13:07:09.383424 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.383397 5693 generic.go:356] "Generic (PLEG): container finished" podID="964ebb98-f4fe-43c0-acda-95c3cc1631c1" containerID="15bdd43cdbe827b6d8f976d63163f5d026da4c0147a599e0392c26dcd0e2acae" exitCode=0 Oct 31 13:07:09.383572 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.383443 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-758f48-c94487788-czlnw" event={"ID":"964ebb98-f4fe-43c0-acda-95c3cc1631c1","Type":"ContainerDied","Data":"15bdd43cdbe827b6d8f976d63163f5d026da4c0147a599e0392c26dcd0e2acae"} Oct 31 13:07:09.383572 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.383486 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-758f48-c94487788-czlnw" event={"ID":"964ebb98-f4fe-43c0-acda-95c3cc1631c1","Type":"ContainerDied","Data":"b9f3a8bd867d9fdc7abc03abc26f9fb4f028fdcdcfa86379d61f07e802b4c8b5"} Oct 31 13:07:09.383572 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.383514 5693 scope.go:122] "RemoveContainer" containerID="15bdd43cdbe827b6d8f976d63163f5d026da4c0147a599e0392c26dcd0e2acae" Oct 31 13:07:09.383806 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.383715 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-758f48-c94487788-czlnw" Oct 31 13:07:09.400526 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.400409 5693 scope.go:122] "RemoveContainer" containerID="15bdd43cdbe827b6d8f976d63163f5d026da4c0147a599e0392c26dcd0e2acae" Oct 31 13:07:09.402045 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:07:09.401626 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15bdd43cdbe827b6d8f976d63163f5d026da4c0147a599e0392c26dcd0e2acae\": container with ID starting with 15bdd43cdbe827b6d8f976d63163f5d026da4c0147a599e0392c26dcd0e2acae not found: ID does not exist" containerID="15bdd43cdbe827b6d8f976d63163f5d026da4c0147a599e0392c26dcd0e2acae" Oct 31 13:07:09.402141 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.402062 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15bdd43cdbe827b6d8f976d63163f5d026da4c0147a599e0392c26dcd0e2acae"} err="failed to get container status \"15bdd43cdbe827b6d8f976d63163f5d026da4c0147a599e0392c26dcd0e2acae\": rpc error: code = NotFound desc = could not find container \"15bdd43cdbe827b6d8f976d63163f5d026da4c0147a599e0392c26dcd0e2acae\": container with ID starting with 15bdd43cdbe827b6d8f976d63163f5d026da4c0147a599e0392c26dcd0e2acae not found: ID does not exist" Oct 31 13:07:09.425526 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.425097 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-758f48-c94487788-czlnw"] Oct 31 13:07:09.431925 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.430885 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-758f48-c94487788-czlnw"] Oct 31 13:07:09.470841 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.470812 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" Oct 31 13:07:09.494448 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.494415 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/024bc3d1-116f-4d0d-b8e6-d37839429e0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/024bc3d1-116f-4d0d-b8e6-d37839429e0e-istio-envoy\") pod \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\" (UID: \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\") " Oct 31 13:07:09.494588 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.494507 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/024bc3d1-116f-4d0d-b8e6-d37839429e0e-istio-token\" (UniqueName: \"kubernetes.io/projected/024bc3d1-116f-4d0d-b8e6-d37839429e0e-istio-token\") pod \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\" (UID: \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\") " Oct 31 13:07:09.494588 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.494537 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/024bc3d1-116f-4d0d-b8e6-d37839429e0e-kube-api-access-562z8\" (UniqueName: \"kubernetes.io/projected/024bc3d1-116f-4d0d-b8e6-d37839429e0e-kube-api-access-562z8\") pod \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\" (UID: \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\") " Oct 31 13:07:09.494721 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.494690 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/024bc3d1-116f-4d0d-b8e6-d37839429e0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/024bc3d1-116f-4d0d-b8e6-d37839429e0e-istio-podinfo\") pod \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\" (UID: \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\") " Oct 31 13:07:09.494851 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.494753 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/024bc3d1-116f-4d0d-b8e6-d37839429e0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/024bc3d1-116f-4d0d-b8e6-d37839429e0e-workload-certs\") pod \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\" (UID: \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\") " Oct 31 13:07:09.495073 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.495040 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/024bc3d1-116f-4d0d-b8e6-d37839429e0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/024bc3d1-116f-4d0d-b8e6-d37839429e0e-workload-socket\") pod \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\" (UID: \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\") " Oct 31 13:07:09.495192 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.495131 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/024bc3d1-116f-4d0d-b8e6-d37839429e0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/024bc3d1-116f-4d0d-b8e6-d37839429e0e-istio-data\") pod \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\" (UID: \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\") " Oct 31 13:07:09.495192 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.495045 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024bc3d1-116f-4d0d-b8e6-d37839429e0e-workload-certs" pod "024bc3d1-116f-4d0d-b8e6-d37839429e0e" (UID: "024bc3d1-116f-4d0d-b8e6-d37839429e0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:07:09.495316 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.495219 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024bc3d1-116f-4d0d-b8e6-d37839429e0e-workload-socket" pod "024bc3d1-116f-4d0d-b8e6-d37839429e0e" (UID: "024bc3d1-116f-4d0d-b8e6-d37839429e0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:07:09.495316 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.495282 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/024bc3d1-116f-4d0d-b8e6-d37839429e0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/024bc3d1-116f-4d0d-b8e6-d37839429e0e-istiod-ca-cert\") pod \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\" (UID: \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\") " Oct 31 13:07:09.495437 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.495343 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/024bc3d1-116f-4d0d-b8e6-d37839429e0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/024bc3d1-116f-4d0d-b8e6-d37839429e0e-credential-socket\") pod \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\" (UID: \"024bc3d1-116f-4d0d-b8e6-d37839429e0e\") " Oct 31 13:07:09.495553 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.495531 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024bc3d1-116f-4d0d-b8e6-d37839429e0e-istio-data" pod "024bc3d1-116f-4d0d-b8e6-d37839429e0e" (UID: "024bc3d1-116f-4d0d-b8e6-d37839429e0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:07:09.496059 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.496038 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/024bc3d1-116f-4d0d-b8e6-d37839429e0e-istiod-ca-cert" pod "024bc3d1-116f-4d0d-b8e6-d37839429e0e" (UID: "024bc3d1-116f-4d0d-b8e6-d37839429e0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:07:09.496185 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.496153 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/024bc3d1-116f-4d0d-b8e6-d37839429e0e-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:09.496185 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.496173 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/024bc3d1-116f-4d0d-b8e6-d37839429e0e-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:09.496317 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.496187 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/024bc3d1-116f-4d0d-b8e6-d37839429e0e-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:09.496317 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.496201 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/024bc3d1-116f-4d0d-b8e6-d37839429e0e-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:09.496317 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.496251 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024bc3d1-116f-4d0d-b8e6-d37839429e0e-credential-socket" pod "024bc3d1-116f-4d0d-b8e6-d37839429e0e" (UID: "024bc3d1-116f-4d0d-b8e6-d37839429e0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:07:09.497480 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.497456 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024bc3d1-116f-4d0d-b8e6-d37839429e0e-istio-envoy" pod "024bc3d1-116f-4d0d-b8e6-d37839429e0e" (UID: "024bc3d1-116f-4d0d-b8e6-d37839429e0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:07:09.497918 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.497893 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/024bc3d1-116f-4d0d-b8e6-d37839429e0e-kube-api-access-562z8" pod "024bc3d1-116f-4d0d-b8e6-d37839429e0e" (UID: "024bc3d1-116f-4d0d-b8e6-d37839429e0e"). InnerVolumeSpecName "kube-api-access-562z8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:07:09.498606 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.498585 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/024bc3d1-116f-4d0d-b8e6-d37839429e0e-istio-token" pod "024bc3d1-116f-4d0d-b8e6-d37839429e0e" (UID: "024bc3d1-116f-4d0d-b8e6-d37839429e0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:07:09.498932 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.498914 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/024bc3d1-116f-4d0d-b8e6-d37839429e0e-istio-podinfo" pod "024bc3d1-116f-4d0d-b8e6-d37839429e0e" (UID: "024bc3d1-116f-4d0d-b8e6-d37839429e0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:07:09.597550 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.597514 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/024bc3d1-116f-4d0d-b8e6-d37839429e0e-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:09.597550 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.597547 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/024bc3d1-116f-4d0d-b8e6-d37839429e0e-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:09.597791 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.597570 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/024bc3d1-116f-4d0d-b8e6-d37839429e0e-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:09.597791 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.597585 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-562z8\" (UniqueName: \"kubernetes.io/projected/024bc3d1-116f-4d0d-b8e6-d37839429e0e-kube-api-access-562z8\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:09.597791 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:09.597598 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/024bc3d1-116f-4d0d-b8e6-d37839429e0e-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:10.024295 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:10.024255 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6"] Oct 31 13:07:10.024812 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:10.024551 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" podUID="2c28ec7e-5395-4112-af19-7ea652fa1dae" containerName="istio-proxy" containerID="cri-o://df438f11c0baebb51552ad4df5e9b40e940937ce9422161ebd4a683a1d9b6741" gracePeriod=30 Oct 31 13:07:10.392172 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:10.392087 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" event={"ID":"024bc3d1-116f-4d0d-b8e6-d37839429e0e","Type":"ContainerDied","Data":"32577a1f15e3d3e6e89f20df5f79820405e6ff95da957c150d0ffd1470d65899"} Oct 31 13:07:10.392172 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:10.392112 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4" Oct 31 13:07:10.392172 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:10.392156 5693 scope.go:122] "RemoveContainer" containerID="bfecb7008e5791b0b5c9d75229ff53f49bd5e673582da6fd1630bb7a7bb4ad58" Oct 31 13:07:10.420257 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:10.420227 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4"] Oct 31 13:07:10.426288 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:10.426263 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-pvgg4"] Oct 31 13:07:11.408603 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:11.408570 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="024bc3d1-116f-4d0d-b8e6-d37839429e0e" path="/var/lib/kubelet/pods/024bc3d1-116f-4d0d-b8e6-d37839429e0e/volumes" Oct 31 13:07:11.409059 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:11.409043 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="964ebb98-f4fe-43c0-acda-95c3cc1631c1" path="/var/lib/kubelet/pods/964ebb98-f4fe-43c0-acda-95c3cc1631c1/volumes" Oct 31 13:07:12.655547 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:12.655508 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-758f48-6456c959dc-665vf" podUID="106bba13-d6d7-4c17-802e-bcf194297900" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:07:12.655982 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:12.655569 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-758f48-6785db86d7-j4phq" podUID="5022d397-de96-44f9-a83b-f28678b75b84" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:07:15.312175 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:15.312142 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" Oct 31 13:07:15.425433 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:15.425393 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c28ec7e-5395-4112-af19-7ea652fa1dae-kube-api-access-xgrmn\" (UniqueName: \"kubernetes.io/projected/2c28ec7e-5395-4112-af19-7ea652fa1dae-kube-api-access-xgrmn\") pod \"2c28ec7e-5395-4112-af19-7ea652fa1dae\" (UID: \"2c28ec7e-5395-4112-af19-7ea652fa1dae\") " Oct 31 13:07:15.425596 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:15.425465 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c28ec7e-5395-4112-af19-7ea652fa1dae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c28ec7e-5395-4112-af19-7ea652fa1dae-credential-socket\") pod \"2c28ec7e-5395-4112-af19-7ea652fa1dae\" (UID: \"2c28ec7e-5395-4112-af19-7ea652fa1dae\") " Oct 31 13:07:15.425596 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:15.425538 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c28ec7e-5395-4112-af19-7ea652fa1dae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c28ec7e-5395-4112-af19-7ea652fa1dae-istio-data\") pod \"2c28ec7e-5395-4112-af19-7ea652fa1dae\" (UID: \"2c28ec7e-5395-4112-af19-7ea652fa1dae\") " Oct 31 13:07:15.425596 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:15.425586 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c28ec7e-5395-4112-af19-7ea652fa1dae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c28ec7e-5395-4112-af19-7ea652fa1dae-workload-socket\") pod \"2c28ec7e-5395-4112-af19-7ea652fa1dae\" (UID: \"2c28ec7e-5395-4112-af19-7ea652fa1dae\") " Oct 31 13:07:15.425785 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:15.425614 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c28ec7e-5395-4112-af19-7ea652fa1dae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c28ec7e-5395-4112-af19-7ea652fa1dae-workload-certs\") pod \"2c28ec7e-5395-4112-af19-7ea652fa1dae\" (UID: \"2c28ec7e-5395-4112-af19-7ea652fa1dae\") " Oct 31 13:07:15.425785 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:15.425676 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c28ec7e-5395-4112-af19-7ea652fa1dae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c28ec7e-5395-4112-af19-7ea652fa1dae-istio-podinfo\") pod \"2c28ec7e-5395-4112-af19-7ea652fa1dae\" (UID: \"2c28ec7e-5395-4112-af19-7ea652fa1dae\") " Oct 31 13:07:15.425785 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:15.425723 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c28ec7e-5395-4112-af19-7ea652fa1dae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c28ec7e-5395-4112-af19-7ea652fa1dae-istiod-ca-cert\") pod \"2c28ec7e-5395-4112-af19-7ea652fa1dae\" (UID: \"2c28ec7e-5395-4112-af19-7ea652fa1dae\") " Oct 31 13:07:15.425943 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:15.425786 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c28ec7e-5395-4112-af19-7ea652fa1dae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c28ec7e-5395-4112-af19-7ea652fa1dae-istio-envoy\") pod \"2c28ec7e-5395-4112-af19-7ea652fa1dae\" (UID: \"2c28ec7e-5395-4112-af19-7ea652fa1dae\") " Oct 31 13:07:15.425943 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:15.425849 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c28ec7e-5395-4112-af19-7ea652fa1dae-credential-socket" pod "2c28ec7e-5395-4112-af19-7ea652fa1dae" (UID: "2c28ec7e-5395-4112-af19-7ea652fa1dae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:07:15.425943 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:15.425938 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c28ec7e-5395-4112-af19-7ea652fa1dae-istio-token\" (UniqueName: \"kubernetes.io/projected/2c28ec7e-5395-4112-af19-7ea652fa1dae-istio-token\") pod \"2c28ec7e-5395-4112-af19-7ea652fa1dae\" (UID: \"2c28ec7e-5395-4112-af19-7ea652fa1dae\") " Oct 31 13:07:15.426113 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:15.425942 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c28ec7e-5395-4112-af19-7ea652fa1dae-workload-socket" pod "2c28ec7e-5395-4112-af19-7ea652fa1dae" (UID: "2c28ec7e-5395-4112-af19-7ea652fa1dae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:07:15.426169 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:15.426148 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c28ec7e-5395-4112-af19-7ea652fa1dae-istio-data" pod "2c28ec7e-5395-4112-af19-7ea652fa1dae" (UID: "2c28ec7e-5395-4112-af19-7ea652fa1dae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:07:15.426391 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:15.426371 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c28ec7e-5395-4112-af19-7ea652fa1dae-istiod-ca-cert" pod "2c28ec7e-5395-4112-af19-7ea652fa1dae" (UID: "2c28ec7e-5395-4112-af19-7ea652fa1dae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:07:15.426625 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:15.426605 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c28ec7e-5395-4112-af19-7ea652fa1dae-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:15.426744 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:15.426630 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c28ec7e-5395-4112-af19-7ea652fa1dae-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:15.426744 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:15.426682 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c28ec7e-5395-4112-af19-7ea652fa1dae-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:15.426744 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:15.426696 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c28ec7e-5395-4112-af19-7ea652fa1dae-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:15.427138 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:15.427079 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c28ec7e-5395-4112-af19-7ea652fa1dae-workload-certs" pod "2c28ec7e-5395-4112-af19-7ea652fa1dae" (UID: "2c28ec7e-5395-4112-af19-7ea652fa1dae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:07:15.428868 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:15.428844 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c28ec7e-5395-4112-af19-7ea652fa1dae-kube-api-access-xgrmn" pod "2c28ec7e-5395-4112-af19-7ea652fa1dae" (UID: "2c28ec7e-5395-4112-af19-7ea652fa1dae"). InnerVolumeSpecName "kube-api-access-xgrmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:07:15.436819 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:15.436777 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c28ec7e-5395-4112-af19-7ea652fa1dae-istio-podinfo" pod "2c28ec7e-5395-4112-af19-7ea652fa1dae" (UID: "2c28ec7e-5395-4112-af19-7ea652fa1dae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:07:15.438053 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:15.438031 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c28ec7e-5395-4112-af19-7ea652fa1dae-istio-envoy" pod "2c28ec7e-5395-4112-af19-7ea652fa1dae" (UID: "2c28ec7e-5395-4112-af19-7ea652fa1dae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:07:15.438190 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:15.438174 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c28ec7e-5395-4112-af19-7ea652fa1dae-istio-token" pod "2c28ec7e-5395-4112-af19-7ea652fa1dae" (UID: "2c28ec7e-5395-4112-af19-7ea652fa1dae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:07:15.441071 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:15.441044 5693 generic.go:356] "Generic (PLEG): container finished" podID="2c28ec7e-5395-4112-af19-7ea652fa1dae" containerID="df438f11c0baebb51552ad4df5e9b40e940937ce9422161ebd4a683a1d9b6741" exitCode=0 Oct 31 13:07:15.441193 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:15.441091 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" Oct 31 13:07:15.441322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:15.441285 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" event={"ID":"2c28ec7e-5395-4112-af19-7ea652fa1dae","Type":"ContainerDied","Data":"df438f11c0baebb51552ad4df5e9b40e940937ce9422161ebd4a683a1d9b6741"} Oct 31 13:07:15.441431 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:15.441357 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6" event={"ID":"2c28ec7e-5395-4112-af19-7ea652fa1dae","Type":"ContainerDied","Data":"38f723732f4012be7f98a6cbdc0a0f41196219e44d323b3a4984cb2de0ccf1dd"} Oct 31 13:07:15.441431 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:15.441392 5693 scope.go:122] "RemoveContainer" containerID="df438f11c0baebb51552ad4df5e9b40e940937ce9422161ebd4a683a1d9b6741" Oct 31 13:07:15.461483 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:15.461447 5693 scope.go:122] "RemoveContainer" containerID="df438f11c0baebb51552ad4df5e9b40e940937ce9422161ebd4a683a1d9b6741" Oct 31 13:07:15.461871 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:07:15.461846 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df438f11c0baebb51552ad4df5e9b40e940937ce9422161ebd4a683a1d9b6741\": container with ID starting with df438f11c0baebb51552ad4df5e9b40e940937ce9422161ebd4a683a1d9b6741 not found: ID does not exist" containerID="df438f11c0baebb51552ad4df5e9b40e940937ce9422161ebd4a683a1d9b6741" Oct 31 13:07:15.461956 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:15.461883 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df438f11c0baebb51552ad4df5e9b40e940937ce9422161ebd4a683a1d9b6741"} err="failed to get container status \"df438f11c0baebb51552ad4df5e9b40e940937ce9422161ebd4a683a1d9b6741\": rpc error: code = NotFound desc = could not find container \"df438f11c0baebb51552ad4df5e9b40e940937ce9422161ebd4a683a1d9b6741\": container with ID starting with df438f11c0baebb51552ad4df5e9b40e940937ce9422161ebd4a683a1d9b6741 not found: ID does not exist" Oct 31 13:07:15.485241 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:15.485219 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6"] Oct 31 13:07:15.491070 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:15.491048 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-758f48-openshift-default-685758985c-sz5j6"] Oct 31 13:07:15.528595 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:15.528574 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgrmn\" (UniqueName: \"kubernetes.io/projected/2c28ec7e-5395-4112-af19-7ea652fa1dae-kube-api-access-xgrmn\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:15.528595 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:15.528595 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c28ec7e-5395-4112-af19-7ea652fa1dae-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:15.528760 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:15.528606 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c28ec7e-5395-4112-af19-7ea652fa1dae-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:15.528760 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:15.528620 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c28ec7e-5395-4112-af19-7ea652fa1dae-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:15.528760 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:15.528630 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c28ec7e-5395-4112-af19-7ea652fa1dae-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:17.410253 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:17.410215 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c28ec7e-5395-4112-af19-7ea652fa1dae" path="/var/lib/kubelet/pods/2c28ec7e-5395-4112-af19-7ea652fa1dae/volumes" Oct 31 13:07:17.900011 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:17.899977 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e4420f-667564665f-vjmfn"] Oct 31 13:07:18.687310 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:07:18.687277 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:07:19.765608 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:19.765580 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-758f48-6456c959dc-665vf" Oct 31 13:07:19.800309 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:19.800281 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-758f48-6785db86d7-j4phq" Oct 31 13:07:19.893397 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:19.893365 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/106bba13-d6d7-4c17-802e-bcf194297900-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/106bba13-d6d7-4c17-802e-bcf194297900-plugins-config-volume\") pod \"106bba13-d6d7-4c17-802e-bcf194297900\" (UID: \"106bba13-d6d7-4c17-802e-bcf194297900\") " Oct 31 13:07:19.893679 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:19.893627 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/106bba13-d6d7-4c17-802e-bcf194297900-kube-api-access-9krvq\" (UniqueName: \"kubernetes.io/projected/106bba13-d6d7-4c17-802e-bcf194297900-kube-api-access-9krvq\") pod \"106bba13-d6d7-4c17-802e-bcf194297900\" (UID: \"106bba13-d6d7-4c17-802e-bcf194297900\") " Oct 31 13:07:19.893778 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:19.893755 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/106bba13-d6d7-4c17-802e-bcf194297900-plugins-config-volume" pod "106bba13-d6d7-4c17-802e-bcf194297900" (UID: "106bba13-d6d7-4c17-802e-bcf194297900"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:07:19.894175 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:19.894150 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/106bba13-d6d7-4c17-802e-bcf194297900-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:19.895907 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:19.895887 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/106bba13-d6d7-4c17-802e-bcf194297900-kube-api-access-9krvq" pod "106bba13-d6d7-4c17-802e-bcf194297900" (UID: "106bba13-d6d7-4c17-802e-bcf194297900"). InnerVolumeSpecName "kube-api-access-9krvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:07:19.995390 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:19.995304 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5022d397-de96-44f9-a83b-f28678b75b84-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5022d397-de96-44f9-a83b-f28678b75b84-plugins-config-volume\") pod \"5022d397-de96-44f9-a83b-f28678b75b84\" (UID: \"5022d397-de96-44f9-a83b-f28678b75b84\") " Oct 31 13:07:19.995529 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:19.995502 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5022d397-de96-44f9-a83b-f28678b75b84-kube-api-access-5fs2g\" (UniqueName: \"kubernetes.io/projected/5022d397-de96-44f9-a83b-f28678b75b84-kube-api-access-5fs2g\") pod \"5022d397-de96-44f9-a83b-f28678b75b84\" (UID: \"5022d397-de96-44f9-a83b-f28678b75b84\") " Oct 31 13:07:19.995771 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:19.995748 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5022d397-de96-44f9-a83b-f28678b75b84-plugins-config-volume" pod "5022d397-de96-44f9-a83b-f28678b75b84" (UID: "5022d397-de96-44f9-a83b-f28678b75b84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:07:19.996085 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:19.996069 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5022d397-de96-44f9-a83b-f28678b75b84-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:19.996128 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:19.996088 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9krvq\" (UniqueName: \"kubernetes.io/projected/106bba13-d6d7-4c17-802e-bcf194297900-kube-api-access-9krvq\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:19.997700 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:19.997672 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5022d397-de96-44f9-a83b-f28678b75b84-kube-api-access-5fs2g" pod "5022d397-de96-44f9-a83b-f28678b75b84" (UID: "5022d397-de96-44f9-a83b-f28678b75b84"). InnerVolumeSpecName "kube-api-access-5fs2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:07:20.097862 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:20.097823 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fs2g\" (UniqueName: \"kubernetes.io/projected/5022d397-de96-44f9-a83b-f28678b75b84-kube-api-access-5fs2g\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:20.472714 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:20.472677 5693 generic.go:356] "Generic (PLEG): container finished" podID="5022d397-de96-44f9-a83b-f28678b75b84" containerID="f0669d25033fd1e26c6788894fc59cba73c04b2b1d71b921b2277cd33dd404ad" exitCode=0 Oct 31 13:07:20.472854 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:20.472725 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-758f48-6785db86d7-j4phq" Oct 31 13:07:20.472854 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:20.472764 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-758f48-6785db86d7-j4phq" event={"ID":"5022d397-de96-44f9-a83b-f28678b75b84","Type":"ContainerDied","Data":"f0669d25033fd1e26c6788894fc59cba73c04b2b1d71b921b2277cd33dd404ad"} Oct 31 13:07:20.472854 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:20.472814 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-758f48-6785db86d7-j4phq" event={"ID":"5022d397-de96-44f9-a83b-f28678b75b84","Type":"ContainerDied","Data":"032fe138747a98266e67143af695348f3551739d5d1c50b5873cf7cf56b32378"} Oct 31 13:07:20.472854 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:20.472834 5693 scope.go:122] "RemoveContainer" containerID="f0669d25033fd1e26c6788894fc59cba73c04b2b1d71b921b2277cd33dd404ad" Oct 31 13:07:20.474311 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:20.474290 5693 generic.go:356] "Generic (PLEG): container finished" podID="106bba13-d6d7-4c17-802e-bcf194297900" containerID="57fc39968384db080e7fa6aa3c1aaa2b285b85e955881527a412b143fa5dbb29" exitCode=0 Oct 31 13:07:20.474415 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:20.474343 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-758f48-6456c959dc-665vf" event={"ID":"106bba13-d6d7-4c17-802e-bcf194297900","Type":"ContainerDied","Data":"57fc39968384db080e7fa6aa3c1aaa2b285b85e955881527a412b143fa5dbb29"} Oct 31 13:07:20.474415 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:20.474376 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-758f48-6456c959dc-665vf" event={"ID":"106bba13-d6d7-4c17-802e-bcf194297900","Type":"ContainerDied","Data":"920341ef52d6744c3e170e6d9a458b41032bae30ff5750eec4377487baa71009"} Oct 31 13:07:20.474488 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:20.474441 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-758f48-6456c959dc-665vf" Oct 31 13:07:20.484230 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:20.484208 5693 scope.go:122] "RemoveContainer" containerID="f0669d25033fd1e26c6788894fc59cba73c04b2b1d71b921b2277cd33dd404ad" Oct 31 13:07:20.484561 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:07:20.484467 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0669d25033fd1e26c6788894fc59cba73c04b2b1d71b921b2277cd33dd404ad\": container with ID starting with f0669d25033fd1e26c6788894fc59cba73c04b2b1d71b921b2277cd33dd404ad not found: ID does not exist" containerID="f0669d25033fd1e26c6788894fc59cba73c04b2b1d71b921b2277cd33dd404ad" Oct 31 13:07:20.484561 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:20.484491 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0669d25033fd1e26c6788894fc59cba73c04b2b1d71b921b2277cd33dd404ad"} err="failed to get container status \"f0669d25033fd1e26c6788894fc59cba73c04b2b1d71b921b2277cd33dd404ad\": rpc error: code = NotFound desc = could not find container \"f0669d25033fd1e26c6788894fc59cba73c04b2b1d71b921b2277cd33dd404ad\": container with ID starting with f0669d25033fd1e26c6788894fc59cba73c04b2b1d71b921b2277cd33dd404ad not found: ID does not exist" Oct 31 13:07:20.484561 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:20.484506 5693 scope.go:122] "RemoveContainer" containerID="57fc39968384db080e7fa6aa3c1aaa2b285b85e955881527a412b143fa5dbb29" Oct 31 13:07:20.493705 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:20.493689 5693 scope.go:122] "RemoveContainer" containerID="57fc39968384db080e7fa6aa3c1aaa2b285b85e955881527a412b143fa5dbb29" Oct 31 13:07:20.493969 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:07:20.493949 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57fc39968384db080e7fa6aa3c1aaa2b285b85e955881527a412b143fa5dbb29\": container with ID starting with 57fc39968384db080e7fa6aa3c1aaa2b285b85e955881527a412b143fa5dbb29 not found: ID does not exist" containerID="57fc39968384db080e7fa6aa3c1aaa2b285b85e955881527a412b143fa5dbb29" Oct 31 13:07:20.494035 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:20.493975 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57fc39968384db080e7fa6aa3c1aaa2b285b85e955881527a412b143fa5dbb29"} err="failed to get container status \"57fc39968384db080e7fa6aa3c1aaa2b285b85e955881527a412b143fa5dbb29\": rpc error: code = NotFound desc = could not find container \"57fc39968384db080e7fa6aa3c1aaa2b285b85e955881527a412b143fa5dbb29\": container with ID starting with 57fc39968384db080e7fa6aa3c1aaa2b285b85e955881527a412b143fa5dbb29 not found: ID does not exist" Oct 31 13:07:20.497518 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:20.497498 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-758f48-6785db86d7-j4phq"] Oct 31 13:07:20.504247 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:20.504228 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-758f48-6785db86d7-j4phq"] Oct 31 13:07:20.515090 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:20.515071 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-758f48-6456c959dc-665vf"] Oct 31 13:07:20.520611 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:20.520591 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-758f48-6456c959dc-665vf"] Oct 31 13:07:21.408991 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:21.408955 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="106bba13-d6d7-4c17-802e-bcf194297900" path="/var/lib/kubelet/pods/106bba13-d6d7-4c17-802e-bcf194297900/volumes" Oct 31 13:07:21.409407 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:21.409385 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5022d397-de96-44f9-a83b-f28678b75b84" path="/var/lib/kubelet/pods/5022d397-de96-44f9-a83b-f28678b75b84/volumes" Oct 31 13:07:22.909809 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:22.909756 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e4420f-667564665f-vjmfn" podUID="d9389331-8d19-426f-ba40-edc6324d7463" containerName="payload-pre-processing" containerID="cri-o://283a9761a31b1abf2e295b9e7ee154c7a8d4c1b22ff83eade706a3deed3708e5" gracePeriod=25 Oct 31 13:07:23.457667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:23.457076 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e4420f-667564665f-vjmfn" Oct 31 13:07:23.547057 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:23.546463 5693 generic.go:356] "Generic (PLEG): container finished" podID="d9389331-8d19-426f-ba40-edc6324d7463" containerID="283a9761a31b1abf2e295b9e7ee154c7a8d4c1b22ff83eade706a3deed3708e5" exitCode=0 Oct 31 13:07:23.547057 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:23.546634 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e4420f-667564665f-vjmfn" event={"ID":"d9389331-8d19-426f-ba40-edc6324d7463","Type":"ContainerDied","Data":"283a9761a31b1abf2e295b9e7ee154c7a8d4c1b22ff83eade706a3deed3708e5"} Oct 31 13:07:23.547057 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:23.546747 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e4420f-667564665f-vjmfn" event={"ID":"d9389331-8d19-426f-ba40-edc6324d7463","Type":"ContainerDied","Data":"1a33324579c8da5393aad398bcbe62f3c8148e6598a36b9df20bd76eb8aa9eea"} Oct 31 13:07:23.547057 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:23.546785 5693 scope.go:122] "RemoveContainer" containerID="283a9761a31b1abf2e295b9e7ee154c7a8d4c1b22ff83eade706a3deed3708e5" Oct 31 13:07:23.547057 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:23.546928 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e4420f-667564665f-vjmfn" Oct 31 13:07:23.567472 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:23.567438 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9389331-8d19-426f-ba40-edc6324d7463-kube-api-access-gsg8d\" (UniqueName: \"kubernetes.io/projected/d9389331-8d19-426f-ba40-edc6324d7463-kube-api-access-gsg8d\") pod \"d9389331-8d19-426f-ba40-edc6324d7463\" (UID: \"d9389331-8d19-426f-ba40-edc6324d7463\") " Oct 31 13:07:23.568178 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:23.568155 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9389331-8d19-426f-ba40-edc6324d7463-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9389331-8d19-426f-ba40-edc6324d7463-plugins-config-volume\") pod \"d9389331-8d19-426f-ba40-edc6324d7463\" (UID: \"d9389331-8d19-426f-ba40-edc6324d7463\") " Oct 31 13:07:23.569981 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:23.568868 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9389331-8d19-426f-ba40-edc6324d7463-plugins-config-volume" pod "d9389331-8d19-426f-ba40-edc6324d7463" (UID: "d9389331-8d19-426f-ba40-edc6324d7463"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:07:23.611561 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:23.611222 5693 scope.go:122] "RemoveContainer" containerID="283a9761a31b1abf2e295b9e7ee154c7a8d4c1b22ff83eade706a3deed3708e5" Oct 31 13:07:23.611984 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:07:23.611873 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"283a9761a31b1abf2e295b9e7ee154c7a8d4c1b22ff83eade706a3deed3708e5\": container with ID starting with 283a9761a31b1abf2e295b9e7ee154c7a8d4c1b22ff83eade706a3deed3708e5 not found: ID does not exist" containerID="283a9761a31b1abf2e295b9e7ee154c7a8d4c1b22ff83eade706a3deed3708e5" Oct 31 13:07:23.611984 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:23.611913 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"283a9761a31b1abf2e295b9e7ee154c7a8d4c1b22ff83eade706a3deed3708e5"} err="failed to get container status \"283a9761a31b1abf2e295b9e7ee154c7a8d4c1b22ff83eade706a3deed3708e5\": rpc error: code = NotFound desc = could not find container \"283a9761a31b1abf2e295b9e7ee154c7a8d4c1b22ff83eade706a3deed3708e5\": container with ID starting with 283a9761a31b1abf2e295b9e7ee154c7a8d4c1b22ff83eade706a3deed3708e5 not found: ID does not exist" Oct 31 13:07:23.668666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:23.662138 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9389331-8d19-426f-ba40-edc6324d7463-kube-api-access-gsg8d" pod "d9389331-8d19-426f-ba40-edc6324d7463" (UID: "d9389331-8d19-426f-ba40-edc6324d7463"). InnerVolumeSpecName "kube-api-access-gsg8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:07:23.695589 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:23.675779 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9389331-8d19-426f-ba40-edc6324d7463-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:23.695589 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:23.675819 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsg8d\" (UniqueName: \"kubernetes.io/projected/d9389331-8d19-426f-ba40-edc6324d7463-kube-api-access-gsg8d\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:24.010667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:23.998716 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e4420f-667564665f-vjmfn"] Oct 31 13:07:24.167817 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:24.153607 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e4420f-667564665f-vjmfn"] Oct 31 13:07:25.092667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:25.091751 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" podUID="1a2abd3f-fb4f-4990-9a32-5f945a588068" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:07:25.351719 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:25.342359 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" podUID="f2b3766f-058c-40e7-9a2d-a7f5028e2b1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:07:25.588668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:25.580581 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" podUID="ff9a9909-6fc3-475b-a9fe-562ec150c76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 31 13:07:25.679664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:25.678975 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9389331-8d19-426f-ba40-edc6324d7463" path="/var/lib/kubelet/pods/d9389331-8d19-426f-ba40-edc6324d7463/volumes" Oct 31 13:07:30.174666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:30.173924 5693 scope.go:122] "RemoveContainer" containerID="f240f8d2ac8eb471a2064dc890a34460cecd5a15bda5f437e23b3fa139eba818" Oct 31 13:07:31.195763 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.195727 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-7cdec6-5969765647-58sz8"] Oct 31 13:07:31.196301 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.196285 5693 cpu_manager.go:414] "containerMap: removing container" podUID="818a6fca-5402-45b0-9851-f31157caddc7" containerName="payload-pre-processing" Oct 31 13:07:31.196301 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.196301 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="818a6fca-5402-45b0-9851-f31157caddc7" containerName="payload-pre-processing" Oct 31 13:07:31.196426 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.196315 5693 cpu_manager.go:414] "containerMap: removing container" podUID="964ebb98-f4fe-43c0-acda-95c3cc1631c1" containerName="maas-api" Oct 31 13:07:31.196426 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.196320 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="964ebb98-f4fe-43c0-acda-95c3cc1631c1" containerName="maas-api" Oct 31 13:07:31.196426 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.196331 5693 cpu_manager.go:414] "containerMap: removing container" podUID="2bf930ee-79a6-4b2e-a18e-77b4948aba78" containerName="payload-processing" Oct 31 13:07:31.196426 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.196336 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bf930ee-79a6-4b2e-a18e-77b4948aba78" containerName="payload-processing" Oct 31 13:07:31.196426 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.196343 5693 cpu_manager.go:414] "containerMap: removing container" podUID="9d0d805e-ce8f-4cd0-91c1-7f00fa21e672" containerName="istio-proxy" Oct 31 13:07:31.196426 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.196347 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d0d805e-ce8f-4cd0-91c1-7f00fa21e672" containerName="istio-proxy" Oct 31 13:07:31.196426 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.196360 5693 cpu_manager.go:414] "containerMap: removing container" podUID="df10a6fe-ce6b-4515-9264-79d420a3389a" containerName="payload-processing" Oct 31 13:07:31.196426 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.196364 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df10a6fe-ce6b-4515-9264-79d420a3389a" containerName="payload-processing" Oct 31 13:07:31.196426 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.196386 5693 cpu_manager.go:414] "containerMap: removing container" podUID="c1b76053-3fdc-49c0-97e1-cc142093b6b7" containerName="payload-pre-processing" Oct 31 13:07:31.196426 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.196391 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1b76053-3fdc-49c0-97e1-cc142093b6b7" containerName="payload-pre-processing" Oct 31 13:07:31.196426 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.196399 5693 cpu_manager.go:414] "containerMap: removing container" podUID="106bba13-d6d7-4c17-802e-bcf194297900" containerName="payload-processing" Oct 31 13:07:31.196426 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.196403 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="106bba13-d6d7-4c17-802e-bcf194297900" containerName="payload-processing" Oct 31 13:07:31.196426 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.196410 5693 cpu_manager.go:414] "containerMap: removing container" podUID="5022d397-de96-44f9-a83b-f28678b75b84" containerName="payload-pre-processing" Oct 31 13:07:31.196426 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.196414 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5022d397-de96-44f9-a83b-f28678b75b84" containerName="payload-pre-processing" Oct 31 13:07:31.196426 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.196423 5693 cpu_manager.go:414] "containerMap: removing container" podUID="024bc3d1-116f-4d0d-b8e6-d37839429e0e" containerName="istio-proxy" Oct 31 13:07:31.196426 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.196428 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="024bc3d1-116f-4d0d-b8e6-d37839429e0e" containerName="istio-proxy" Oct 31 13:07:31.196426 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.196435 5693 cpu_manager.go:414] "containerMap: removing container" podUID="2c28ec7e-5395-4112-af19-7ea652fa1dae" containerName="istio-proxy" Oct 31 13:07:31.197403 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.196441 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c28ec7e-5395-4112-af19-7ea652fa1dae" containerName="istio-proxy" Oct 31 13:07:31.197403 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.196448 5693 cpu_manager.go:414] "containerMap: removing container" podUID="416c552c-b82d-472a-8f81-ac404b2c8a31" containerName="maas-api" Oct 31 13:07:31.197403 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.196453 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="416c552c-b82d-472a-8f81-ac404b2c8a31" containerName="maas-api" Oct 31 13:07:31.197403 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.196463 5693 cpu_manager.go:414] "containerMap: removing container" podUID="d9389331-8d19-426f-ba40-edc6324d7463" containerName="payload-pre-processing" Oct 31 13:07:31.197403 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.196467 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9389331-8d19-426f-ba40-edc6324d7463" containerName="payload-pre-processing" Oct 31 13:07:31.197403 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.196546 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1b76053-3fdc-49c0-97e1-cc142093b6b7" containerName="payload-pre-processing" Oct 31 13:07:31.197403 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.196553 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="106bba13-d6d7-4c17-802e-bcf194297900" containerName="payload-processing" Oct 31 13:07:31.197403 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.196561 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="5022d397-de96-44f9-a83b-f28678b75b84" containerName="payload-pre-processing" Oct 31 13:07:31.197403 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.196567 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="024bc3d1-116f-4d0d-b8e6-d37839429e0e" containerName="istio-proxy" Oct 31 13:07:31.197403 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.196572 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d0d805e-ce8f-4cd0-91c1-7f00fa21e672" containerName="istio-proxy" Oct 31 13:07:31.197403 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.196577 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c28ec7e-5395-4112-af19-7ea652fa1dae" containerName="istio-proxy" Oct 31 13:07:31.197403 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.196584 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="416c552c-b82d-472a-8f81-ac404b2c8a31" containerName="maas-api" Oct 31 13:07:31.197403 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.196590 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9389331-8d19-426f-ba40-edc6324d7463" containerName="payload-pre-processing" Oct 31 13:07:31.197403 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.196597 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="818a6fca-5402-45b0-9851-f31157caddc7" containerName="payload-pre-processing" Oct 31 13:07:31.197403 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.196604 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="964ebb98-f4fe-43c0-acda-95c3cc1631c1" containerName="maas-api" Oct 31 13:07:31.197403 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.196610 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bf930ee-79a6-4b2e-a18e-77b4948aba78" containerName="payload-processing" Oct 31 13:07:31.197403 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.196616 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d0d805e-ce8f-4cd0-91c1-7f00fa21e672" containerName="istio-proxy" Oct 31 13:07:31.197403 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.196621 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="df10a6fe-ce6b-4515-9264-79d420a3389a" containerName="payload-processing" Oct 31 13:07:31.202009 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.201987 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-7cdec6-5969765647-58sz8" Oct 31 13:07:31.270037 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.270002 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-7cdec6\"" Oct 31 13:07:31.270423 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.270399 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-7cdec6-dockercfg-cvxp6\"" Oct 31 13:07:31.271714 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.271683 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq2f5\" (UniqueName: \"kubernetes.io/projected/4de1aea2-a2ee-4257-8255-e63c353af739-kube-api-access-rq2f5\") pod \"payload-pre-processing-e2e-shared-a-w0-7cdec6-5969765647-58sz8\" (UID: \"4de1aea2-a2ee-4257-8255-e63c353af739\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-7cdec6-5969765647-58sz8" Oct 31 13:07:31.304604 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.304565 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4de1aea2-a2ee-4257-8255-e63c353af739-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-7cdec6-5969765647-58sz8\" (UID: \"4de1aea2-a2ee-4257-8255-e63c353af739\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-7cdec6-5969765647-58sz8" Oct 31 13:07:31.305013 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.285706 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-7cdec6-5969765647-58sz8"] Oct 31 13:07:31.406901 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.406863 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4de1aea2-a2ee-4257-8255-e63c353af739-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-7cdec6-5969765647-58sz8\" (UID: \"4de1aea2-a2ee-4257-8255-e63c353af739\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-7cdec6-5969765647-58sz8" Oct 31 13:07:31.407220 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.407195 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq2f5\" (UniqueName: \"kubernetes.io/projected/4de1aea2-a2ee-4257-8255-e63c353af739-kube-api-access-rq2f5\") pod \"payload-pre-processing-e2e-shared-a-w0-7cdec6-5969765647-58sz8\" (UID: \"4de1aea2-a2ee-4257-8255-e63c353af739\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-7cdec6-5969765647-58sz8" Oct 31 13:07:31.408695 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.408669 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4de1aea2-a2ee-4257-8255-e63c353af739-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-7cdec6-5969765647-58sz8\" (UID: \"4de1aea2-a2ee-4257-8255-e63c353af739\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-7cdec6-5969765647-58sz8" Oct 31 13:07:31.428486 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.428450 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq2f5\" (UniqueName: \"kubernetes.io/projected/4de1aea2-a2ee-4257-8255-e63c353af739-kube-api-access-rq2f5\") pod \"payload-pre-processing-e2e-shared-a-w0-7cdec6-5969765647-58sz8\" (UID: \"4de1aea2-a2ee-4257-8255-e63c353af739\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-7cdec6-5969765647-58sz8" Oct 31 13:07:31.515071 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.514992 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-7cdec6-5969765647-58sz8" Oct 31 13:07:31.839403 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:31.839362 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-7cdec6-5969765647-58sz8"] Oct 31 13:07:31.847011 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:07:31.846981 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4de1aea2_a2ee_4257_8255_e63c353af739.slice/crio-6edc01cf32e328ef79fdcfe3da0ad8605990c47b4043f63282160540a12f08f4 WatchSource:0}: Error finding container 6edc01cf32e328ef79fdcfe3da0ad8605990c47b4043f63282160540a12f08f4: Status 404 returned error can't find the container with id 6edc01cf32e328ef79fdcfe3da0ad8605990c47b4043f63282160540a12f08f4 Oct 31 13:07:32.014410 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:32.014371 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-7cdec6-5969765647-58sz8" event={"ID":"4de1aea2-a2ee-4257-8255-e63c353af739","Type":"ContainerStarted","Data":"a56fc10c3e41c1ff6df791eb819bea017bd0ab07f913c98d491145e7a20d8218"} Oct 31 13:07:32.014605 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:32.014429 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-7cdec6-5969765647-58sz8" event={"ID":"4de1aea2-a2ee-4257-8255-e63c353af739","Type":"ContainerStarted","Data":"6edc01cf32e328ef79fdcfe3da0ad8605990c47b4043f63282160540a12f08f4"} Oct 31 13:07:32.014694 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:32.014633 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-7cdec6-5969765647-58sz8" Oct 31 13:07:32.048450 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:32.048394 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-7cdec6-5969765647-58sz8" podStartSLOduration=1.048376094 podStartE2EDuration="1.048376094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:07:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:07:32.04619746 +0000 UTC m=+2102.753783752" watchObservedRunningTime="2026-10-31 13:07:32.048376094 +0000 UTC m=+2102.755962392" Oct 31 13:07:32.247709 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:32.234760 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6cef79-7dd4f54867-sqjfq"] Oct 31 13:07:32.247709 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:32.235724 5693 cpu_manager.go:414] "containerMap: removing container" podUID="9d0d805e-ce8f-4cd0-91c1-7f00fa21e672" containerName="istio-proxy" Oct 31 13:07:32.247709 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:32.235744 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d0d805e-ce8f-4cd0-91c1-7f00fa21e672" containerName="istio-proxy" Oct 31 13:07:32.247709 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:32.239687 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6cef79-7dd4f54867-sqjfq" Oct 31 13:07:32.259661 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:32.252914 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6cef79-7dd4f54867-sqjfq"] Oct 31 13:07:32.259661 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:32.253188 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-6cef79\"" Oct 31 13:07:32.261593 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:32.261559 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-6cef79-8695b89999-mg5vs"] Oct 31 13:07:32.282007 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:32.281976 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2rz4\" (UniqueName: \"kubernetes.io/projected/d6d2c321-1fae-4360-b6fa-c11cf95fece6-kube-api-access-k2rz4\") pod \"payload-pre-processing-e2e-shared-a-w1-6cef79-7dd4f54867-sqjfq\" (UID: \"d6d2c321-1fae-4360-b6fa-c11cf95fece6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6cef79-7dd4f54867-sqjfq" Oct 31 13:07:32.282181 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:32.282149 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6d2c321-1fae-4360-b6fa-c11cf95fece6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-6cef79-7dd4f54867-sqjfq\" (UID: \"d6d2c321-1fae-4360-b6fa-c11cf95fece6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6cef79-7dd4f54867-sqjfq" Oct 31 13:07:32.284135 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:32.284089 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6cef79-8695b89999-mg5vs" Oct 31 13:07:32.292748 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:32.292624 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-6cef79-8695b89999-mg5vs"] Oct 31 13:07:32.302002 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:32.301975 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-6cef79-dockercfg-rkbwl\"" Oct 31 13:07:32.388883 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:32.388833 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2rz4\" (UniqueName: \"kubernetes.io/projected/d6d2c321-1fae-4360-b6fa-c11cf95fece6-kube-api-access-k2rz4\") pod \"payload-pre-processing-e2e-shared-a-w1-6cef79-7dd4f54867-sqjfq\" (UID: \"d6d2c321-1fae-4360-b6fa-c11cf95fece6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6cef79-7dd4f54867-sqjfq" Oct 31 13:07:32.389106 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:32.388933 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl4f8\" (UniqueName: \"kubernetes.io/projected/6b617953-8a53-49e1-bc81-80d029d098bd-kube-api-access-zl4f8\") pod \"payload-processing-e2e-shared-a-w1-6cef79-8695b89999-mg5vs\" (UID: \"6b617953-8a53-49e1-bc81-80d029d098bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6cef79-8695b89999-mg5vs" Oct 31 13:07:32.389106 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:32.389073 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b617953-8a53-49e1-bc81-80d029d098bd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-6cef79-8695b89999-mg5vs\" (UID: \"6b617953-8a53-49e1-bc81-80d029d098bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6cef79-8695b89999-mg5vs" Oct 31 13:07:32.389240 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:32.389111 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6d2c321-1fae-4360-b6fa-c11cf95fece6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-6cef79-7dd4f54867-sqjfq\" (UID: \"d6d2c321-1fae-4360-b6fa-c11cf95fece6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6cef79-7dd4f54867-sqjfq" Oct 31 13:07:32.390304 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:32.390272 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6d2c321-1fae-4360-b6fa-c11cf95fece6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-6cef79-7dd4f54867-sqjfq\" (UID: \"d6d2c321-1fae-4360-b6fa-c11cf95fece6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6cef79-7dd4f54867-sqjfq" Oct 31 13:07:32.401458 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:32.401434 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2rz4\" (UniqueName: \"kubernetes.io/projected/d6d2c321-1fae-4360-b6fa-c11cf95fece6-kube-api-access-k2rz4\") pod \"payload-pre-processing-e2e-shared-a-w1-6cef79-7dd4f54867-sqjfq\" (UID: \"d6d2c321-1fae-4360-b6fa-c11cf95fece6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6cef79-7dd4f54867-sqjfq" Oct 31 13:07:32.491376 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:32.491323 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl4f8\" (UniqueName: \"kubernetes.io/projected/6b617953-8a53-49e1-bc81-80d029d098bd-kube-api-access-zl4f8\") pod \"payload-processing-e2e-shared-a-w1-6cef79-8695b89999-mg5vs\" (UID: \"6b617953-8a53-49e1-bc81-80d029d098bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6cef79-8695b89999-mg5vs" Oct 31 13:07:32.491575 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:32.491473 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b617953-8a53-49e1-bc81-80d029d098bd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-6cef79-8695b89999-mg5vs\" (UID: \"6b617953-8a53-49e1-bc81-80d029d098bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6cef79-8695b89999-mg5vs" Oct 31 13:07:32.492588 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:32.492562 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b617953-8a53-49e1-bc81-80d029d098bd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-6cef79-8695b89999-mg5vs\" (UID: \"6b617953-8a53-49e1-bc81-80d029d098bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6cef79-8695b89999-mg5vs" Oct 31 13:07:32.505008 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:32.504916 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl4f8\" (UniqueName: \"kubernetes.io/projected/6b617953-8a53-49e1-bc81-80d029d098bd-kube-api-access-zl4f8\") pod \"payload-processing-e2e-shared-a-w1-6cef79-8695b89999-mg5vs\" (UID: \"6b617953-8a53-49e1-bc81-80d029d098bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6cef79-8695b89999-mg5vs" Oct 31 13:07:32.588488 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:32.588454 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6cef79-7dd4f54867-sqjfq" Oct 31 13:07:32.621459 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:32.621414 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6cef79-8695b89999-mg5vs" Oct 31 13:07:32.876071 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:32.876041 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6cef79-7dd4f54867-sqjfq"] Oct 31 13:07:32.877505 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:07:32.877479 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6d2c321_1fae_4360_b6fa_c11cf95fece6.slice/crio-bc51ffc9c6e272f706d5e3fe70a991cdd2c3671713fef6a519e4bf81f9af995a WatchSource:0}: Error finding container bc51ffc9c6e272f706d5e3fe70a991cdd2c3671713fef6a519e4bf81f9af995a: Status 404 returned error can't find the container with id bc51ffc9c6e272f706d5e3fe70a991cdd2c3671713fef6a519e4bf81f9af995a Oct 31 13:07:32.925980 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:32.925952 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-6cef79-8695b89999-mg5vs"] Oct 31 13:07:32.927244 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:07:32.927212 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b617953_8a53_49e1_bc81_80d029d098bd.slice/crio-550bbfb6110f3244dfbc62798290ee5ff2cd8a3e3984ed35095e753bc425c6cf WatchSource:0}: Error finding container 550bbfb6110f3244dfbc62798290ee5ff2cd8a3e3984ed35095e753bc425c6cf: Status 404 returned error can't find the container with id 550bbfb6110f3244dfbc62798290ee5ff2cd8a3e3984ed35095e753bc425c6cf Oct 31 13:07:33.029691 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:33.029623 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6cef79-7dd4f54867-sqjfq" event={"ID":"d6d2c321-1fae-4360-b6fa-c11cf95fece6","Type":"ContainerStarted","Data":"824060e06c4e5f30dcdfeab243410f4264962620f57cffa4a0a0b5bf274082c0"} Oct 31 13:07:33.029876 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:33.029734 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6cef79-7dd4f54867-sqjfq" event={"ID":"d6d2c321-1fae-4360-b6fa-c11cf95fece6","Type":"ContainerStarted","Data":"bc51ffc9c6e272f706d5e3fe70a991cdd2c3671713fef6a519e4bf81f9af995a"} Oct 31 13:07:33.029967 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:33.029941 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6cef79-7dd4f54867-sqjfq" Oct 31 13:07:33.031162 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:33.031138 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6cef79-8695b89999-mg5vs" event={"ID":"6b617953-8a53-49e1-bc81-80d029d098bd","Type":"ContainerStarted","Data":"5c36c64c851e9a50c669aefdef3888b830343a27b5d2ebffb565e3e324836f76"} Oct 31 13:07:33.031247 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:33.031186 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6cef79-8695b89999-mg5vs" event={"ID":"6b617953-8a53-49e1-bc81-80d029d098bd","Type":"ContainerStarted","Data":"550bbfb6110f3244dfbc62798290ee5ff2cd8a3e3984ed35095e753bc425c6cf"} Oct 31 13:07:33.057257 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:33.057007 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6cef79-7dd4f54867-sqjfq" podStartSLOduration=1.056987857 podStartE2EDuration="1.056987857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:07:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:07:33.056247104 +0000 UTC m=+2103.763833433" watchObservedRunningTime="2026-10-31 13:07:33.056987857 +0000 UTC m=+2103.764574155" Oct 31 13:07:33.085773 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:33.085615 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6cef79-8695b89999-mg5vs" podStartSLOduration=1.085597347 podStartE2EDuration="1.085597347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:07:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:07:33.081717707 +0000 UTC m=+2103.789304003" watchObservedRunningTime="2026-10-31 13:07:33.085597347 +0000 UTC m=+2103.793183644" Oct 31 13:07:33.383545 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:33.383512 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-45bcad-77bfcf8bc9-pk9mj"] Oct 31 13:07:33.387714 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:33.387696 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-45bcad-77bfcf8bc9-pk9mj" Oct 31 13:07:33.421074 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:33.421009 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-45bcad-77bfcf8bc9-pk9mj"] Oct 31 13:07:33.432140 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:33.432097 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-45bcad-dockercfg-tgmct\"" Oct 31 13:07:33.432334 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:33.432191 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-45bcad\"" Oct 31 13:07:33.523698 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:33.523489 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84wt5\" (UniqueName: \"kubernetes.io/projected/a1b0fe6e-146d-4f30-90c2-d44f49d6e17f-kube-api-access-84wt5\") pod \"payload-processing-e2e-shared-a-w2-45bcad-77bfcf8bc9-pk9mj\" (UID: \"a1b0fe6e-146d-4f30-90c2-d44f49d6e17f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-45bcad-77bfcf8bc9-pk9mj" Oct 31 13:07:33.523698 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:33.523592 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1b0fe6e-146d-4f30-90c2-d44f49d6e17f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-45bcad-77bfcf8bc9-pk9mj\" (UID: \"a1b0fe6e-146d-4f30-90c2-d44f49d6e17f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-45bcad-77bfcf8bc9-pk9mj" Oct 31 13:07:33.625754 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:33.625717 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84wt5\" (UniqueName: \"kubernetes.io/projected/a1b0fe6e-146d-4f30-90c2-d44f49d6e17f-kube-api-access-84wt5\") pod \"payload-processing-e2e-shared-a-w2-45bcad-77bfcf8bc9-pk9mj\" (UID: \"a1b0fe6e-146d-4f30-90c2-d44f49d6e17f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-45bcad-77bfcf8bc9-pk9mj" Oct 31 13:07:33.625978 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:33.625807 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1b0fe6e-146d-4f30-90c2-d44f49d6e17f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-45bcad-77bfcf8bc9-pk9mj\" (UID: \"a1b0fe6e-146d-4f30-90c2-d44f49d6e17f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-45bcad-77bfcf8bc9-pk9mj" Oct 31 13:07:33.626562 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:33.626539 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1b0fe6e-146d-4f30-90c2-d44f49d6e17f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-45bcad-77bfcf8bc9-pk9mj\" (UID: \"a1b0fe6e-146d-4f30-90c2-d44f49d6e17f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-45bcad-77bfcf8bc9-pk9mj" Oct 31 13:07:33.638741 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:33.638672 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84wt5\" (UniqueName: \"kubernetes.io/projected/a1b0fe6e-146d-4f30-90c2-d44f49d6e17f-kube-api-access-84wt5\") pod \"payload-processing-e2e-shared-a-w2-45bcad-77bfcf8bc9-pk9mj\" (UID: \"a1b0fe6e-146d-4f30-90c2-d44f49d6e17f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-45bcad-77bfcf8bc9-pk9mj" Oct 31 13:07:33.702626 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:33.702587 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-45bcad-77bfcf8bc9-pk9mj" Oct 31 13:07:34.015358 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:34.015329 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-45bcad-77bfcf8bc9-pk9mj"] Oct 31 13:07:34.016485 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:07:34.016459 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1b0fe6e_146d_4f30_90c2_d44f49d6e17f.slice/crio-477ae209df062042d03a2a8a2578ea39f4dc81581d8a8181cc9c69e4fbe283e1 WatchSource:0}: Error finding container 477ae209df062042d03a2a8a2578ea39f4dc81581d8a8181cc9c69e4fbe283e1: Status 404 returned error can't find the container with id 477ae209df062042d03a2a8a2578ea39f4dc81581d8a8181cc9c69e4fbe283e1 Oct 31 13:07:34.039901 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:34.039872 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-45bcad-77bfcf8bc9-pk9mj" event={"ID":"a1b0fe6e-146d-4f30-90c2-d44f49d6e17f","Type":"ContainerStarted","Data":"477ae209df062042d03a2a8a2578ea39f4dc81581d8a8181cc9c69e4fbe283e1"} Oct 31 13:07:34.040261 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:34.040241 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6cef79-8695b89999-mg5vs" Oct 31 13:07:35.050414 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:35.050381 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-45bcad-77bfcf8bc9-pk9mj" event={"ID":"a1b0fe6e-146d-4f30-90c2-d44f49d6e17f","Type":"ContainerStarted","Data":"799602286a71706e495ec5e66c4f20420baefa9d2087cd03f2cc44b52694b5e5"} Oct 31 13:07:35.050931 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:35.050674 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-45bcad-77bfcf8bc9-pk9mj" Oct 31 13:07:35.077665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:35.076581 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-45bcad-77bfcf8bc9-pk9mj" podStartSLOduration=2.076561254 podStartE2EDuration="2.076561254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:07:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:07:35.074770634 +0000 UTC m=+2105.782356913" watchObservedRunningTime="2026-10-31 13:07:35.076561254 +0000 UTC m=+2105.784147548" Oct 31 13:07:37.915197 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:37.915166 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4574ec-bdd99c999-65drk"] Oct 31 13:07:37.919354 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:37.919336 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4574ec-bdd99c999-65drk" Oct 31 13:07:37.931503 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:37.931469 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-4574ec\"" Oct 31 13:07:37.935416 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:37.935386 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4574ec-bdd99c999-65drk"] Oct 31 13:07:37.935952 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:37.935926 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-4574ec-dockercfg-f9ppw\"" Oct 31 13:07:37.984487 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:37.984455 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-4574ec-76b8ddbc7f-7jbsx"] Oct 31 13:07:37.989477 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:37.989457 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4574ec-76b8ddbc7f-7jbsx" Oct 31 13:07:37.995595 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:37.995572 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-4574ec-76b8ddbc7f-7jbsx"] Oct 31 13:07:38.009624 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:38.009576 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fbf694c-e0a8-4a8a-a214-a06c58f7642b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-4574ec-bdd99c999-65drk\" (UID: \"8fbf694c-e0a8-4a8a-a214-a06c58f7642b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4574ec-bdd99c999-65drk" Oct 31 13:07:38.009799 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:38.009750 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lvxs\" (UniqueName: \"kubernetes.io/projected/8fbf694c-e0a8-4a8a-a214-a06c58f7642b-kube-api-access-5lvxs\") pod \"payload-pre-processing-e2e-shared-a-w3-4574ec-bdd99c999-65drk\" (UID: \"8fbf694c-e0a8-4a8a-a214-a06c58f7642b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4574ec-bdd99c999-65drk" Oct 31 13:07:38.131662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:38.129665 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fbf694c-e0a8-4a8a-a214-a06c58f7642b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-4574ec-bdd99c999-65drk\" (UID: \"8fbf694c-e0a8-4a8a-a214-a06c58f7642b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4574ec-bdd99c999-65drk" Oct 31 13:07:38.131662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:38.129834 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll2gv\" (UniqueName: \"kubernetes.io/projected/eccfdc26-fb5e-4d07-8f67-43723a93450b-kube-api-access-ll2gv\") pod \"payload-processing-e2e-shared-a-w3-4574ec-76b8ddbc7f-7jbsx\" (UID: \"eccfdc26-fb5e-4d07-8f67-43723a93450b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4574ec-76b8ddbc7f-7jbsx" Oct 31 13:07:38.131662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:38.129887 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eccfdc26-fb5e-4d07-8f67-43723a93450b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-4574ec-76b8ddbc7f-7jbsx\" (UID: \"eccfdc26-fb5e-4d07-8f67-43723a93450b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4574ec-76b8ddbc7f-7jbsx" Oct 31 13:07:38.131662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:38.129921 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lvxs\" (UniqueName: \"kubernetes.io/projected/8fbf694c-e0a8-4a8a-a214-a06c58f7642b-kube-api-access-5lvxs\") pod \"payload-pre-processing-e2e-shared-a-w3-4574ec-bdd99c999-65drk\" (UID: \"8fbf694c-e0a8-4a8a-a214-a06c58f7642b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4574ec-bdd99c999-65drk" Oct 31 13:07:38.131662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:38.131386 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fbf694c-e0a8-4a8a-a214-a06c58f7642b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-4574ec-bdd99c999-65drk\" (UID: \"8fbf694c-e0a8-4a8a-a214-a06c58f7642b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4574ec-bdd99c999-65drk" Oct 31 13:07:38.201135 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:38.201030 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lvxs\" (UniqueName: \"kubernetes.io/projected/8fbf694c-e0a8-4a8a-a214-a06c58f7642b-kube-api-access-5lvxs\") pod \"payload-pre-processing-e2e-shared-a-w3-4574ec-bdd99c999-65drk\" (UID: \"8fbf694c-e0a8-4a8a-a214-a06c58f7642b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4574ec-bdd99c999-65drk" Oct 31 13:07:38.237724 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:38.231590 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll2gv\" (UniqueName: \"kubernetes.io/projected/eccfdc26-fb5e-4d07-8f67-43723a93450b-kube-api-access-ll2gv\") pod \"payload-processing-e2e-shared-a-w3-4574ec-76b8ddbc7f-7jbsx\" (UID: \"eccfdc26-fb5e-4d07-8f67-43723a93450b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4574ec-76b8ddbc7f-7jbsx" Oct 31 13:07:38.237724 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:38.231669 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eccfdc26-fb5e-4d07-8f67-43723a93450b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-4574ec-76b8ddbc7f-7jbsx\" (UID: \"eccfdc26-fb5e-4d07-8f67-43723a93450b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4574ec-76b8ddbc7f-7jbsx" Oct 31 13:07:38.237724 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:38.232778 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eccfdc26-fb5e-4d07-8f67-43723a93450b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-4574ec-76b8ddbc7f-7jbsx\" (UID: \"eccfdc26-fb5e-4d07-8f67-43723a93450b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4574ec-76b8ddbc7f-7jbsx" Oct 31 13:07:38.237724 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:38.234961 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4574ec-bdd99c999-65drk" Oct 31 13:07:38.250558 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:38.248109 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll2gv\" (UniqueName: \"kubernetes.io/projected/eccfdc26-fb5e-4d07-8f67-43723a93450b-kube-api-access-ll2gv\") pod \"payload-processing-e2e-shared-a-w3-4574ec-76b8ddbc7f-7jbsx\" (UID: \"eccfdc26-fb5e-4d07-8f67-43723a93450b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4574ec-76b8ddbc7f-7jbsx" Oct 31 13:07:38.307896 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:38.307860 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4574ec-76b8ddbc7f-7jbsx" Oct 31 13:07:38.579219 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:38.579190 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4574ec-bdd99c999-65drk"] Oct 31 13:07:38.580513 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:07:38.580482 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fbf694c_e0a8_4a8a_a214_a06c58f7642b.slice/crio-5ec1f014603c0fd1629bb6c3c247a01e0823b422e0daa122fe9acd0a9259a1ff WatchSource:0}: Error finding container 5ec1f014603c0fd1629bb6c3c247a01e0823b422e0daa122fe9acd0a9259a1ff: Status 404 returned error can't find the container with id 5ec1f014603c0fd1629bb6c3c247a01e0823b422e0daa122fe9acd0a9259a1ff Oct 31 13:07:38.665615 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:38.665574 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-4574ec-76b8ddbc7f-7jbsx"] Oct 31 13:07:38.666615 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:07:38.666580 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeccfdc26_fb5e_4d07_8f67_43723a93450b.slice/crio-c2ef6d7c5bc7690f66a664027080d639de3614f22a8341f917b56774dff9205d WatchSource:0}: Error finding container c2ef6d7c5bc7690f66a664027080d639de3614f22a8341f917b56774dff9205d: Status 404 returned error can't find the container with id c2ef6d7c5bc7690f66a664027080d639de3614f22a8341f917b56774dff9205d Oct 31 13:07:39.192660 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:39.192600 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4574ec-76b8ddbc7f-7jbsx" event={"ID":"eccfdc26-fb5e-4d07-8f67-43723a93450b","Type":"ContainerStarted","Data":"ce153237605e2a96e3b560565ea5d20b532e109bb30efa56acb7487771f1afd0"} Oct 31 13:07:39.193112 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:39.192705 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4574ec-76b8ddbc7f-7jbsx" Oct 31 13:07:39.193112 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:39.192732 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4574ec-76b8ddbc7f-7jbsx" event={"ID":"eccfdc26-fb5e-4d07-8f67-43723a93450b","Type":"ContainerStarted","Data":"c2ef6d7c5bc7690f66a664027080d639de3614f22a8341f917b56774dff9205d"} Oct 31 13:07:39.194419 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:39.194394 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4574ec-bdd99c999-65drk" event={"ID":"8fbf694c-e0a8-4a8a-a214-a06c58f7642b","Type":"ContainerStarted","Data":"14a881eda934b8b064d3794928889c4c4473eaa892f0a40f2572e33c56ae0640"} Oct 31 13:07:39.194533 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:39.194450 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4574ec-bdd99c999-65drk" Oct 31 13:07:39.194533 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:39.194474 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4574ec-bdd99c999-65drk" event={"ID":"8fbf694c-e0a8-4a8a-a214-a06c58f7642b","Type":"ContainerStarted","Data":"5ec1f014603c0fd1629bb6c3c247a01e0823b422e0daa122fe9acd0a9259a1ff"} Oct 31 13:07:39.212948 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:39.212907 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4574ec-76b8ddbc7f-7jbsx" podStartSLOduration=2.212891258 podStartE2EDuration="2.212891258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:07:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:07:39.210815479 +0000 UTC m=+2109.918401774" watchObservedRunningTime="2026-10-31 13:07:39.212891258 +0000 UTC m=+2109.920477553" Oct 31 13:07:39.231351 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:39.231299 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4574ec-bdd99c999-65drk" podStartSLOduration=2.231287571 podStartE2EDuration="2.231287571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:07:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:07:39.228739358 +0000 UTC m=+2109.936325656" watchObservedRunningTime="2026-10-31 13:07:39.231287571 +0000 UTC m=+2109.938873869" Oct 31 13:07:40.255071 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:40.255035 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r"] Oct 31 13:07:40.259419 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:40.259397 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r" Oct 31 13:07:40.270657 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:40.270208 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r"] Oct 31 13:07:40.270657 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:40.270286 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-45bcad-serving-cert\"" Oct 31 13:07:40.396016 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:40.395982 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f31ac998-2cc1-482d-8bbe-32ec51bc5dd1-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r\" (UID: \"f31ac998-2cc1-482d-8bbe-32ec51bc5dd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r" Oct 31 13:07:40.396255 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:40.396234 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f31ac998-2cc1-482d-8bbe-32ec51bc5dd1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r\" (UID: \"f31ac998-2cc1-482d-8bbe-32ec51bc5dd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r" Oct 31 13:07:40.396340 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:40.396313 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f31ac998-2cc1-482d-8bbe-32ec51bc5dd1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r\" (UID: \"f31ac998-2cc1-482d-8bbe-32ec51bc5dd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r" Oct 31 13:07:40.396400 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:40.396365 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c2x6\" (UniqueName: \"kubernetes.io/projected/f31ac998-2cc1-482d-8bbe-32ec51bc5dd1-kube-api-access-9c2x6\") pod \"maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r\" (UID: \"f31ac998-2cc1-482d-8bbe-32ec51bc5dd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r" Oct 31 13:07:40.498091 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:40.498049 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f31ac998-2cc1-482d-8bbe-32ec51bc5dd1-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r\" (UID: \"f31ac998-2cc1-482d-8bbe-32ec51bc5dd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r" Oct 31 13:07:40.498306 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:40.498150 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f31ac998-2cc1-482d-8bbe-32ec51bc5dd1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r\" (UID: \"f31ac998-2cc1-482d-8bbe-32ec51bc5dd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r" Oct 31 13:07:40.498306 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:40.498242 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f31ac998-2cc1-482d-8bbe-32ec51bc5dd1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r\" (UID: \"f31ac998-2cc1-482d-8bbe-32ec51bc5dd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r" Oct 31 13:07:40.498438 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:40.498329 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c2x6\" (UniqueName: \"kubernetes.io/projected/f31ac998-2cc1-482d-8bbe-32ec51bc5dd1-kube-api-access-9c2x6\") pod \"maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r\" (UID: \"f31ac998-2cc1-482d-8bbe-32ec51bc5dd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r" Oct 31 13:07:40.498438 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:07:40.498426 5693 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45bcad-serving-cert: secret "maas-api-e2e-shared-a-w2-45bcad-serving-cert" not found Oct 31 13:07:40.498554 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:07:40.498498 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f31ac998-2cc1-482d-8bbe-32ec51bc5dd1-maas-api-tls podName:f31ac998-2cc1-482d-8bbe-32ec51bc5dd1 nodeName:}" failed. No retries permitted until 2026-10-31 13:07:40.998477034 +0000 UTC m=+2111.706063327 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f31ac998-2cc1-482d-8bbe-32ec51bc5dd1-maas-api-tls") pod "maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r" (UID: "f31ac998-2cc1-482d-8bbe-32ec51bc5dd1") : secret "maas-api-e2e-shared-a-w2-45bcad-serving-cert" not found Oct 31 13:07:40.499359 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:40.499333 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f31ac998-2cc1-482d-8bbe-32ec51bc5dd1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r\" (UID: \"f31ac998-2cc1-482d-8bbe-32ec51bc5dd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r" Oct 31 13:07:40.501543 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:40.501520 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f31ac998-2cc1-482d-8bbe-32ec51bc5dd1-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r\" (UID: \"f31ac998-2cc1-482d-8bbe-32ec51bc5dd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r" Oct 31 13:07:40.513678 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:40.513591 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c2x6\" (UniqueName: \"kubernetes.io/projected/f31ac998-2cc1-482d-8bbe-32ec51bc5dd1-kube-api-access-9c2x6\") pod \"maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r\" (UID: \"f31ac998-2cc1-482d-8bbe-32ec51bc5dd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r" Oct 31 13:07:41.006458 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:41.006400 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f31ac998-2cc1-482d-8bbe-32ec51bc5dd1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r\" (UID: \"f31ac998-2cc1-482d-8bbe-32ec51bc5dd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r" Oct 31 13:07:41.012120 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:41.009682 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f31ac998-2cc1-482d-8bbe-32ec51bc5dd1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r\" (UID: \"f31ac998-2cc1-482d-8bbe-32ec51bc5dd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r" Oct 31 13:07:41.188194 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:41.188151 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r" Oct 31 13:07:41.486948 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:41.486919 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r"] Oct 31 13:07:41.487591 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:07:41.487566 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf31ac998_2cc1_482d_8bbe_32ec51bc5dd1.slice/crio-4b6bddd1095e38d3146bcde82679aedddbb52e5915e10e4f94a01b6ec4255ef1 WatchSource:0}: Error finding container 4b6bddd1095e38d3146bcde82679aedddbb52e5915e10e4f94a01b6ec4255ef1: Status 404 returned error can't find the container with id 4b6bddd1095e38d3146bcde82679aedddbb52e5915e10e4f94a01b6ec4255ef1 Oct 31 13:07:42.213794 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:42.213760 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r" event={"ID":"f31ac998-2cc1-482d-8bbe-32ec51bc5dd1","Type":"ContainerStarted","Data":"f0d1273800fcbdd1b2fb9664ab02d6e8e2fbf74165ff406c8b638a9da587d89a"} Oct 31 13:07:42.213954 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:42.213810 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r" event={"ID":"f31ac998-2cc1-482d-8bbe-32ec51bc5dd1","Type":"ContainerStarted","Data":"4b6bddd1095e38d3146bcde82679aedddbb52e5915e10e4f94a01b6ec4255ef1"} Oct 31 13:07:42.213954 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:42.213876 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r" Oct 31 13:07:42.237065 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:42.237014 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r" podStartSLOduration=1.722948682 podStartE2EDuration="2.236999603s" totalImagesPullingTime="514.050921ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:07:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:07:42.234284928 +0000 UTC m=+2112.941871222" watchObservedRunningTime="2026-10-31 13:07:42.236999603 +0000 UTC m=+2112.944585898" Oct 31 13:07:43.034563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:43.034529 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-7cdec6-5969765647-58sz8" Oct 31 13:07:44.045264 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:44.044702 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6cef79-7dd4f54867-sqjfq" Oct 31 13:07:45.054107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:45.054077 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6cef79-8695b89999-mg5vs" Oct 31 13:07:46.098724 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:46.092834 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-45bcad-77bfcf8bc9-pk9mj" Oct 31 13:07:46.860725 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:46.859771 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm"] Oct 31 13:07:46.860725 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:46.860019 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" podUID="2f1de21e-24e3-46c6-a3cc-d8c16157ae4d" containerName="istio-proxy" containerID="cri-o://eaf95044644324f6393b7ae78b510bcb7e3fcf0be76b5262e677a3aa6393af21" gracePeriod=30 Oct 31 13:07:47.578667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:47.577774 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" podUID="ff9a9909-6fc3-475b-a9fe-562ec150c76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 31 13:07:47.816679 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:47.815055 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" podUID="710a52fa-2bc4-4bb9-96be-4b08dc60de67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:07:47.845723 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:47.824727 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" podUID="8fb66a94-c399-43fb-945c-610ca63f0d76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:07:47.947667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:47.933770 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" podUID="e5a40b79-9879-4981-ab9f-446e1750a013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 13:07:48.238379 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:48.238337 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r" Oct 31 13:07:48.694725 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:07:48.694688 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:07:50.204154 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:50.204126 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4574ec-76b8ddbc7f-7jbsx" Oct 31 13:07:50.204534 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:50.204196 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4574ec-bdd99c999-65drk" Oct 31 13:07:51.004819 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:50.998707 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx"] Oct 31 13:07:51.006668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:51.006617 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" Oct 31 13:07:51.018828 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:51.018553 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx"] Oct 31 13:07:51.058258 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:51.058226 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-377f60-openshift-default-dockercfg-4cfn6\"" Oct 31 13:07:51.083494 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:51.083464 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-istio-data\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx\" (UID: \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" Oct 31 13:07:51.083723 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:51.083521 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg9ws\" (UniqueName: \"kubernetes.io/projected/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-kube-api-access-pg9ws\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx\" (UID: \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" Oct 31 13:07:51.083723 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:51.083629 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-credential-socket\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx\" (UID: \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" Oct 31 13:07:51.083873 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:51.083724 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-workload-socket\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx\" (UID: \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" Oct 31 13:07:51.083873 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:51.083747 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-istio-podinfo\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx\" (UID: \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" Oct 31 13:07:51.083873 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:51.083852 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-istio-envoy\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx\" (UID: \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" Oct 31 13:07:51.084050 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:51.083943 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-istiod-ca-cert\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx\" (UID: \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" Oct 31 13:07:51.084050 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:51.084020 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-workload-certs\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx\" (UID: \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" Oct 31 13:07:51.084159 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:51.084048 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-istio-token\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx\" (UID: \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" Oct 31 13:07:51.185310 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:51.185277 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-credential-socket\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx\" (UID: \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" Oct 31 13:07:51.185499 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:51.185368 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-workload-socket\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx\" (UID: \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" Oct 31 13:07:51.185499 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:51.185405 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-istio-podinfo\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx\" (UID: \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" Oct 31 13:07:51.185499 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:51.185454 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-istio-envoy\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx\" (UID: \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" Oct 31 13:07:51.185718 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:51.185537 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-istiod-ca-cert\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx\" (UID: \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" Oct 31 13:07:51.185718 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:51.185620 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-workload-certs\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx\" (UID: \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" Oct 31 13:07:51.185718 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:51.185668 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-istio-token\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx\" (UID: \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" Oct 31 13:07:51.185718 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:51.185680 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-credential-socket\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx\" (UID: \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" Oct 31 13:07:51.185907 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:51.185732 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-istio-data\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx\" (UID: \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" Oct 31 13:07:51.185907 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:51.185769 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg9ws\" (UniqueName: \"kubernetes.io/projected/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-kube-api-access-pg9ws\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx\" (UID: \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" Oct 31 13:07:51.186192 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:51.186165 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-workload-socket\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx\" (UID: \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" Oct 31 13:07:51.186528 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:51.186504 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-workload-certs\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx\" (UID: \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" Oct 31 13:07:51.186609 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:51.186586 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-istiod-ca-cert\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx\" (UID: \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" Oct 31 13:07:51.186609 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:51.186178 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-istio-data\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx\" (UID: \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" Oct 31 13:07:51.188978 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:51.188954 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-istio-envoy\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx\" (UID: \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" Oct 31 13:07:51.189289 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:51.189243 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-istio-podinfo\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx\" (UID: \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" Oct 31 13:07:51.201954 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:51.201920 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-istio-token\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx\" (UID: \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" Oct 31 13:07:51.202979 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:51.202951 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg9ws\" (UniqueName: \"kubernetes.io/projected/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-kube-api-access-pg9ws\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx\" (UID: \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" Oct 31 13:07:51.342767 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:51.342686 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" Oct 31 13:07:51.668299 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:51.668267 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx"] Oct 31 13:07:51.676407 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:51.676372 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:07:51.676498 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:51.676443 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:07:51.676498 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:51.676478 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:07:52.193101 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:52.193073 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" Oct 31 13:07:52.238244 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:52.238213 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-istiod-ca-cert\") pod \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\" (UID: \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\") " Oct 31 13:07:52.238426 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:52.238297 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-credential-socket\") pod \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\" (UID: \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\") " Oct 31 13:07:52.238426 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:52.238322 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-workload-certs\") pod \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\" (UID: \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\") " Oct 31 13:07:52.238426 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:52.238357 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-kube-api-access-rrpc4\" (UniqueName: \"kubernetes.io/projected/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-kube-api-access-rrpc4\") pod \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\" (UID: \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\") " Oct 31 13:07:52.238600 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:52.238492 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-workload-socket\") pod \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\" (UID: \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\") " Oct 31 13:07:52.238600 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:52.238558 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-istio-data\") pod \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\" (UID: \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\") " Oct 31 13:07:52.238728 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:52.238611 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-istio-token\" (UniqueName: \"kubernetes.io/projected/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-istio-token\") pod \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\" (UID: \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\") " Oct 31 13:07:52.238728 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:52.238662 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-istio-envoy\") pod \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\" (UID: \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\") " Oct 31 13:07:52.238834 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:52.238747 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-istio-podinfo\") pod \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\" (UID: \"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d\") " Oct 31 13:07:52.238834 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:52.238803 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-credential-socket" pod "2f1de21e-24e3-46c6-a3cc-d8c16157ae4d" (UID: "2f1de21e-24e3-46c6-a3cc-d8c16157ae4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:07:52.238916 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:52.238754 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-istiod-ca-cert" pod "2f1de21e-24e3-46c6-a3cc-d8c16157ae4d" (UID: "2f1de21e-24e3-46c6-a3cc-d8c16157ae4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:07:52.238916 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:52.238902 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-workload-socket" pod "2f1de21e-24e3-46c6-a3cc-d8c16157ae4d" (UID: "2f1de21e-24e3-46c6-a3cc-d8c16157ae4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:07:52.239191 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:52.239165 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-istio-data" pod "2f1de21e-24e3-46c6-a3cc-d8c16157ae4d" (UID: "2f1de21e-24e3-46c6-a3cc-d8c16157ae4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:07:52.239362 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:52.239319 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-workload-certs" pod "2f1de21e-24e3-46c6-a3cc-d8c16157ae4d" (UID: "2f1de21e-24e3-46c6-a3cc-d8c16157ae4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:07:52.239607 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:52.239559 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:52.239607 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:52.239584 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:52.239607 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:52.239599 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:52.239839 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:52.239616 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:52.239839 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:52.239630 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:52.241261 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:52.241231 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-kube-api-access-rrpc4" pod "2f1de21e-24e3-46c6-a3cc-d8c16157ae4d" (UID: "2f1de21e-24e3-46c6-a3cc-d8c16157ae4d"). InnerVolumeSpecName "kube-api-access-rrpc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:07:52.241261 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:52.241249 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-istio-token" pod "2f1de21e-24e3-46c6-a3cc-d8c16157ae4d" (UID: "2f1de21e-24e3-46c6-a3cc-d8c16157ae4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:07:52.241547 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:52.241525 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-istio-podinfo" pod "2f1de21e-24e3-46c6-a3cc-d8c16157ae4d" (UID: "2f1de21e-24e3-46c6-a3cc-d8c16157ae4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:07:52.241682 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:52.241621 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-istio-envoy" pod "2f1de21e-24e3-46c6-a3cc-d8c16157ae4d" (UID: "2f1de21e-24e3-46c6-a3cc-d8c16157ae4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:07:52.341013 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:52.340979 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:52.341013 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:52.341009 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrpc4\" (UniqueName: \"kubernetes.io/projected/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-kube-api-access-rrpc4\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:52.341013 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:52.341021 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:52.341242 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:52.341030 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:52.398257 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:52.398225 5693 generic.go:356] "Generic (PLEG): container finished" podID="2f1de21e-24e3-46c6-a3cc-d8c16157ae4d" containerID="eaf95044644324f6393b7ae78b510bcb7e3fcf0be76b5262e677a3aa6393af21" exitCode=0 Oct 31 13:07:52.398776 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:52.398287 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" Oct 31 13:07:52.398776 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:52.398303 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" event={"ID":"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d","Type":"ContainerDied","Data":"eaf95044644324f6393b7ae78b510bcb7e3fcf0be76b5262e677a3aa6393af21"} Oct 31 13:07:52.398776 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:52.398369 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm" event={"ID":"2f1de21e-24e3-46c6-a3cc-d8c16157ae4d","Type":"ContainerDied","Data":"ad0bc17ead5620d550d5287fa2628550acb7c246ebd3f9b99bd1555f37af0928"} Oct 31 13:07:52.398776 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:52.398399 5693 scope.go:122] "RemoveContainer" containerID="eaf95044644324f6393b7ae78b510bcb7e3fcf0be76b5262e677a3aa6393af21" Oct 31 13:07:52.400712 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:52.400684 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" event={"ID":"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6","Type":"ContainerStarted","Data":"f6f09ceee1023a3138095e8e21c5e30b90f34d0b64a389504ff470751a3e7e1c"} Oct 31 13:07:52.400834 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:52.400733 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" event={"ID":"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6","Type":"ContainerStarted","Data":"d84bae351fa140b4104960bc3a9c449179583aefac6fe9bc7251a4f3c19fe2d3"} Oct 31 13:07:52.410494 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:52.410477 5693 scope.go:122] "RemoveContainer" containerID="eaf95044644324f6393b7ae78b510bcb7e3fcf0be76b5262e677a3aa6393af21" Oct 31 13:07:52.410851 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:07:52.410831 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaf95044644324f6393b7ae78b510bcb7e3fcf0be76b5262e677a3aa6393af21\": container with ID starting with eaf95044644324f6393b7ae78b510bcb7e3fcf0be76b5262e677a3aa6393af21 not found: ID does not exist" containerID="eaf95044644324f6393b7ae78b510bcb7e3fcf0be76b5262e677a3aa6393af21" Oct 31 13:07:52.410926 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:52.410859 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaf95044644324f6393b7ae78b510bcb7e3fcf0be76b5262e677a3aa6393af21"} err="failed to get container status \"eaf95044644324f6393b7ae78b510bcb7e3fcf0be76b5262e677a3aa6393af21\": rpc error: code = NotFound desc = could not find container \"eaf95044644324f6393b7ae78b510bcb7e3fcf0be76b5262e677a3aa6393af21\": container with ID starting with eaf95044644324f6393b7ae78b510bcb7e3fcf0be76b5262e677a3aa6393af21 not found: ID does not exist" Oct 31 13:07:52.426605 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:52.426554 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" podStartSLOduration=2.426541503 podStartE2EDuration="2.426541503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:07:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:07:52.424274609 +0000 UTC m=+2123.131860917" watchObservedRunningTime="2026-10-31 13:07:52.426541503 +0000 UTC m=+2123.134127798" Oct 31 13:07:52.441053 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:52.441025 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm"] Oct 31 13:07:52.445610 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:52.445587 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-jftvm"] Oct 31 13:07:53.344039 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:53.344007 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" Oct 31 13:07:53.349844 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:53.349818 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" Oct 31 13:07:53.408885 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:53.408854 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f1de21e-24e3-46c6-a3cc-d8c16157ae4d" path="/var/lib/kubelet/pods/2f1de21e-24e3-46c6-a3cc-d8c16157ae4d/volumes" Oct 31 13:07:53.409340 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:53.409278 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" Oct 31 13:07:53.409402 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:53.409353 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" Oct 31 13:07:56.683703 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:56.683668 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-377f60-bdf4746c-r7hng"] Oct 31 13:07:56.684323 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:56.684308 5693 cpu_manager.go:414] "containerMap: removing container" podUID="2f1de21e-24e3-46c6-a3cc-d8c16157ae4d" containerName="istio-proxy" Oct 31 13:07:56.684365 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:56.684325 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f1de21e-24e3-46c6-a3cc-d8c16157ae4d" containerName="istio-proxy" Oct 31 13:07:56.684431 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:56.684421 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f1de21e-24e3-46c6-a3cc-d8c16157ae4d" containerName="istio-proxy" Oct 31 13:07:56.689188 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:56.689170 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-377f60-bdf4746c-r7hng" Oct 31 13:07:56.695783 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:56.695758 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-377f60\"" Oct 31 13:07:56.698060 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:56.698043 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-377f60-dockercfg-gsnkp\"" Oct 31 13:07:56.705293 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:56.705271 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-377f60-bdf4746c-r7hng"] Oct 31 13:07:56.740924 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:56.740888 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-377f60-c59666d89-b6xbr"] Oct 31 13:07:56.744754 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:56.744732 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-377f60-c59666d89-b6xbr" Oct 31 13:07:56.781082 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:56.781052 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-377f60-c59666d89-b6xbr"] Oct 31 13:07:56.846670 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:56.835692 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjnmm\" (UniqueName: \"kubernetes.io/projected/c2ca0d7c-7fb8-438d-9c27-09c1602339fa-kube-api-access-rjnmm\") pod \"payload-pre-processing-e2e-shared-b-w0-377f60-bdf4746c-r7hng\" (UID: \"c2ca0d7c-7fb8-438d-9c27-09c1602339fa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-377f60-bdf4746c-r7hng" Oct 31 13:07:56.891086 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:56.846980 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03b3ba33-5c21-487d-bee4-abc978d8cf06-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-377f60-c59666d89-b6xbr\" (UID: \"03b3ba33-5c21-487d-bee4-abc978d8cf06\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-377f60-c59666d89-b6xbr" Oct 31 13:07:56.891572 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:56.891528 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ca0d7c-7fb8-438d-9c27-09c1602339fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-377f60-bdf4746c-r7hng\" (UID: \"c2ca0d7c-7fb8-438d-9c27-09c1602339fa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-377f60-bdf4746c-r7hng" Oct 31 13:07:56.891894 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:56.891865 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldhms\" (UniqueName: \"kubernetes.io/projected/03b3ba33-5c21-487d-bee4-abc978d8cf06-kube-api-access-ldhms\") pod \"payload-processing-e2e-shared-b-w0-377f60-c59666d89-b6xbr\" (UID: \"03b3ba33-5c21-487d-bee4-abc978d8cf06\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-377f60-c59666d89-b6xbr" Oct 31 13:07:56.993935 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:56.993846 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldhms\" (UniqueName: \"kubernetes.io/projected/03b3ba33-5c21-487d-bee4-abc978d8cf06-kube-api-access-ldhms\") pod \"payload-processing-e2e-shared-b-w0-377f60-c59666d89-b6xbr\" (UID: \"03b3ba33-5c21-487d-bee4-abc978d8cf06\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-377f60-c59666d89-b6xbr" Oct 31 13:07:56.994107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:56.993951 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjnmm\" (UniqueName: \"kubernetes.io/projected/c2ca0d7c-7fb8-438d-9c27-09c1602339fa-kube-api-access-rjnmm\") pod \"payload-pre-processing-e2e-shared-b-w0-377f60-bdf4746c-r7hng\" (UID: \"c2ca0d7c-7fb8-438d-9c27-09c1602339fa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-377f60-bdf4746c-r7hng" Oct 31 13:07:56.994107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:56.993993 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03b3ba33-5c21-487d-bee4-abc978d8cf06-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-377f60-c59666d89-b6xbr\" (UID: \"03b3ba33-5c21-487d-bee4-abc978d8cf06\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-377f60-c59666d89-b6xbr" Oct 31 13:07:56.994107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:56.994028 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ca0d7c-7fb8-438d-9c27-09c1602339fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-377f60-bdf4746c-r7hng\" (UID: \"c2ca0d7c-7fb8-438d-9c27-09c1602339fa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-377f60-bdf4746c-r7hng" Oct 31 13:07:56.994829 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:56.994808 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03b3ba33-5c21-487d-bee4-abc978d8cf06-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-377f60-c59666d89-b6xbr\" (UID: \"03b3ba33-5c21-487d-bee4-abc978d8cf06\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-377f60-c59666d89-b6xbr" Oct 31 13:07:56.994931 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:56.994808 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ca0d7c-7fb8-438d-9c27-09c1602339fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-377f60-bdf4746c-r7hng\" (UID: \"c2ca0d7c-7fb8-438d-9c27-09c1602339fa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-377f60-bdf4746c-r7hng" Oct 31 13:07:57.007374 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:57.007344 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjnmm\" (UniqueName: \"kubernetes.io/projected/c2ca0d7c-7fb8-438d-9c27-09c1602339fa-kube-api-access-rjnmm\") pod \"payload-pre-processing-e2e-shared-b-w0-377f60-bdf4746c-r7hng\" (UID: \"c2ca0d7c-7fb8-438d-9c27-09c1602339fa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-377f60-bdf4746c-r7hng" Oct 31 13:07:57.007374 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:57.007364 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldhms\" (UniqueName: \"kubernetes.io/projected/03b3ba33-5c21-487d-bee4-abc978d8cf06-kube-api-access-ldhms\") pod \"payload-processing-e2e-shared-b-w0-377f60-c59666d89-b6xbr\" (UID: \"03b3ba33-5c21-487d-bee4-abc978d8cf06\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-377f60-c59666d89-b6xbr" Oct 31 13:07:57.061259 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:57.061229 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-377f60-c59666d89-b6xbr" Oct 31 13:07:57.303707 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:57.303570 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-377f60-bdf4746c-r7hng" Oct 31 13:07:57.413271 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:57.413246 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-377f60-c59666d89-b6xbr"] Oct 31 13:07:57.459150 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:57.459107 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-377f60-c59666d89-b6xbr" event={"ID":"03b3ba33-5c21-487d-bee4-abc978d8cf06","Type":"ContainerStarted","Data":"2e79eeec393f37db8469ae00b8c5846db44617f045a2dd0d68518b11dc39221e"} Oct 31 13:07:57.588689 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:57.588621 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-377f60-bdf4746c-r7hng"] Oct 31 13:07:57.589558 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:07:57.589529 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2ca0d7c_7fb8_438d_9c27_09c1602339fa.slice/crio-5eafc09a521875e0eb7b99f892e3846a4a272695d25f30a18b50538ab374c113 WatchSource:0}: Error finding container 5eafc09a521875e0eb7b99f892e3846a4a272695d25f30a18b50538ab374c113: Status 404 returned error can't find the container with id 5eafc09a521875e0eb7b99f892e3846a4a272695d25f30a18b50538ab374c113 Oct 31 13:07:58.091889 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:58.091852 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-6106a3-545f977779-44hnw"] Oct 31 13:07:58.096162 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:58.096139 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-6106a3-545f977779-44hnw" Oct 31 13:07:58.105147 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:58.105125 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-6106a3\"" Oct 31 13:07:58.108296 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:58.108269 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-6106a3-545f977779-44hnw"] Oct 31 13:07:58.114512 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:58.114484 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcqkn\" (UniqueName: \"kubernetes.io/projected/6665c331-4ab3-4ef2-8b08-498509dfe585-kube-api-access-zcqkn\") pod \"payload-processing-e2e-explicit-6106a3-545f977779-44hnw\" (UID: \"6665c331-4ab3-4ef2-8b08-498509dfe585\") " pod="openshift-ingress/payload-processing-e2e-explicit-6106a3-545f977779-44hnw" Oct 31 13:07:58.114737 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:58.114716 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6665c331-4ab3-4ef2-8b08-498509dfe585-plugins-config-volume\") pod \"payload-processing-e2e-explicit-6106a3-545f977779-44hnw\" (UID: \"6665c331-4ab3-4ef2-8b08-498509dfe585\") " pod="openshift-ingress/payload-processing-e2e-explicit-6106a3-545f977779-44hnw" Oct 31 13:07:58.215899 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:58.215856 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcqkn\" (UniqueName: \"kubernetes.io/projected/6665c331-4ab3-4ef2-8b08-498509dfe585-kube-api-access-zcqkn\") pod \"payload-processing-e2e-explicit-6106a3-545f977779-44hnw\" (UID: \"6665c331-4ab3-4ef2-8b08-498509dfe585\") " pod="openshift-ingress/payload-processing-e2e-explicit-6106a3-545f977779-44hnw" Oct 31 13:07:58.216079 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:58.215952 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6665c331-4ab3-4ef2-8b08-498509dfe585-plugins-config-volume\") pod \"payload-processing-e2e-explicit-6106a3-545f977779-44hnw\" (UID: \"6665c331-4ab3-4ef2-8b08-498509dfe585\") " pod="openshift-ingress/payload-processing-e2e-explicit-6106a3-545f977779-44hnw" Oct 31 13:07:58.216666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:58.216627 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6665c331-4ab3-4ef2-8b08-498509dfe585-plugins-config-volume\") pod \"payload-processing-e2e-explicit-6106a3-545f977779-44hnw\" (UID: \"6665c331-4ab3-4ef2-8b08-498509dfe585\") " pod="openshift-ingress/payload-processing-e2e-explicit-6106a3-545f977779-44hnw" Oct 31 13:07:58.233970 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:58.230930 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcqkn\" (UniqueName: \"kubernetes.io/projected/6665c331-4ab3-4ef2-8b08-498509dfe585-kube-api-access-zcqkn\") pod \"payload-processing-e2e-explicit-6106a3-545f977779-44hnw\" (UID: \"6665c331-4ab3-4ef2-8b08-498509dfe585\") " pod="openshift-ingress/payload-processing-e2e-explicit-6106a3-545f977779-44hnw" Oct 31 13:07:58.415790 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:58.412414 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-6106a3-545f977779-44hnw" Oct 31 13:07:58.470573 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:58.470538 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-377f60-bdf4746c-r7hng" event={"ID":"c2ca0d7c-7fb8-438d-9c27-09c1602339fa","Type":"ContainerStarted","Data":"03e53c48bb964e273baa0e73e65451c78201e1bb16ff6f30c355bb861cff99d1"} Oct 31 13:07:58.470748 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:58.470594 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-377f60-bdf4746c-r7hng" event={"ID":"c2ca0d7c-7fb8-438d-9c27-09c1602339fa","Type":"ContainerStarted","Data":"5eafc09a521875e0eb7b99f892e3846a4a272695d25f30a18b50538ab374c113"} Oct 31 13:07:58.471498 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:58.471478 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-377f60-bdf4746c-r7hng" Oct 31 13:07:58.473827 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:58.473798 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-377f60-c59666d89-b6xbr" event={"ID":"03b3ba33-5c21-487d-bee4-abc978d8cf06","Type":"ContainerStarted","Data":"5fe013ac611b437e3e7d5c90ba1c3cb96e945b7e27e66c10d0f697611e0fccd9"} Oct 31 13:07:58.474458 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:58.474441 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-377f60-c59666d89-b6xbr" Oct 31 13:07:58.493942 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:58.493878 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-377f60-bdf4746c-r7hng" podStartSLOduration=2.4938581490000002 podStartE2EDuration="2.493858149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:07:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:07:58.489919328 +0000 UTC m=+2129.197505637" watchObservedRunningTime="2026-10-31 13:07:58.493858149 +0000 UTC m=+2129.201444435" Oct 31 13:07:58.520124 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:58.516731 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-377f60-c59666d89-b6xbr" podStartSLOduration=2.516713375 podStartE2EDuration="2.516713375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:07:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:07:58.513899087 +0000 UTC m=+2129.221485374" watchObservedRunningTime="2026-10-31 13:07:58.516713375 +0000 UTC m=+2129.224299672" Oct 31 13:07:58.709880 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:58.709809 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c24e1-54b97985f9-zvhx8"] Oct 31 13:07:58.715195 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:58.715172 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c24e1-54b97985f9-zvhx8" Oct 31 13:07:58.721700 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:58.721676 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-5c24e1\"" Oct 31 13:07:58.723551 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:58.723527 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-5c24e1-dockercfg-dn8x4\"" Oct 31 13:07:58.726252 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:58.726229 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v5wj\" (UniqueName: \"kubernetes.io/projected/a0257070-74ef-4b97-acd3-ad2a93fe5342-kube-api-access-6v5wj\") pod \"payload-pre-processing-e2e-shared-b-w1-5c24e1-54b97985f9-zvhx8\" (UID: \"a0257070-74ef-4b97-acd3-ad2a93fe5342\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c24e1-54b97985f9-zvhx8" Oct 31 13:07:58.726424 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:58.726404 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0257070-74ef-4b97-acd3-ad2a93fe5342-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-5c24e1-54b97985f9-zvhx8\" (UID: \"a0257070-74ef-4b97-acd3-ad2a93fe5342\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c24e1-54b97985f9-zvhx8" Oct 31 13:07:58.727009 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:58.726987 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c24e1-54b97985f9-zvhx8"] Oct 31 13:07:58.740174 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:58.740146 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-6106a3-545f977779-44hnw"] Oct 31 13:07:58.740542 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:07:58.740518 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6665c331_4ab3_4ef2_8b08_498509dfe585.slice/crio-30d239d24c4b6edc9ac9166817c43fefb887e8311de9a0db5217841facaadb3c WatchSource:0}: Error finding container 30d239d24c4b6edc9ac9166817c43fefb887e8311de9a0db5217841facaadb3c: Status 404 returned error can't find the container with id 30d239d24c4b6edc9ac9166817c43fefb887e8311de9a0db5217841facaadb3c Oct 31 13:07:58.839033 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:58.838988 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0257070-74ef-4b97-acd3-ad2a93fe5342-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-5c24e1-54b97985f9-zvhx8\" (UID: \"a0257070-74ef-4b97-acd3-ad2a93fe5342\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c24e1-54b97985f9-zvhx8" Oct 31 13:07:58.839272 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:58.839253 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v5wj\" (UniqueName: \"kubernetes.io/projected/a0257070-74ef-4b97-acd3-ad2a93fe5342-kube-api-access-6v5wj\") pod \"payload-pre-processing-e2e-shared-b-w1-5c24e1-54b97985f9-zvhx8\" (UID: \"a0257070-74ef-4b97-acd3-ad2a93fe5342\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c24e1-54b97985f9-zvhx8" Oct 31 13:07:58.840835 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:58.840806 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0257070-74ef-4b97-acd3-ad2a93fe5342-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-5c24e1-54b97985f9-zvhx8\" (UID: \"a0257070-74ef-4b97-acd3-ad2a93fe5342\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c24e1-54b97985f9-zvhx8" Oct 31 13:07:58.909536 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:58.909496 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v5wj\" (UniqueName: \"kubernetes.io/projected/a0257070-74ef-4b97-acd3-ad2a93fe5342-kube-api-access-6v5wj\") pod \"payload-pre-processing-e2e-shared-b-w1-5c24e1-54b97985f9-zvhx8\" (UID: \"a0257070-74ef-4b97-acd3-ad2a93fe5342\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c24e1-54b97985f9-zvhx8" Oct 31 13:07:59.036985 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:59.036899 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c24e1-54b97985f9-zvhx8" Oct 31 13:07:59.331149 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:59.331129 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c24e1-54b97985f9-zvhx8"] Oct 31 13:07:59.334047 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:07:59.334016 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0257070_74ef_4b97_acd3_ad2a93fe5342.slice/crio-23c42fc43d308dd861e3012cb09185ab408fa34eaee0222d3d3f8924befbb78a WatchSource:0}: Error finding container 23c42fc43d308dd861e3012cb09185ab408fa34eaee0222d3d3f8924befbb78a: Status 404 returned error can't find the container with id 23c42fc43d308dd861e3012cb09185ab408fa34eaee0222d3d3f8924befbb78a Oct 31 13:07:59.501182 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:59.501140 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c24e1-54b97985f9-zvhx8" event={"ID":"a0257070-74ef-4b97-acd3-ad2a93fe5342","Type":"ContainerStarted","Data":"87db9712426c9e787e206a0af3a7433e4d691edb162369a0a5e477234d35335b"} Oct 31 13:07:59.501378 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:59.501209 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c24e1-54b97985f9-zvhx8" event={"ID":"a0257070-74ef-4b97-acd3-ad2a93fe5342","Type":"ContainerStarted","Data":"23c42fc43d308dd861e3012cb09185ab408fa34eaee0222d3d3f8924befbb78a"} Oct 31 13:07:59.502293 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:59.502268 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c24e1-54b97985f9-zvhx8" Oct 31 13:07:59.506604 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:59.506561 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-6106a3-545f977779-44hnw" event={"ID":"6665c331-4ab3-4ef2-8b08-498509dfe585","Type":"ContainerStarted","Data":"7d22362eb26f7991cf325e73f2d2ad3272cb5d2f4255bc560129dad26867166d"} Oct 31 13:07:59.506807 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:59.506663 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-6106a3-545f977779-44hnw" Oct 31 13:07:59.506807 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:59.506747 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-6106a3-545f977779-44hnw" event={"ID":"6665c331-4ab3-4ef2-8b08-498509dfe585","Type":"ContainerStarted","Data":"30d239d24c4b6edc9ac9166817c43fefb887e8311de9a0db5217841facaadb3c"} Oct 31 13:07:59.530394 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:59.530338 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c24e1-54b97985f9-zvhx8" podStartSLOduration=1.5303246160000001 podStartE2EDuration="1.530324616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:07:59.528439019 +0000 UTC m=+2130.236025342" watchObservedRunningTime="2026-10-31 13:07:59.530324616 +0000 UTC m=+2130.237910941" Oct 31 13:07:59.560209 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:07:59.560161 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-6106a3-545f977779-44hnw" podStartSLOduration=1.56014745 podStartE2EDuration="1.56014745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:07:59.558575048 +0000 UTC m=+2130.266161341" watchObservedRunningTime="2026-10-31 13:07:59.56014745 +0000 UTC m=+2130.267733745" Oct 31 13:08:03.540148 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:03.540109 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-569075-67d86d8547-blfvq"] Oct 31 13:08:03.545456 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:03.545438 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-569075-67d86d8547-blfvq" Oct 31 13:08:03.553075 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:03.553052 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-569075\"" Oct 31 13:08:03.554044 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:03.554023 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-569075-dockercfg-2tnj5\"" Oct 31 13:08:03.554215 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:03.554026 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-569075-6b667f965f-cfjvr"] Oct 31 13:08:03.559072 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:03.559051 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-569075-67d86d8547-blfvq"] Oct 31 13:08:03.559167 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:03.559154 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-569075-6b667f965f-cfjvr" Oct 31 13:08:03.660666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:03.654658 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29da7f75-0ee1-4981-8ee3-b23f46a67876-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-569075-67d86d8547-blfvq\" (UID: \"29da7f75-0ee1-4981-8ee3-b23f46a67876\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-569075-67d86d8547-blfvq" Oct 31 13:08:03.660666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:03.654903 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dc3c2a0-104c-4a6e-a708-9b212dc74267-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-569075-6b667f965f-cfjvr\" (UID: \"3dc3c2a0-104c-4a6e-a708-9b212dc74267\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-569075-6b667f965f-cfjvr" Oct 31 13:08:03.660666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:03.655049 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cl24\" (UniqueName: \"kubernetes.io/projected/29da7f75-0ee1-4981-8ee3-b23f46a67876-kube-api-access-7cl24\") pod \"payload-pre-processing-e2e-shared-b-w2-569075-67d86d8547-blfvq\" (UID: \"29da7f75-0ee1-4981-8ee3-b23f46a67876\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-569075-67d86d8547-blfvq" Oct 31 13:08:03.660666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:03.655088 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bnvw\" (UniqueName: \"kubernetes.io/projected/3dc3c2a0-104c-4a6e-a708-9b212dc74267-kube-api-access-4bnvw\") pod \"payload-processing-e2e-shared-b-w2-569075-6b667f965f-cfjvr\" (UID: \"3dc3c2a0-104c-4a6e-a708-9b212dc74267\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-569075-6b667f965f-cfjvr" Oct 31 13:08:03.763704 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:03.763660 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cl24\" (UniqueName: \"kubernetes.io/projected/29da7f75-0ee1-4981-8ee3-b23f46a67876-kube-api-access-7cl24\") pod \"payload-pre-processing-e2e-shared-b-w2-569075-67d86d8547-blfvq\" (UID: \"29da7f75-0ee1-4981-8ee3-b23f46a67876\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-569075-67d86d8547-blfvq" Oct 31 13:08:03.763982 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:03.763959 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bnvw\" (UniqueName: \"kubernetes.io/projected/3dc3c2a0-104c-4a6e-a708-9b212dc74267-kube-api-access-4bnvw\") pod \"payload-processing-e2e-shared-b-w2-569075-6b667f965f-cfjvr\" (UID: \"3dc3c2a0-104c-4a6e-a708-9b212dc74267\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-569075-6b667f965f-cfjvr" Oct 31 13:08:03.775027 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:03.764166 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29da7f75-0ee1-4981-8ee3-b23f46a67876-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-569075-67d86d8547-blfvq\" (UID: \"29da7f75-0ee1-4981-8ee3-b23f46a67876\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-569075-67d86d8547-blfvq" Oct 31 13:08:03.776871 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:03.776839 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29da7f75-0ee1-4981-8ee3-b23f46a67876-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-569075-67d86d8547-blfvq\" (UID: \"29da7f75-0ee1-4981-8ee3-b23f46a67876\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-569075-67d86d8547-blfvq" Oct 31 13:08:03.792832 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:03.781151 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dc3c2a0-104c-4a6e-a708-9b212dc74267-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-569075-6b667f965f-cfjvr\" (UID: \"3dc3c2a0-104c-4a6e-a708-9b212dc74267\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-569075-6b667f965f-cfjvr" Oct 31 13:08:03.793180 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:03.781951 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dc3c2a0-104c-4a6e-a708-9b212dc74267-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-569075-6b667f965f-cfjvr\" (UID: \"3dc3c2a0-104c-4a6e-a708-9b212dc74267\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-569075-6b667f965f-cfjvr" Oct 31 13:08:03.809596 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:03.809549 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bnvw\" (UniqueName: \"kubernetes.io/projected/3dc3c2a0-104c-4a6e-a708-9b212dc74267-kube-api-access-4bnvw\") pod \"payload-processing-e2e-shared-b-w2-569075-6b667f965f-cfjvr\" (UID: \"3dc3c2a0-104c-4a6e-a708-9b212dc74267\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-569075-6b667f965f-cfjvr" Oct 31 13:08:03.812252 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:03.812227 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-569075-6b667f965f-cfjvr"] Oct 31 13:08:03.812790 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:03.812752 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cl24\" (UniqueName: \"kubernetes.io/projected/29da7f75-0ee1-4981-8ee3-b23f46a67876-kube-api-access-7cl24\") pod \"payload-pre-processing-e2e-shared-b-w2-569075-67d86d8547-blfvq\" (UID: \"29da7f75-0ee1-4981-8ee3-b23f46a67876\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-569075-67d86d8547-blfvq" Oct 31 13:08:03.862864 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:03.862834 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-569075-67d86d8547-blfvq" Oct 31 13:08:03.876351 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:03.876319 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-569075-6b667f965f-cfjvr" Oct 31 13:08:04.219879 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:04.216046 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-569075-67d86d8547-blfvq"] Oct 31 13:08:04.233688 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:04.229760 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-569075-6b667f965f-cfjvr"] Oct 31 13:08:04.244559 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:08:04.244523 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dc3c2a0_104c_4a6e_a708_9b212dc74267.slice/crio-5199e4ebcee2531c01a080295d592658f418b0df930e90de45981f2ea83cc8fc WatchSource:0}: Error finding container 5199e4ebcee2531c01a080295d592658f418b0df930e90de45981f2ea83cc8fc: Status 404 returned error can't find the container with id 5199e4ebcee2531c01a080295d592658f418b0df930e90de45981f2ea83cc8fc Oct 31 13:08:04.609059 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:04.608942 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-569075-67d86d8547-blfvq" event={"ID":"29da7f75-0ee1-4981-8ee3-b23f46a67876","Type":"ContainerStarted","Data":"32beede0c6559c47a12501f657ffc1c0f0d8d23ad08599d8018827e8a7332a0a"} Oct 31 13:08:04.609059 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:04.609020 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-569075-67d86d8547-blfvq" event={"ID":"29da7f75-0ee1-4981-8ee3-b23f46a67876","Type":"ContainerStarted","Data":"15f659dee4ad6512617499d6f189cbd455fa72a9a381eff5306663fb80a632e8"} Oct 31 13:08:04.609059 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:04.609052 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-569075-67d86d8547-blfvq" Oct 31 13:08:04.610773 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:04.610744 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-569075-6b667f965f-cfjvr" event={"ID":"3dc3c2a0-104c-4a6e-a708-9b212dc74267","Type":"ContainerStarted","Data":"2b3cd993a5642f729e432c3b420830b5cc94ea6d3c4a96da5a893fb04bf79b50"} Oct 31 13:08:04.610905 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:04.610796 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-569075-6b667f965f-cfjvr" event={"ID":"3dc3c2a0-104c-4a6e-a708-9b212dc74267","Type":"ContainerStarted","Data":"5199e4ebcee2531c01a080295d592658f418b0df930e90de45981f2ea83cc8fc"} Oct 31 13:08:04.610905 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:04.610872 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-569075-6b667f965f-cfjvr" Oct 31 13:08:04.657124 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:04.657075 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-569075-67d86d8547-blfvq" podStartSLOduration=1.6570609840000001 podStartE2EDuration="1.657060984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:08:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:08:04.655137657 +0000 UTC m=+2135.362723949" watchObservedRunningTime="2026-10-31 13:08:04.657060984 +0000 UTC m=+2135.364647314" Oct 31 13:08:04.696848 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:04.696795 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-569075-6b667f965f-cfjvr" podStartSLOduration=1.6967819419999999 podStartE2EDuration="1.696781942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:08:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:08:04.694196533 +0000 UTC m=+2135.401782841" watchObservedRunningTime="2026-10-31 13:08:04.696781942 +0000 UTC m=+2135.404368237" Oct 31 13:08:06.283570 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:06.283536 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74"] Oct 31 13:08:06.289109 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:06.289084 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74" Oct 31 13:08:06.300865 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:06.300844 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-6106a3-serving-cert\"" Oct 31 13:08:06.309013 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:06.308984 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74"] Oct 31 13:08:06.445756 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:06.445714 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00951dea-adc0-4e22-a003-221c5c19bd80-openshift-service-ca\") pod \"maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74\" (UID: \"00951dea-adc0-4e22-a003-221c5c19bd80\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74" Oct 31 13:08:06.451761 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:06.446395 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00951dea-adc0-4e22-a003-221c5c19bd80-metrics-certs\") pod \"maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74\" (UID: \"00951dea-adc0-4e22-a003-221c5c19bd80\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74" Oct 31 13:08:06.451761 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:06.446482 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00951dea-adc0-4e22-a003-221c5c19bd80-maas-api-tls\") pod \"maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74\" (UID: \"00951dea-adc0-4e22-a003-221c5c19bd80\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74" Oct 31 13:08:06.451761 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:06.446630 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fvql\" (UniqueName: \"kubernetes.io/projected/00951dea-adc0-4e22-a003-221c5c19bd80-kube-api-access-5fvql\") pod \"maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74\" (UID: \"00951dea-adc0-4e22-a003-221c5c19bd80\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74" Oct 31 13:08:06.548085 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:06.548001 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fvql\" (UniqueName: \"kubernetes.io/projected/00951dea-adc0-4e22-a003-221c5c19bd80-kube-api-access-5fvql\") pod \"maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74\" (UID: \"00951dea-adc0-4e22-a003-221c5c19bd80\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74" Oct 31 13:08:06.548287 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:06.548116 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00951dea-adc0-4e22-a003-221c5c19bd80-openshift-service-ca\") pod \"maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74\" (UID: \"00951dea-adc0-4e22-a003-221c5c19bd80\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74" Oct 31 13:08:06.548287 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:06.548166 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00951dea-adc0-4e22-a003-221c5c19bd80-metrics-certs\") pod \"maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74\" (UID: \"00951dea-adc0-4e22-a003-221c5c19bd80\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74" Oct 31 13:08:06.548287 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:06.548214 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00951dea-adc0-4e22-a003-221c5c19bd80-maas-api-tls\") pod \"maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74\" (UID: \"00951dea-adc0-4e22-a003-221c5c19bd80\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74" Oct 31 13:08:06.548858 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:06.548834 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00951dea-adc0-4e22-a003-221c5c19bd80-openshift-service-ca\") pod \"maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74\" (UID: \"00951dea-adc0-4e22-a003-221c5c19bd80\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74" Oct 31 13:08:06.550912 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:06.550877 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00951dea-adc0-4e22-a003-221c5c19bd80-metrics-certs\") pod \"maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74\" (UID: \"00951dea-adc0-4e22-a003-221c5c19bd80\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74" Oct 31 13:08:06.550999 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:06.550886 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00951dea-adc0-4e22-a003-221c5c19bd80-maas-api-tls\") pod \"maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74\" (UID: \"00951dea-adc0-4e22-a003-221c5c19bd80\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74" Oct 31 13:08:06.560609 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:06.560584 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fvql\" (UniqueName: \"kubernetes.io/projected/00951dea-adc0-4e22-a003-221c5c19bd80-kube-api-access-5fvql\") pod \"maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74\" (UID: \"00951dea-adc0-4e22-a003-221c5c19bd80\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74" Oct 31 13:08:06.605456 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:06.605424 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74" Oct 31 13:08:06.906066 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:06.906038 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74"] Oct 31 13:08:06.908846 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:08:06.908810 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00951dea_adc0_4e22_a003_221c5c19bd80.slice/crio-35a78cecd2422edfc1e653a8758c7507fb53ec5d3d7e5bbce2ef98c2e40b38af WatchSource:0}: Error finding container 35a78cecd2422edfc1e653a8758c7507fb53ec5d3d7e5bbce2ef98c2e40b38af: Status 404 returned error can't find the container with id 35a78cecd2422edfc1e653a8758c7507fb53ec5d3d7e5bbce2ef98c2e40b38af Oct 31 13:08:07.639873 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:07.639765 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74" event={"ID":"00951dea-adc0-4e22-a003-221c5c19bd80","Type":"ContainerStarted","Data":"93d8c0782b3c942b07e7d93be2adef4e5887c40a6f244fa4be3a5f9a88dc3a5a"} Oct 31 13:08:07.639873 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:07.639843 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74" Oct 31 13:08:07.639873 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:07.639871 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74" event={"ID":"00951dea-adc0-4e22-a003-221c5c19bd80","Type":"ContainerStarted","Data":"35a78cecd2422edfc1e653a8758c7507fb53ec5d3d7e5bbce2ef98c2e40b38af"} Oct 31 13:08:07.668092 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:07.668045 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74" podStartSLOduration=1.18406639 podStartE2EDuration="1.668032128s" totalImagesPullingTime="483.965738ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:08:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:08:07.665303503 +0000 UTC m=+2138.372889798" watchObservedRunningTime="2026-10-31 13:08:07.668032128 +0000 UTC m=+2138.375618422" Oct 31 13:08:08.627014 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:08.626978 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5452fc-768b7c485f-z66t4"] Oct 31 13:08:08.632020 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:08.631991 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5452fc-768b7c485f-z66t4" Oct 31 13:08:08.644042 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:08.644009 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5452fc-768b7c485f-z66t4"] Oct 31 13:08:08.722850 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:08.713278 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzx6l\" (UniqueName: \"kubernetes.io/projected/55afb55f-1e19-420a-a30e-7c14e5082e97-kube-api-access-tzx6l\") pod \"payload-pre-processing-e2e-shared-b-w3-5452fc-768b7c485f-z66t4\" (UID: \"55afb55f-1e19-420a-a30e-7c14e5082e97\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5452fc-768b7c485f-z66t4" Oct 31 13:08:08.722850 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:08.713432 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55afb55f-1e19-420a-a30e-7c14e5082e97-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-5452fc-768b7c485f-z66t4\" (UID: \"55afb55f-1e19-420a-a30e-7c14e5082e97\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5452fc-768b7c485f-z66t4" Oct 31 13:08:08.722850 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:08.721916 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-5452fc\"" Oct 31 13:08:08.722850 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:08.722069 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-5452fc-dockercfg-c8hvg\"" Oct 31 13:08:08.875629 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:08.864282 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzx6l\" (UniqueName: \"kubernetes.io/projected/55afb55f-1e19-420a-a30e-7c14e5082e97-kube-api-access-tzx6l\") pod \"payload-pre-processing-e2e-shared-b-w3-5452fc-768b7c485f-z66t4\" (UID: \"55afb55f-1e19-420a-a30e-7c14e5082e97\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5452fc-768b7c485f-z66t4" Oct 31 13:08:08.875629 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:08.864433 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55afb55f-1e19-420a-a30e-7c14e5082e97-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-5452fc-768b7c485f-z66t4\" (UID: \"55afb55f-1e19-420a-a30e-7c14e5082e97\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5452fc-768b7c485f-z66t4" Oct 31 13:08:08.875629 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:08.865340 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55afb55f-1e19-420a-a30e-7c14e5082e97-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-5452fc-768b7c485f-z66t4\" (UID: \"55afb55f-1e19-420a-a30e-7c14e5082e97\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5452fc-768b7c485f-z66t4" Oct 31 13:08:08.909706 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:08.909611 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzx6l\" (UniqueName: \"kubernetes.io/projected/55afb55f-1e19-420a-a30e-7c14e5082e97-kube-api-access-tzx6l\") pod \"payload-pre-processing-e2e-shared-b-w3-5452fc-768b7c485f-z66t4\" (UID: \"55afb55f-1e19-420a-a30e-7c14e5082e97\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5452fc-768b7c485f-z66t4" Oct 31 13:08:08.965289 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:08.965253 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5452fc-768b7c485f-z66t4" Oct 31 13:08:09.298773 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:09.292595 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5452fc-768b7c485f-z66t4"] Oct 31 13:08:09.847723 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:09.846723 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5452fc-768b7c485f-z66t4" event={"ID":"55afb55f-1e19-420a-a30e-7c14e5082e97","Type":"ContainerStarted","Data":"369384b6952962f1f80d03073eef7bbdd7bb20c205bbfc115738ee2cff148b74"} Oct 31 13:08:10.518219 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:10.518036 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-377f60-bdf4746c-r7hng" Oct 31 13:08:10.521274 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:10.521244 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-377f60-c59666d89-b6xbr" Oct 31 13:08:10.522521 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:10.521843 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-6106a3-545f977779-44hnw" Oct 31 13:08:10.855004 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:10.854904 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5452fc-768b7c485f-z66t4" event={"ID":"55afb55f-1e19-420a-a30e-7c14e5082e97","Type":"ContainerStarted","Data":"3c487137982fc2a6d565cfcbf2d149751f9089137b1407903adc5f114104d02e"} Oct 31 13:08:10.855450 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:10.855063 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5452fc-768b7c485f-z66t4" Oct 31 13:08:10.890258 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:10.890207 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5452fc-768b7c485f-z66t4" podStartSLOduration=2.890192848 podStartE2EDuration="2.890192848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:08:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:08:10.888939974 +0000 UTC m=+2141.596526266" watchObservedRunningTime="2026-10-31 13:08:10.890192848 +0000 UTC m=+2141.597779143" Oct 31 13:08:11.520097 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:11.520070 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c24e1-54b97985f9-zvhx8" Oct 31 13:08:13.675969 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:13.675934 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74" Oct 31 13:08:14.359462 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:14.359426 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7"] Oct 31 13:08:14.388662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:14.388612 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7"] Oct 31 13:08:14.388926 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:14.388781 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7" Oct 31 13:08:14.395289 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:14.395266 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-569075-serving-cert\"" Oct 31 13:08:14.493502 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:14.493462 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/421c561e-1dcd-46b7-85bf-1416e1e007cd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7\" (UID: \"421c561e-1dcd-46b7-85bf-1416e1e007cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7" Oct 31 13:08:14.493875 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:14.493850 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/421c561e-1dcd-46b7-85bf-1416e1e007cd-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7\" (UID: \"421c561e-1dcd-46b7-85bf-1416e1e007cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7" Oct 31 13:08:14.494484 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:14.494442 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z9lk\" (UniqueName: \"kubernetes.io/projected/421c561e-1dcd-46b7-85bf-1416e1e007cd-kube-api-access-4z9lk\") pod \"maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7\" (UID: \"421c561e-1dcd-46b7-85bf-1416e1e007cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7" Oct 31 13:08:14.494812 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:14.494793 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/421c561e-1dcd-46b7-85bf-1416e1e007cd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7\" (UID: \"421c561e-1dcd-46b7-85bf-1416e1e007cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7" Oct 31 13:08:14.597983 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:14.597445 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z9lk\" (UniqueName: \"kubernetes.io/projected/421c561e-1dcd-46b7-85bf-1416e1e007cd-kube-api-access-4z9lk\") pod \"maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7\" (UID: \"421c561e-1dcd-46b7-85bf-1416e1e007cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7" Oct 31 13:08:14.597983 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:14.597563 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/421c561e-1dcd-46b7-85bf-1416e1e007cd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7\" (UID: \"421c561e-1dcd-46b7-85bf-1416e1e007cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7" Oct 31 13:08:14.597983 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:14.597834 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/421c561e-1dcd-46b7-85bf-1416e1e007cd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7\" (UID: \"421c561e-1dcd-46b7-85bf-1416e1e007cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7" Oct 31 13:08:14.597983 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:14.597870 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/421c561e-1dcd-46b7-85bf-1416e1e007cd-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7\" (UID: \"421c561e-1dcd-46b7-85bf-1416e1e007cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7" Oct 31 13:08:14.599543 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:14.599518 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/421c561e-1dcd-46b7-85bf-1416e1e007cd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7\" (UID: \"421c561e-1dcd-46b7-85bf-1416e1e007cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7" Oct 31 13:08:14.601592 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:14.601564 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/421c561e-1dcd-46b7-85bf-1416e1e007cd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7\" (UID: \"421c561e-1dcd-46b7-85bf-1416e1e007cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7" Oct 31 13:08:14.601751 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:14.601726 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/421c561e-1dcd-46b7-85bf-1416e1e007cd-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7\" (UID: \"421c561e-1dcd-46b7-85bf-1416e1e007cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7" Oct 31 13:08:14.612770 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:14.612587 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z9lk\" (UniqueName: \"kubernetes.io/projected/421c561e-1dcd-46b7-85bf-1416e1e007cd-kube-api-access-4z9lk\") pod \"maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7\" (UID: \"421c561e-1dcd-46b7-85bf-1416e1e007cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7" Oct 31 13:08:14.711886 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:14.711853 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7" Oct 31 13:08:15.023717 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:15.023680 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7"] Oct 31 13:08:15.034612 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:08:15.033990 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod421c561e_1dcd_46b7_85bf_1416e1e007cd.slice/crio-d876c47b7f936f11616b91270a3a621dbf8f6d5d011c7a0b6aaf49748537ca00 WatchSource:0}: Error finding container d876c47b7f936f11616b91270a3a621dbf8f6d5d011c7a0b6aaf49748537ca00: Status 404 returned error can't find the container with id d876c47b7f936f11616b91270a3a621dbf8f6d5d011c7a0b6aaf49748537ca00 Oct 31 13:08:15.620827 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:15.620791 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-569075-6b667f965f-cfjvr" Oct 31 13:08:15.620977 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:15.620861 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-569075-67d86d8547-blfvq" Oct 31 13:08:15.898103 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:15.898067 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7" event={"ID":"421c561e-1dcd-46b7-85bf-1416e1e007cd","Type":"ContainerStarted","Data":"5025613ea840a66fc294cd9ecd26b5a042ebfe8aa1425cc8663af19da43dc910"} Oct 31 13:08:15.898776 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:15.898124 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7" event={"ID":"421c561e-1dcd-46b7-85bf-1416e1e007cd","Type":"ContainerStarted","Data":"d876c47b7f936f11616b91270a3a621dbf8f6d5d011c7a0b6aaf49748537ca00"} Oct 31 13:08:15.898776 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:15.898154 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7" Oct 31 13:08:15.931690 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:15.931601 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7" podStartSLOduration=1.488670185 podStartE2EDuration="1.931581668s" totalImagesPullingTime="442.911483ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:08:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:08:15.931210687 +0000 UTC m=+2146.638797009" watchObservedRunningTime="2026-10-31 13:08:15.931581668 +0000 UTC m=+2146.639167966" Oct 31 13:08:18.745074 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:08:18.745042 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:08:21.929882 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:21.929851 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5452fc-768b7c485f-z66t4" Oct 31 13:08:21.948533 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:21.948504 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7" Oct 31 13:08:39.449875 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:39.449784 5693 patch_prober.go:29] interesting pod/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:08:39.450256 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:39.449875 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" podUID="76570ffc-6a6b-4f4d-b8f0-0db5cde967c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:08:41.529500 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:41.529462 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-6cef79/auth-test-model-6cef79-kserve-7645c89c97-8kjxg"] Oct 31 13:08:41.533853 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:41.533836 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-6cef79/auth-test-model-6cef79-kserve-7645c89c97-8kjxg" Oct 31 13:08:41.543297 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:41.542990 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-6cef79/auth-test-model-6cef79-kserve-7645c89c97-8kjxg"] Oct 31 13:08:41.543297 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:41.543221 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-6cef79\"/\"auth-test-model-6cef79-kserve-self-signed-certs\"" Oct 31 13:08:41.545169 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:41.545143 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-6cef79\"/\"kube-root-ca.crt\"" Oct 31 13:08:41.545286 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:41.545231 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-6cef79\"/\"default-dockercfg-wg2c4\"" Oct 31 13:08:41.545484 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:41.545435 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-6cef79\"/\"openshift-service-ca.crt\"" Oct 31 13:08:41.647673 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:41.647626 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8997c531-623c-4b8a-b446-79db29e64bbb-tls-certs\") pod \"auth-test-model-6cef79-kserve-7645c89c97-8kjxg\" (UID: \"8997c531-623c-4b8a-b446-79db29e64bbb\") " pod="ai-tenant-e2e-shared-a-w1-6cef79/auth-test-model-6cef79-kserve-7645c89c97-8kjxg" Oct 31 13:08:41.647792 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:41.647731 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8997c531-623c-4b8a-b446-79db29e64bbb-dshm\") pod \"auth-test-model-6cef79-kserve-7645c89c97-8kjxg\" (UID: \"8997c531-623c-4b8a-b446-79db29e64bbb\") " pod="ai-tenant-e2e-shared-a-w1-6cef79/auth-test-model-6cef79-kserve-7645c89c97-8kjxg" Oct 31 13:08:41.647895 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:41.647879 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz8kx\" (UniqueName: \"kubernetes.io/projected/8997c531-623c-4b8a-b446-79db29e64bbb-kube-api-access-nz8kx\") pod \"auth-test-model-6cef79-kserve-7645c89c97-8kjxg\" (UID: \"8997c531-623c-4b8a-b446-79db29e64bbb\") " pod="ai-tenant-e2e-shared-a-w1-6cef79/auth-test-model-6cef79-kserve-7645c89c97-8kjxg" Oct 31 13:08:41.647936 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:41.647905 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8997c531-623c-4b8a-b446-79db29e64bbb-model-cache\") pod \"auth-test-model-6cef79-kserve-7645c89c97-8kjxg\" (UID: \"8997c531-623c-4b8a-b446-79db29e64bbb\") " pod="ai-tenant-e2e-shared-a-w1-6cef79/auth-test-model-6cef79-kserve-7645c89c97-8kjxg" Oct 31 13:08:41.648059 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:41.648045 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8997c531-623c-4b8a-b446-79db29e64bbb-home\") pod \"auth-test-model-6cef79-kserve-7645c89c97-8kjxg\" (UID: \"8997c531-623c-4b8a-b446-79db29e64bbb\") " pod="ai-tenant-e2e-shared-a-w1-6cef79/auth-test-model-6cef79-kserve-7645c89c97-8kjxg" Oct 31 13:08:41.648100 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:41.648090 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8997c531-623c-4b8a-b446-79db29e64bbb-tmp-dir\") pod \"auth-test-model-6cef79-kserve-7645c89c97-8kjxg\" (UID: \"8997c531-623c-4b8a-b446-79db29e64bbb\") " pod="ai-tenant-e2e-shared-a-w1-6cef79/auth-test-model-6cef79-kserve-7645c89c97-8kjxg" Oct 31 13:08:41.750663 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:41.749749 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8997c531-623c-4b8a-b446-79db29e64bbb-dshm\") pod \"auth-test-model-6cef79-kserve-7645c89c97-8kjxg\" (UID: \"8997c531-623c-4b8a-b446-79db29e64bbb\") " pod="ai-tenant-e2e-shared-a-w1-6cef79/auth-test-model-6cef79-kserve-7645c89c97-8kjxg" Oct 31 13:08:41.750663 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:41.749892 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz8kx\" (UniqueName: \"kubernetes.io/projected/8997c531-623c-4b8a-b446-79db29e64bbb-kube-api-access-nz8kx\") pod \"auth-test-model-6cef79-kserve-7645c89c97-8kjxg\" (UID: \"8997c531-623c-4b8a-b446-79db29e64bbb\") " pod="ai-tenant-e2e-shared-a-w1-6cef79/auth-test-model-6cef79-kserve-7645c89c97-8kjxg" Oct 31 13:08:41.750663 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:41.749924 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8997c531-623c-4b8a-b446-79db29e64bbb-model-cache\") pod \"auth-test-model-6cef79-kserve-7645c89c97-8kjxg\" (UID: \"8997c531-623c-4b8a-b446-79db29e64bbb\") " pod="ai-tenant-e2e-shared-a-w1-6cef79/auth-test-model-6cef79-kserve-7645c89c97-8kjxg" Oct 31 13:08:41.750663 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:41.750061 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8997c531-623c-4b8a-b446-79db29e64bbb-home\") pod \"auth-test-model-6cef79-kserve-7645c89c97-8kjxg\" (UID: \"8997c531-623c-4b8a-b446-79db29e64bbb\") " pod="ai-tenant-e2e-shared-a-w1-6cef79/auth-test-model-6cef79-kserve-7645c89c97-8kjxg" Oct 31 13:08:41.750663 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:41.750119 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8997c531-623c-4b8a-b446-79db29e64bbb-tmp-dir\") pod \"auth-test-model-6cef79-kserve-7645c89c97-8kjxg\" (UID: \"8997c531-623c-4b8a-b446-79db29e64bbb\") " pod="ai-tenant-e2e-shared-a-w1-6cef79/auth-test-model-6cef79-kserve-7645c89c97-8kjxg" Oct 31 13:08:41.750663 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:41.750192 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8997c531-623c-4b8a-b446-79db29e64bbb-tls-certs\") pod \"auth-test-model-6cef79-kserve-7645c89c97-8kjxg\" (UID: \"8997c531-623c-4b8a-b446-79db29e64bbb\") " pod="ai-tenant-e2e-shared-a-w1-6cef79/auth-test-model-6cef79-kserve-7645c89c97-8kjxg" Oct 31 13:08:41.751147 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:41.751030 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8997c531-623c-4b8a-b446-79db29e64bbb-model-cache\") pod \"auth-test-model-6cef79-kserve-7645c89c97-8kjxg\" (UID: \"8997c531-623c-4b8a-b446-79db29e64bbb\") " pod="ai-tenant-e2e-shared-a-w1-6cef79/auth-test-model-6cef79-kserve-7645c89c97-8kjxg" Oct 31 13:08:41.754663 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:41.751679 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8997c531-623c-4b8a-b446-79db29e64bbb-home\") pod \"auth-test-model-6cef79-kserve-7645c89c97-8kjxg\" (UID: \"8997c531-623c-4b8a-b446-79db29e64bbb\") " pod="ai-tenant-e2e-shared-a-w1-6cef79/auth-test-model-6cef79-kserve-7645c89c97-8kjxg" Oct 31 13:08:41.754663 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:41.752207 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8997c531-623c-4b8a-b446-79db29e64bbb-tmp-dir\") pod \"auth-test-model-6cef79-kserve-7645c89c97-8kjxg\" (UID: \"8997c531-623c-4b8a-b446-79db29e64bbb\") " pod="ai-tenant-e2e-shared-a-w1-6cef79/auth-test-model-6cef79-kserve-7645c89c97-8kjxg" Oct 31 13:08:41.758704 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:41.758287 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8997c531-623c-4b8a-b446-79db29e64bbb-dshm\") pod \"auth-test-model-6cef79-kserve-7645c89c97-8kjxg\" (UID: \"8997c531-623c-4b8a-b446-79db29e64bbb\") " pod="ai-tenant-e2e-shared-a-w1-6cef79/auth-test-model-6cef79-kserve-7645c89c97-8kjxg" Oct 31 13:08:41.764988 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:41.762865 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8997c531-623c-4b8a-b446-79db29e64bbb-tls-certs\") pod \"auth-test-model-6cef79-kserve-7645c89c97-8kjxg\" (UID: \"8997c531-623c-4b8a-b446-79db29e64bbb\") " pod="ai-tenant-e2e-shared-a-w1-6cef79/auth-test-model-6cef79-kserve-7645c89c97-8kjxg" Oct 31 13:08:41.778778 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:41.775978 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz8kx\" (UniqueName: \"kubernetes.io/projected/8997c531-623c-4b8a-b446-79db29e64bbb-kube-api-access-nz8kx\") pod \"auth-test-model-6cef79-kserve-7645c89c97-8kjxg\" (UID: \"8997c531-623c-4b8a-b446-79db29e64bbb\") " pod="ai-tenant-e2e-shared-a-w1-6cef79/auth-test-model-6cef79-kserve-7645c89c97-8kjxg" Oct 31 13:08:41.850897 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:41.850801 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-6cef79/auth-test-model-6cef79-kserve-7645c89c97-8kjxg" Oct 31 13:08:42.149275 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:42.149248 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-6cef79/auth-test-model-6cef79-kserve-7645c89c97-8kjxg"] Oct 31 13:08:42.150623 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:08:42.150594 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8997c531_623c_4b8a_b446_79db29e64bbb.slice/crio-9b84a41780842db83d804217f2196f977e46ccf3a8dd9e4eac6881d908ea723d WatchSource:0}: Error finding container 9b84a41780842db83d804217f2196f977e46ccf3a8dd9e4eac6881d908ea723d: Status 404 returned error can't find the container with id 9b84a41780842db83d804217f2196f977e46ccf3a8dd9e4eac6881d908ea723d Oct 31 13:08:42.271118 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:42.270963 5693 generic.go:356] "Generic (PLEG): container finished" podID="76570ffc-6a6b-4f4d-b8f0-0db5cde967c6" containerID="f6f09ceee1023a3138095e8e21c5e30b90f34d0b64a389504ff470751a3e7e1c" exitCode=0 Oct 31 13:08:42.271118 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:42.271033 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" event={"ID":"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6","Type":"ContainerDied","Data":"f6f09ceee1023a3138095e8e21c5e30b90f34d0b64a389504ff470751a3e7e1c"} Oct 31 13:08:42.271494 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:42.271471 5693 scope.go:122] "RemoveContainer" containerID="f6f09ceee1023a3138095e8e21c5e30b90f34d0b64a389504ff470751a3e7e1c" Oct 31 13:08:42.273074 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:42.273047 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-6cef79/auth-test-model-6cef79-kserve-7645c89c97-8kjxg" event={"ID":"8997c531-623c-4b8a-b446-79db29e64bbb","Type":"ContainerStarted","Data":"d8ae1cf1a256a0aa4cfd3a65bdf521ddc8794dff1cbd5f70e81a876bcdb34595"} Oct 31 13:08:42.273185 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:42.273079 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-6cef79/auth-test-model-6cef79-kserve-7645c89c97-8kjxg" event={"ID":"8997c531-623c-4b8a-b446-79db29e64bbb","Type":"ContainerStarted","Data":"9b84a41780842db83d804217f2196f977e46ccf3a8dd9e4eac6881d908ea723d"} Oct 31 13:08:42.273185 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:42.273122 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:08:42.273302 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:42.273208 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:08:42.273302 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:42.273258 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:08:42.321736 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:42.321675 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-6cef79/auth-test-model-6cef79-kserve-7645c89c97-8kjxg" podStartSLOduration=1.321629897 podStartE2EDuration="1.321629897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:08:42.320204514 +0000 UTC m=+2173.027790810" watchObservedRunningTime="2026-10-31 13:08:42.321629897 +0000 UTC m=+2173.029216194" Oct 31 13:08:42.851626 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:42.851594 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-6cef79/auth-test-model-6cef79-kserve-7645c89c97-8kjxg" Oct 31 13:08:43.308465 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:43.308425 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" event={"ID":"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6","Type":"ContainerStarted","Data":"0e95c2e859ab2c48f3918e71e9bd286dba34af97ef1aa9c93f7738fff7d99044"} Oct 31 13:08:43.343162 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:43.343131 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" Oct 31 13:08:43.343330 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:43.343270 5693 patch_prober.go:29] interesting pod/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 31 13:08:43.343330 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:43.343312 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" podUID="76570ffc-6a6b-4f4d-b8f0-0db5cde967c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 31 13:08:44.343896 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:44.343863 5693 patch_prober.go:29] interesting pod/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 31 13:08:44.344307 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:44.343916 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" podUID="76570ffc-6a6b-4f4d-b8f0-0db5cde967c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 31 13:08:45.343616 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:45.343586 5693 patch_prober.go:29] interesting pod/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 31 13:08:45.343828 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:45.343657 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" podUID="76570ffc-6a6b-4f4d-b8f0-0db5cde967c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 31 13:08:46.343725 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:46.343687 5693 patch_prober.go:29] interesting pod/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 31 13:08:46.344147 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:46.343772 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" podUID="76570ffc-6a6b-4f4d-b8f0-0db5cde967c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 31 13:08:47.345097 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:47.345061 5693 patch_prober.go:29] interesting pod/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 31 13:08:47.345600 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:47.345135 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" podUID="76570ffc-6a6b-4f4d-b8f0-0db5cde967c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 31 13:08:48.344203 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:48.344166 5693 patch_prober.go:29] interesting pod/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 31 13:08:48.344400 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:48.344230 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" podUID="76570ffc-6a6b-4f4d-b8f0-0db5cde967c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 31 13:08:48.692594 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:08:48.692560 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:08:49.347988 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:49.347957 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" Oct 31 13:08:49.348237 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:49.348221 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" Oct 31 13:08:49.349087 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:49.349068 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" Oct 31 13:08:51.851382 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:51.851348 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-6cef79/auth-test-model-6cef79-kserve-7645c89c97-8kjxg" Oct 31 13:08:51.864801 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:51.864775 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-6cef79/auth-test-model-6cef79-kserve-7645c89c97-8kjxg" Oct 31 13:08:52.406389 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:52.406353 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-6cef79/auth-test-model-6cef79-kserve-7645c89c97-8kjxg" Oct 31 13:08:55.359463 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:55.359431 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-4574ec/auth-test-model-4574ec-kserve-85d54f44b9-nzpcc"] Oct 31 13:08:55.365525 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:55.365492 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4574ec/auth-test-model-4574ec-kserve-85d54f44b9-nzpcc" Oct 31 13:08:55.371859 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:55.371805 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-4574ec\"/\"openshift-service-ca.crt\"" Oct 31 13:08:55.372626 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:55.372596 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-4574ec\"/\"auth-test-model-4574ec-kserve-self-signed-certs\"" Oct 31 13:08:55.372814 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:55.372678 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-4574ec\"/\"kube-root-ca.crt\"" Oct 31 13:08:55.372814 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:55.372791 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-4574ec\"/\"default-dockercfg-2qql5\"" Oct 31 13:08:55.377017 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:55.376995 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4574ec/auth-test-model-4574ec-kserve-85d54f44b9-nzpcc"] Oct 31 13:08:55.437403 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:55.437374 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66wjh\" (UniqueName: \"kubernetes.io/projected/a2085991-6765-4533-aec8-b0f40de40642-kube-api-access-66wjh\") pod \"auth-test-model-4574ec-kserve-85d54f44b9-nzpcc\" (UID: \"a2085991-6765-4533-aec8-b0f40de40642\") " pod="ai-tenant-e2e-shared-a-w3-4574ec/auth-test-model-4574ec-kserve-85d54f44b9-nzpcc" Oct 31 13:08:55.437403 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:55.437406 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2085991-6765-4533-aec8-b0f40de40642-model-cache\") pod \"auth-test-model-4574ec-kserve-85d54f44b9-nzpcc\" (UID: \"a2085991-6765-4533-aec8-b0f40de40642\") " pod="ai-tenant-e2e-shared-a-w3-4574ec/auth-test-model-4574ec-kserve-85d54f44b9-nzpcc" Oct 31 13:08:55.437608 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:55.437432 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2085991-6765-4533-aec8-b0f40de40642-tmp-dir\") pod \"auth-test-model-4574ec-kserve-85d54f44b9-nzpcc\" (UID: \"a2085991-6765-4533-aec8-b0f40de40642\") " pod="ai-tenant-e2e-shared-a-w3-4574ec/auth-test-model-4574ec-kserve-85d54f44b9-nzpcc" Oct 31 13:08:55.437608 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:55.437569 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2085991-6765-4533-aec8-b0f40de40642-home\") pod \"auth-test-model-4574ec-kserve-85d54f44b9-nzpcc\" (UID: \"a2085991-6765-4533-aec8-b0f40de40642\") " pod="ai-tenant-e2e-shared-a-w3-4574ec/auth-test-model-4574ec-kserve-85d54f44b9-nzpcc" Oct 31 13:08:55.437703 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:55.437624 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2085991-6765-4533-aec8-b0f40de40642-tls-certs\") pod \"auth-test-model-4574ec-kserve-85d54f44b9-nzpcc\" (UID: \"a2085991-6765-4533-aec8-b0f40de40642\") " pod="ai-tenant-e2e-shared-a-w3-4574ec/auth-test-model-4574ec-kserve-85d54f44b9-nzpcc" Oct 31 13:08:55.437793 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:55.437780 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2085991-6765-4533-aec8-b0f40de40642-dshm\") pod \"auth-test-model-4574ec-kserve-85d54f44b9-nzpcc\" (UID: \"a2085991-6765-4533-aec8-b0f40de40642\") " pod="ai-tenant-e2e-shared-a-w3-4574ec/auth-test-model-4574ec-kserve-85d54f44b9-nzpcc" Oct 31 13:08:55.539065 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:55.539028 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66wjh\" (UniqueName: \"kubernetes.io/projected/a2085991-6765-4533-aec8-b0f40de40642-kube-api-access-66wjh\") pod \"auth-test-model-4574ec-kserve-85d54f44b9-nzpcc\" (UID: \"a2085991-6765-4533-aec8-b0f40de40642\") " pod="ai-tenant-e2e-shared-a-w3-4574ec/auth-test-model-4574ec-kserve-85d54f44b9-nzpcc" Oct 31 13:08:55.539065 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:55.539060 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2085991-6765-4533-aec8-b0f40de40642-model-cache\") pod \"auth-test-model-4574ec-kserve-85d54f44b9-nzpcc\" (UID: \"a2085991-6765-4533-aec8-b0f40de40642\") " pod="ai-tenant-e2e-shared-a-w3-4574ec/auth-test-model-4574ec-kserve-85d54f44b9-nzpcc" Oct 31 13:08:55.539301 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:55.539182 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2085991-6765-4533-aec8-b0f40de40642-tmp-dir\") pod \"auth-test-model-4574ec-kserve-85d54f44b9-nzpcc\" (UID: \"a2085991-6765-4533-aec8-b0f40de40642\") " pod="ai-tenant-e2e-shared-a-w3-4574ec/auth-test-model-4574ec-kserve-85d54f44b9-nzpcc" Oct 31 13:08:55.539301 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:55.539240 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2085991-6765-4533-aec8-b0f40de40642-home\") pod \"auth-test-model-4574ec-kserve-85d54f44b9-nzpcc\" (UID: \"a2085991-6765-4533-aec8-b0f40de40642\") " pod="ai-tenant-e2e-shared-a-w3-4574ec/auth-test-model-4574ec-kserve-85d54f44b9-nzpcc" Oct 31 13:08:55.539301 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:55.539299 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2085991-6765-4533-aec8-b0f40de40642-tls-certs\") pod \"auth-test-model-4574ec-kserve-85d54f44b9-nzpcc\" (UID: \"a2085991-6765-4533-aec8-b0f40de40642\") " pod="ai-tenant-e2e-shared-a-w3-4574ec/auth-test-model-4574ec-kserve-85d54f44b9-nzpcc" Oct 31 13:08:55.539475 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:55.539402 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2085991-6765-4533-aec8-b0f40de40642-model-cache\") pod \"auth-test-model-4574ec-kserve-85d54f44b9-nzpcc\" (UID: \"a2085991-6765-4533-aec8-b0f40de40642\") " pod="ai-tenant-e2e-shared-a-w3-4574ec/auth-test-model-4574ec-kserve-85d54f44b9-nzpcc" Oct 31 13:08:55.539565 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:55.539541 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2085991-6765-4533-aec8-b0f40de40642-dshm\") pod \"auth-test-model-4574ec-kserve-85d54f44b9-nzpcc\" (UID: \"a2085991-6765-4533-aec8-b0f40de40642\") " pod="ai-tenant-e2e-shared-a-w3-4574ec/auth-test-model-4574ec-kserve-85d54f44b9-nzpcc" Oct 31 13:08:55.539676 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:55.539571 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2085991-6765-4533-aec8-b0f40de40642-tmp-dir\") pod \"auth-test-model-4574ec-kserve-85d54f44b9-nzpcc\" (UID: \"a2085991-6765-4533-aec8-b0f40de40642\") " pod="ai-tenant-e2e-shared-a-w3-4574ec/auth-test-model-4574ec-kserve-85d54f44b9-nzpcc" Oct 31 13:08:55.539676 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:55.539660 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2085991-6765-4533-aec8-b0f40de40642-home\") pod \"auth-test-model-4574ec-kserve-85d54f44b9-nzpcc\" (UID: \"a2085991-6765-4533-aec8-b0f40de40642\") " pod="ai-tenant-e2e-shared-a-w3-4574ec/auth-test-model-4574ec-kserve-85d54f44b9-nzpcc" Oct 31 13:08:55.541909 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:55.541884 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2085991-6765-4533-aec8-b0f40de40642-dshm\") pod \"auth-test-model-4574ec-kserve-85d54f44b9-nzpcc\" (UID: \"a2085991-6765-4533-aec8-b0f40de40642\") " pod="ai-tenant-e2e-shared-a-w3-4574ec/auth-test-model-4574ec-kserve-85d54f44b9-nzpcc" Oct 31 13:08:55.542022 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:55.541968 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2085991-6765-4533-aec8-b0f40de40642-tls-certs\") pod \"auth-test-model-4574ec-kserve-85d54f44b9-nzpcc\" (UID: \"a2085991-6765-4533-aec8-b0f40de40642\") " pod="ai-tenant-e2e-shared-a-w3-4574ec/auth-test-model-4574ec-kserve-85d54f44b9-nzpcc" Oct 31 13:08:55.570726 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:55.566503 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66wjh\" (UniqueName: \"kubernetes.io/projected/a2085991-6765-4533-aec8-b0f40de40642-kube-api-access-66wjh\") pod \"auth-test-model-4574ec-kserve-85d54f44b9-nzpcc\" (UID: \"a2085991-6765-4533-aec8-b0f40de40642\") " pod="ai-tenant-e2e-shared-a-w3-4574ec/auth-test-model-4574ec-kserve-85d54f44b9-nzpcc" Oct 31 13:08:55.685280 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:55.685240 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4574ec/auth-test-model-4574ec-kserve-85d54f44b9-nzpcc" Oct 31 13:08:55.989970 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:55.989936 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4574ec/auth-test-model-4574ec-kserve-85d54f44b9-nzpcc"] Oct 31 13:08:55.990858 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:08:55.990825 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2085991_6765_4533_aec8_b0f40de40642.slice/crio-460e36cb64c3a117e7eb5a9c5ebebba6b041fc70ae4e75a2ff5684ace051a9d7 WatchSource:0}: Error finding container 460e36cb64c3a117e7eb5a9c5ebebba6b041fc70ae4e75a2ff5684ace051a9d7: Status 404 returned error can't find the container with id 460e36cb64c3a117e7eb5a9c5ebebba6b041fc70ae4e75a2ff5684ace051a9d7 Oct 31 13:08:56.434983 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:56.434949 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4574ec/auth-test-model-4574ec-kserve-85d54f44b9-nzpcc" event={"ID":"a2085991-6765-4533-aec8-b0f40de40642","Type":"ContainerStarted","Data":"c3e4fca749625bd6a320e80c9874ed83d6c86a402be95d6d1674d7d5d576ae38"} Oct 31 13:08:56.435383 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:56.434991 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4574ec/auth-test-model-4574ec-kserve-85d54f44b9-nzpcc" event={"ID":"a2085991-6765-4533-aec8-b0f40de40642","Type":"ContainerStarted","Data":"460e36cb64c3a117e7eb5a9c5ebebba6b041fc70ae4e75a2ff5684ace051a9d7"} Oct 31 13:08:56.455671 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:56.455601 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-4574ec/auth-test-model-4574ec-kserve-85d54f44b9-nzpcc" podStartSLOduration=1.455587964 podStartE2EDuration="1.455587964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:08:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:08:56.454034081 +0000 UTC m=+2187.161620377" watchObservedRunningTime="2026-10-31 13:08:56.455587964 +0000 UTC m=+2187.163174313" Oct 31 13:08:56.686290 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:08:56.686214 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-4574ec/auth-test-model-4574ec-kserve-85d54f44b9-nzpcc" Oct 31 13:09:05.685711 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:05.685667 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-4574ec/auth-test-model-4574ec-kserve-85d54f44b9-nzpcc" Oct 31 13:09:05.698756 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:05.698732 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-4574ec/auth-test-model-4574ec-kserve-85d54f44b9-nzpcc" Oct 31 13:09:06.557482 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:06.557452 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-4574ec/auth-test-model-4574ec-kserve-85d54f44b9-nzpcc" Oct 31 13:09:10.812505 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:10.812466 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p"] Oct 31 13:09:10.819016 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:10.818985 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" Oct 31 13:09:10.863586 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:10.838673 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p"] Oct 31 13:09:10.908392 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:10.908352 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5455d8fe-86dd-413e-a001-80b5e91c7414-istio-data\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p\" (UID: \"5455d8fe-86dd-413e-a001-80b5e91c7414\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" Oct 31 13:09:10.908572 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:10.908417 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5455d8fe-86dd-413e-a001-80b5e91c7414-credential-socket\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p\" (UID: \"5455d8fe-86dd-413e-a001-80b5e91c7414\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" Oct 31 13:09:10.908572 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:10.908509 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5455d8fe-86dd-413e-a001-80b5e91c7414-istio-podinfo\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p\" (UID: \"5455d8fe-86dd-413e-a001-80b5e91c7414\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" Oct 31 13:09:10.913539 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:10.908571 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5455d8fe-86dd-413e-a001-80b5e91c7414-workload-certs\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p\" (UID: \"5455d8fe-86dd-413e-a001-80b5e91c7414\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" Oct 31 13:09:10.913539 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:10.908616 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5455d8fe-86dd-413e-a001-80b5e91c7414-istio-envoy\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p\" (UID: \"5455d8fe-86dd-413e-a001-80b5e91c7414\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" Oct 31 13:09:10.913539 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:10.908674 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5455d8fe-86dd-413e-a001-80b5e91c7414-istio-token\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p\" (UID: \"5455d8fe-86dd-413e-a001-80b5e91c7414\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" Oct 31 13:09:10.913539 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:10.908744 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5455d8fe-86dd-413e-a001-80b5e91c7414-workload-socket\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p\" (UID: \"5455d8fe-86dd-413e-a001-80b5e91c7414\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" Oct 31 13:09:10.913539 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:10.908901 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gghgv\" (UniqueName: \"kubernetes.io/projected/5455d8fe-86dd-413e-a001-80b5e91c7414-kube-api-access-gghgv\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p\" (UID: \"5455d8fe-86dd-413e-a001-80b5e91c7414\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" Oct 31 13:09:10.913539 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:10.908941 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5455d8fe-86dd-413e-a001-80b5e91c7414-istiod-ca-cert\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p\" (UID: \"5455d8fe-86dd-413e-a001-80b5e91c7414\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" Oct 31 13:09:10.913539 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:10.912372 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-7cdec6-openshift-default-dockercfg-85mvx\"" Oct 31 13:09:11.010855 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:11.010797 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gghgv\" (UniqueName: \"kubernetes.io/projected/5455d8fe-86dd-413e-a001-80b5e91c7414-kube-api-access-gghgv\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p\" (UID: \"5455d8fe-86dd-413e-a001-80b5e91c7414\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" Oct 31 13:09:11.011057 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:11.010863 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5455d8fe-86dd-413e-a001-80b5e91c7414-istiod-ca-cert\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p\" (UID: \"5455d8fe-86dd-413e-a001-80b5e91c7414\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" Oct 31 13:09:11.011057 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:11.011005 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5455d8fe-86dd-413e-a001-80b5e91c7414-istio-data\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p\" (UID: \"5455d8fe-86dd-413e-a001-80b5e91c7414\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" Oct 31 13:09:11.011057 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:11.011032 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5455d8fe-86dd-413e-a001-80b5e91c7414-credential-socket\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p\" (UID: \"5455d8fe-86dd-413e-a001-80b5e91c7414\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" Oct 31 13:09:11.011268 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:11.011130 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5455d8fe-86dd-413e-a001-80b5e91c7414-istio-podinfo\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p\" (UID: \"5455d8fe-86dd-413e-a001-80b5e91c7414\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" Oct 31 13:09:11.011334 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:11.011293 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5455d8fe-86dd-413e-a001-80b5e91c7414-workload-certs\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p\" (UID: \"5455d8fe-86dd-413e-a001-80b5e91c7414\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" Oct 31 13:09:11.011391 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:11.011340 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5455d8fe-86dd-413e-a001-80b5e91c7414-istio-envoy\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p\" (UID: \"5455d8fe-86dd-413e-a001-80b5e91c7414\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" Oct 31 13:09:11.011450 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:11.011398 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5455d8fe-86dd-413e-a001-80b5e91c7414-istio-token\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p\" (UID: \"5455d8fe-86dd-413e-a001-80b5e91c7414\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" Oct 31 13:09:11.011503 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:11.011484 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5455d8fe-86dd-413e-a001-80b5e91c7414-workload-socket\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p\" (UID: \"5455d8fe-86dd-413e-a001-80b5e91c7414\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" Oct 31 13:09:11.014220 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:11.012402 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5455d8fe-86dd-413e-a001-80b5e91c7414-istiod-ca-cert\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p\" (UID: \"5455d8fe-86dd-413e-a001-80b5e91c7414\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" Oct 31 13:09:11.014917 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:11.014892 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5455d8fe-86dd-413e-a001-80b5e91c7414-credential-socket\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p\" (UID: \"5455d8fe-86dd-413e-a001-80b5e91c7414\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" Oct 31 13:09:11.015329 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:11.015304 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5455d8fe-86dd-413e-a001-80b5e91c7414-workload-socket\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p\" (UID: \"5455d8fe-86dd-413e-a001-80b5e91c7414\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" Oct 31 13:09:11.030981 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:11.015553 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5455d8fe-86dd-413e-a001-80b5e91c7414-istio-data\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p\" (UID: \"5455d8fe-86dd-413e-a001-80b5e91c7414\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" Oct 31 13:09:11.030981 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:11.015839 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5455d8fe-86dd-413e-a001-80b5e91c7414-workload-certs\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p\" (UID: \"5455d8fe-86dd-413e-a001-80b5e91c7414\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" Oct 31 13:09:11.030981 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:11.017516 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5455d8fe-86dd-413e-a001-80b5e91c7414-istio-podinfo\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p\" (UID: \"5455d8fe-86dd-413e-a001-80b5e91c7414\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" Oct 31 13:09:11.030981 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:11.018547 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5455d8fe-86dd-413e-a001-80b5e91c7414-istio-envoy\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p\" (UID: \"5455d8fe-86dd-413e-a001-80b5e91c7414\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" Oct 31 13:09:11.043935 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:11.038311 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5455d8fe-86dd-413e-a001-80b5e91c7414-istio-token\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p\" (UID: \"5455d8fe-86dd-413e-a001-80b5e91c7414\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" Oct 31 13:09:11.047038 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:11.047005 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gghgv\" (UniqueName: \"kubernetes.io/projected/5455d8fe-86dd-413e-a001-80b5e91c7414-kube-api-access-gghgv\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p\" (UID: \"5455d8fe-86dd-413e-a001-80b5e91c7414\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" Oct 31 13:09:11.249876 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:11.249843 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" Oct 31 13:09:11.540943 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:11.540911 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p"] Oct 31 13:09:11.542236 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:09:11.542211 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5455d8fe_86dd_413e_a001_80b5e91c7414.slice/crio-f73bac3b4ad5a0768b43b2123d37d477e4df3d02f09c57dea6affc387edc1b9b WatchSource:0}: Error finding container f73bac3b4ad5a0768b43b2123d37d477e4df3d02f09c57dea6affc387edc1b9b: Status 404 returned error can't find the container with id f73bac3b4ad5a0768b43b2123d37d477e4df3d02f09c57dea6affc387edc1b9b Oct 31 13:09:11.545056 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:11.545024 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:09:11.545122 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:11.545091 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:09:11.545174 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:11.545121 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:09:11.582024 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:11.581990 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" event={"ID":"5455d8fe-86dd-413e-a001-80b5e91c7414","Type":"ContainerStarted","Data":"f73bac3b4ad5a0768b43b2123d37d477e4df3d02f09c57dea6affc387edc1b9b"} Oct 31 13:09:12.615812 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:12.615760 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" event={"ID":"5455d8fe-86dd-413e-a001-80b5e91c7414","Type":"ContainerStarted","Data":"a8c49580c1a5b520cfafd57e24b0bf03b80a995d4e27fda418772adf7252cfdb"} Oct 31 13:09:12.644757 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:12.644710 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" podStartSLOduration=2.644691798 podStartE2EDuration="2.644691798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:09:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:09:12.644335039 +0000 UTC m=+2203.351921345" watchObservedRunningTime="2026-10-31 13:09:12.644691798 +0000 UTC m=+2203.352278096" Oct 31 13:09:13.250440 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:13.250405 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" Oct 31 13:09:13.252419 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:13.252388 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 31 13:09:13.252568 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:13.252444 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" podUID="5455d8fe-86dd-413e-a001-80b5e91c7414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 31 13:09:14.250542 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:14.250510 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 31 13:09:14.251008 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:14.250572 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" podUID="5455d8fe-86dd-413e-a001-80b5e91c7414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 31 13:09:15.251155 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:15.250937 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 31 13:09:15.251155 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:15.250993 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" podUID="5455d8fe-86dd-413e-a001-80b5e91c7414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 31 13:09:16.250511 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:16.250475 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 31 13:09:16.250770 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:16.250544 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" podUID="5455d8fe-86dd-413e-a001-80b5e91c7414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 31 13:09:17.250816 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:17.250783 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 31 13:09:17.251354 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:17.250842 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" podUID="5455d8fe-86dd-413e-a001-80b5e91c7414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 31 13:09:18.255128 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:18.255097 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" Oct 31 13:09:18.255598 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:18.255589 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" Oct 31 13:09:18.256320 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:18.256287 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" Oct 31 13:09:18.689258 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:09:18.689223 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:09:36.372589 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:36.372550 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-6106a3-545f977779-44hnw"] Oct 31 13:09:40.845690 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:40.841121 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74"] Oct 31 13:09:40.845690 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:40.841400 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74" podUID="00951dea-adc0-4e22-a003-221c5c19bd80" containerName="maas-api" containerID="cri-o://93d8c0782b3c942b07e7d93be2adef4e5887c40a6f244fa4be3a5f9a88dc3a5a" gracePeriod=30 Oct 31 13:09:41.297988 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:41.297958 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74" Oct 31 13:09:41.373749 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:41.373701 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-6106a3-545f977779-44hnw" podUID="6665c331-4ab3-4ef2-8b08-498509dfe585" containerName="payload-processing" containerID="cri-o://7d22362eb26f7991cf325e73f2d2ad3272cb5d2f4255bc560129dad26867166d" gracePeriod=25 Oct 31 13:09:41.455905 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:41.455874 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00951dea-adc0-4e22-a003-221c5c19bd80-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00951dea-adc0-4e22-a003-221c5c19bd80-openshift-service-ca\") pod \"00951dea-adc0-4e22-a003-221c5c19bd80\" (UID: \"00951dea-adc0-4e22-a003-221c5c19bd80\") " Oct 31 13:09:41.456095 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:41.455920 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00951dea-adc0-4e22-a003-221c5c19bd80-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00951dea-adc0-4e22-a003-221c5c19bd80-maas-api-tls\") pod \"00951dea-adc0-4e22-a003-221c5c19bd80\" (UID: \"00951dea-adc0-4e22-a003-221c5c19bd80\") " Oct 31 13:09:41.456095 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:41.455968 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00951dea-adc0-4e22-a003-221c5c19bd80-kube-api-access-5fvql\" (UniqueName: \"kubernetes.io/projected/00951dea-adc0-4e22-a003-221c5c19bd80-kube-api-access-5fvql\") pod \"00951dea-adc0-4e22-a003-221c5c19bd80\" (UID: \"00951dea-adc0-4e22-a003-221c5c19bd80\") " Oct 31 13:09:41.456095 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:41.456003 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00951dea-adc0-4e22-a003-221c5c19bd80-metrics-certs\" (UniqueName: \"kubernetes.io/secret/00951dea-adc0-4e22-a003-221c5c19bd80-metrics-certs\") pod \"00951dea-adc0-4e22-a003-221c5c19bd80\" (UID: \"00951dea-adc0-4e22-a003-221c5c19bd80\") " Oct 31 13:09:41.456340 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:41.456312 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00951dea-adc0-4e22-a003-221c5c19bd80-openshift-service-ca" pod "00951dea-adc0-4e22-a003-221c5c19bd80" (UID: "00951dea-adc0-4e22-a003-221c5c19bd80"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:09:41.456682 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:41.456666 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00951dea-adc0-4e22-a003-221c5c19bd80-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:41.458393 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:41.458364 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00951dea-adc0-4e22-a003-221c5c19bd80-maas-api-tls" pod "00951dea-adc0-4e22-a003-221c5c19bd80" (UID: "00951dea-adc0-4e22-a003-221c5c19bd80"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:09:41.458492 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:41.458413 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00951dea-adc0-4e22-a003-221c5c19bd80-metrics-certs" pod "00951dea-adc0-4e22-a003-221c5c19bd80" (UID: "00951dea-adc0-4e22-a003-221c5c19bd80"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:09:41.458492 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:41.458437 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00951dea-adc0-4e22-a003-221c5c19bd80-kube-api-access-5fvql" pod "00951dea-adc0-4e22-a003-221c5c19bd80" (UID: "00951dea-adc0-4e22-a003-221c5c19bd80"). InnerVolumeSpecName "kube-api-access-5fvql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:09:41.558331 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:41.558026 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00951dea-adc0-4e22-a003-221c5c19bd80-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:41.558331 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:41.558061 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fvql\" (UniqueName: \"kubernetes.io/projected/00951dea-adc0-4e22-a003-221c5c19bd80-kube-api-access-5fvql\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:41.558331 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:41.558078 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00951dea-adc0-4e22-a003-221c5c19bd80-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:41.649315 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:41.649289 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-6106a3-545f977779-44hnw" Oct 31 13:09:41.761665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:41.761551 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6665c331-4ab3-4ef2-8b08-498509dfe585-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6665c331-4ab3-4ef2-8b08-498509dfe585-plugins-config-volume\") pod \"6665c331-4ab3-4ef2-8b08-498509dfe585\" (UID: \"6665c331-4ab3-4ef2-8b08-498509dfe585\") " Oct 31 13:09:41.761867 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:41.761680 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6665c331-4ab3-4ef2-8b08-498509dfe585-kube-api-access-zcqkn\" (UniqueName: \"kubernetes.io/projected/6665c331-4ab3-4ef2-8b08-498509dfe585-kube-api-access-zcqkn\") pod \"6665c331-4ab3-4ef2-8b08-498509dfe585\" (UID: \"6665c331-4ab3-4ef2-8b08-498509dfe585\") " Oct 31 13:09:41.765666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:41.762973 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6665c331-4ab3-4ef2-8b08-498509dfe585-plugins-config-volume" pod "6665c331-4ab3-4ef2-8b08-498509dfe585" (UID: "6665c331-4ab3-4ef2-8b08-498509dfe585"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:09:41.765666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:41.764033 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6665c331-4ab3-4ef2-8b08-498509dfe585-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:41.768695 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:41.767552 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6665c331-4ab3-4ef2-8b08-498509dfe585-kube-api-access-zcqkn" pod "6665c331-4ab3-4ef2-8b08-498509dfe585" (UID: "6665c331-4ab3-4ef2-8b08-498509dfe585"). InnerVolumeSpecName "kube-api-access-zcqkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:09:41.865693 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:41.865561 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcqkn\" (UniqueName: \"kubernetes.io/projected/6665c331-4ab3-4ef2-8b08-498509dfe585-kube-api-access-zcqkn\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:41.951324 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:41.951291 5693 generic.go:356] "Generic (PLEG): container finished" podID="6665c331-4ab3-4ef2-8b08-498509dfe585" containerID="7d22362eb26f7991cf325e73f2d2ad3272cb5d2f4255bc560129dad26867166d" exitCode=0 Oct 31 13:09:41.951523 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:41.951328 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-6106a3-545f977779-44hnw" event={"ID":"6665c331-4ab3-4ef2-8b08-498509dfe585","Type":"ContainerDied","Data":"7d22362eb26f7991cf325e73f2d2ad3272cb5d2f4255bc560129dad26867166d"} Oct 31 13:09:41.951523 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:41.951396 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-6106a3-545f977779-44hnw" event={"ID":"6665c331-4ab3-4ef2-8b08-498509dfe585","Type":"ContainerDied","Data":"30d239d24c4b6edc9ac9166817c43fefb887e8311de9a0db5217841facaadb3c"} Oct 31 13:09:41.951523 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:41.951343 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-6106a3-545f977779-44hnw" Oct 31 13:09:41.951523 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:41.951428 5693 scope.go:122] "RemoveContainer" containerID="7d22362eb26f7991cf325e73f2d2ad3272cb5d2f4255bc560129dad26867166d" Oct 31 13:09:41.952982 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:41.952873 5693 generic.go:356] "Generic (PLEG): container finished" podID="00951dea-adc0-4e22-a003-221c5c19bd80" containerID="93d8c0782b3c942b07e7d93be2adef4e5887c40a6f244fa4be3a5f9a88dc3a5a" exitCode=0 Oct 31 13:09:41.952982 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:41.952928 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74" Oct 31 13:09:41.953122 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:41.952965 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74" event={"ID":"00951dea-adc0-4e22-a003-221c5c19bd80","Type":"ContainerDied","Data":"93d8c0782b3c942b07e7d93be2adef4e5887c40a6f244fa4be3a5f9a88dc3a5a"} Oct 31 13:09:41.953122 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:41.953029 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74" event={"ID":"00951dea-adc0-4e22-a003-221c5c19bd80","Type":"ContainerDied","Data":"35a78cecd2422edfc1e653a8758c7507fb53ec5d3d7e5bbce2ef98c2e40b38af"} Oct 31 13:09:41.969736 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:41.969714 5693 scope.go:122] "RemoveContainer" containerID="7d22362eb26f7991cf325e73f2d2ad3272cb5d2f4255bc560129dad26867166d" Oct 31 13:09:41.970258 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:09:41.970235 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d22362eb26f7991cf325e73f2d2ad3272cb5d2f4255bc560129dad26867166d\": container with ID starting with 7d22362eb26f7991cf325e73f2d2ad3272cb5d2f4255bc560129dad26867166d not found: ID does not exist" containerID="7d22362eb26f7991cf325e73f2d2ad3272cb5d2f4255bc560129dad26867166d" Oct 31 13:09:41.970358 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:41.970267 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d22362eb26f7991cf325e73f2d2ad3272cb5d2f4255bc560129dad26867166d"} err="failed to get container status \"7d22362eb26f7991cf325e73f2d2ad3272cb5d2f4255bc560129dad26867166d\": rpc error: code = NotFound desc = could not find container \"7d22362eb26f7991cf325e73f2d2ad3272cb5d2f4255bc560129dad26867166d\": container with ID starting with 7d22362eb26f7991cf325e73f2d2ad3272cb5d2f4255bc560129dad26867166d not found: ID does not exist" Oct 31 13:09:41.970358 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:41.970289 5693 scope.go:122] "RemoveContainer" containerID="93d8c0782b3c942b07e7d93be2adef4e5887c40a6f244fa4be3a5f9a88dc3a5a" Oct 31 13:09:41.980088 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:41.980060 5693 scope.go:122] "RemoveContainer" containerID="93d8c0782b3c942b07e7d93be2adef4e5887c40a6f244fa4be3a5f9a88dc3a5a" Oct 31 13:09:41.980443 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:09:41.980421 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93d8c0782b3c942b07e7d93be2adef4e5887c40a6f244fa4be3a5f9a88dc3a5a\": container with ID starting with 93d8c0782b3c942b07e7d93be2adef4e5887c40a6f244fa4be3a5f9a88dc3a5a not found: ID does not exist" containerID="93d8c0782b3c942b07e7d93be2adef4e5887c40a6f244fa4be3a5f9a88dc3a5a" Oct 31 13:09:41.980517 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:41.980455 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93d8c0782b3c942b07e7d93be2adef4e5887c40a6f244fa4be3a5f9a88dc3a5a"} err="failed to get container status \"93d8c0782b3c942b07e7d93be2adef4e5887c40a6f244fa4be3a5f9a88dc3a5a\": rpc error: code = NotFound desc = could not find container \"93d8c0782b3c942b07e7d93be2adef4e5887c40a6f244fa4be3a5f9a88dc3a5a\": container with ID starting with 93d8c0782b3c942b07e7d93be2adef4e5887c40a6f244fa4be3a5f9a88dc3a5a not found: ID does not exist" Oct 31 13:09:42.046530 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:42.046497 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74"] Oct 31 13:09:42.087419 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:42.087385 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-6106a3-5ddc879cf4-xvf74"] Oct 31 13:09:42.144591 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:42.144555 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-6106a3-545f977779-44hnw"] Oct 31 13:09:42.159625 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:42.159596 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-6106a3-545f977779-44hnw"] Oct 31 13:09:43.411241 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:43.411209 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00951dea-adc0-4e22-a003-221c5c19bd80" path="/var/lib/kubelet/pods/00951dea-adc0-4e22-a003-221c5c19bd80/volumes" Oct 31 13:09:43.411687 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:43.411669 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6665c331-4ab3-4ef2-8b08-498509dfe585" path="/var/lib/kubelet/pods/6665c331-4ab3-4ef2-8b08-498509dfe585/volumes" Oct 31 13:09:46.654030 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:46.653992 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98"] Oct 31 13:09:46.655062 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:46.655042 5693 cpu_manager.go:414] "containerMap: removing container" podUID="6665c331-4ab3-4ef2-8b08-498509dfe585" containerName="payload-processing" Oct 31 13:09:46.655124 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:46.655067 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6665c331-4ab3-4ef2-8b08-498509dfe585" containerName="payload-processing" Oct 31 13:09:46.655159 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:46.655130 5693 cpu_manager.go:414] "containerMap: removing container" podUID="00951dea-adc0-4e22-a003-221c5c19bd80" containerName="maas-api" Oct 31 13:09:46.655159 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:46.655139 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00951dea-adc0-4e22-a003-221c5c19bd80" containerName="maas-api" Oct 31 13:09:46.655327 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:46.655314 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="6665c331-4ab3-4ef2-8b08-498509dfe585" containerName="payload-processing" Oct 31 13:09:46.655360 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:46.655341 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="00951dea-adc0-4e22-a003-221c5c19bd80" containerName="maas-api" Oct 31 13:09:46.661050 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:46.661026 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" Oct 31 13:09:46.682324 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:46.682303 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-95dfcc-openshift-default-dockercfg-66xmc\"" Oct 31 13:09:46.736255 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:46.736221 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98"] Oct 31 13:09:46.760546 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:46.760511 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6372c7-26c3-452c-a48f-e181e70bffd5-workload-socket\") pod \"e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98\" (UID: \"9d6372c7-26c3-452c-a48f-e181e70bffd5\") " pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" Oct 31 13:09:46.760782 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:46.760587 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d6372c7-26c3-452c-a48f-e181e70bffd5-istiod-ca-cert\") pod \"e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98\" (UID: \"9d6372c7-26c3-452c-a48f-e181e70bffd5\") " pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" Oct 31 13:09:46.760782 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:46.760717 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d6372c7-26c3-452c-a48f-e181e70bffd5-istio-envoy\") pod \"e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98\" (UID: \"9d6372c7-26c3-452c-a48f-e181e70bffd5\") " pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" Oct 31 13:09:46.760782 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:46.760761 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4nxc\" (UniqueName: \"kubernetes.io/projected/9d6372c7-26c3-452c-a48f-e181e70bffd5-kube-api-access-d4nxc\") pod \"e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98\" (UID: \"9d6372c7-26c3-452c-a48f-e181e70bffd5\") " pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" Oct 31 13:09:46.760976 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:46.760833 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6372c7-26c3-452c-a48f-e181e70bffd5-credential-socket\") pod \"e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98\" (UID: \"9d6372c7-26c3-452c-a48f-e181e70bffd5\") " pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" Oct 31 13:09:46.760976 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:46.760867 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d6372c7-26c3-452c-a48f-e181e70bffd5-istio-token\") pod \"e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98\" (UID: \"9d6372c7-26c3-452c-a48f-e181e70bffd5\") " pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" Oct 31 13:09:46.760976 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:46.760936 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d6372c7-26c3-452c-a48f-e181e70bffd5-istio-data\") pod \"e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98\" (UID: \"9d6372c7-26c3-452c-a48f-e181e70bffd5\") " pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" Oct 31 13:09:46.760976 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:46.760969 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d6372c7-26c3-452c-a48f-e181e70bffd5-istio-podinfo\") pod \"e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98\" (UID: \"9d6372c7-26c3-452c-a48f-e181e70bffd5\") " pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" Oct 31 13:09:46.761178 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:46.761041 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d6372c7-26c3-452c-a48f-e181e70bffd5-workload-certs\") pod \"e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98\" (UID: \"9d6372c7-26c3-452c-a48f-e181e70bffd5\") " pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" Oct 31 13:09:46.863230 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:46.863175 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6372c7-26c3-452c-a48f-e181e70bffd5-workload-socket\") pod \"e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98\" (UID: \"9d6372c7-26c3-452c-a48f-e181e70bffd5\") " pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" Oct 31 13:09:46.863443 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:46.863277 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d6372c7-26c3-452c-a48f-e181e70bffd5-istiod-ca-cert\") pod \"e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98\" (UID: \"9d6372c7-26c3-452c-a48f-e181e70bffd5\") " pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" Oct 31 13:09:46.863443 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:46.863349 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d6372c7-26c3-452c-a48f-e181e70bffd5-istio-envoy\") pod \"e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98\" (UID: \"9d6372c7-26c3-452c-a48f-e181e70bffd5\") " pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" Oct 31 13:09:46.863443 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:46.863389 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4nxc\" (UniqueName: \"kubernetes.io/projected/9d6372c7-26c3-452c-a48f-e181e70bffd5-kube-api-access-d4nxc\") pod \"e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98\" (UID: \"9d6372c7-26c3-452c-a48f-e181e70bffd5\") " pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" Oct 31 13:09:46.863630 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:46.863464 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6372c7-26c3-452c-a48f-e181e70bffd5-credential-socket\") pod \"e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98\" (UID: \"9d6372c7-26c3-452c-a48f-e181e70bffd5\") " pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" Oct 31 13:09:46.863630 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:46.863493 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d6372c7-26c3-452c-a48f-e181e70bffd5-istio-token\") pod \"e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98\" (UID: \"9d6372c7-26c3-452c-a48f-e181e70bffd5\") " pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" Oct 31 13:09:46.863630 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:46.863565 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d6372c7-26c3-452c-a48f-e181e70bffd5-istio-data\") pod \"e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98\" (UID: \"9d6372c7-26c3-452c-a48f-e181e70bffd5\") " pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" Oct 31 13:09:46.863630 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:46.863597 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d6372c7-26c3-452c-a48f-e181e70bffd5-istio-podinfo\") pod \"e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98\" (UID: \"9d6372c7-26c3-452c-a48f-e181e70bffd5\") " pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" Oct 31 13:09:46.863913 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:46.863726 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d6372c7-26c3-452c-a48f-e181e70bffd5-workload-certs\") pod \"e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98\" (UID: \"9d6372c7-26c3-452c-a48f-e181e70bffd5\") " pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" Oct 31 13:09:46.864791 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:46.864153 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d6372c7-26c3-452c-a48f-e181e70bffd5-istiod-ca-cert\") pod \"e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98\" (UID: \"9d6372c7-26c3-452c-a48f-e181e70bffd5\") " pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" Oct 31 13:09:46.864791 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:46.864362 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d6372c7-26c3-452c-a48f-e181e70bffd5-workload-certs\") pod \"e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98\" (UID: \"9d6372c7-26c3-452c-a48f-e181e70bffd5\") " pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" Oct 31 13:09:46.864791 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:46.864442 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6372c7-26c3-452c-a48f-e181e70bffd5-credential-socket\") pod \"e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98\" (UID: \"9d6372c7-26c3-452c-a48f-e181e70bffd5\") " pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" Oct 31 13:09:46.865020 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:46.864957 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d6372c7-26c3-452c-a48f-e181e70bffd5-istio-data\") pod \"e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98\" (UID: \"9d6372c7-26c3-452c-a48f-e181e70bffd5\") " pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" Oct 31 13:09:46.866007 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:46.865921 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6372c7-26c3-452c-a48f-e181e70bffd5-workload-socket\") pod \"e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98\" (UID: \"9d6372c7-26c3-452c-a48f-e181e70bffd5\") " pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" Oct 31 13:09:46.868686 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:46.868612 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d6372c7-26c3-452c-a48f-e181e70bffd5-istio-podinfo\") pod \"e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98\" (UID: \"9d6372c7-26c3-452c-a48f-e181e70bffd5\") " pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" Oct 31 13:09:46.869321 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:46.869291 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d6372c7-26c3-452c-a48f-e181e70bffd5-istio-envoy\") pod \"e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98\" (UID: \"9d6372c7-26c3-452c-a48f-e181e70bffd5\") " pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" Oct 31 13:09:46.898846 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:46.898663 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d6372c7-26c3-452c-a48f-e181e70bffd5-istio-token\") pod \"e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98\" (UID: \"9d6372c7-26c3-452c-a48f-e181e70bffd5\") " pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" Oct 31 13:09:46.898846 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:46.898816 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4nxc\" (UniqueName: \"kubernetes.io/projected/9d6372c7-26c3-452c-a48f-e181e70bffd5-kube-api-access-d4nxc\") pod \"e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98\" (UID: \"9d6372c7-26c3-452c-a48f-e181e70bffd5\") " pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" Oct 31 13:09:46.979741 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:46.979607 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" Oct 31 13:09:47.288984 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:47.288952 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98"] Oct 31 13:09:47.289527 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:09:47.289503 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d6372c7_26c3_452c_a48f_e181e70bffd5.slice/crio-617e23a0294845976b542a7f03bb10b6f42e36880f476a63654e31102140ef13 WatchSource:0}: Error finding container 617e23a0294845976b542a7f03bb10b6f42e36880f476a63654e31102140ef13: Status 404 returned error can't find the container with id 617e23a0294845976b542a7f03bb10b6f42e36880f476a63654e31102140ef13 Oct 31 13:09:47.292039 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:47.292006 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:09:47.292119 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:47.292073 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:09:47.292119 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:47.292104 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:09:48.061226 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:48.061184 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" event={"ID":"9d6372c7-26c3-452c-a48f-e181e70bffd5","Type":"ContainerStarted","Data":"53e222b54aa010658e9101021aa03e18b7f698153b53d8f2448c5144bf2492a9"} Oct 31 13:09:48.061617 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:48.061253 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" event={"ID":"9d6372c7-26c3-452c-a48f-e181e70bffd5","Type":"ContainerStarted","Data":"617e23a0294845976b542a7f03bb10b6f42e36880f476a63654e31102140ef13"} Oct 31 13:09:48.087091 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:48.087041 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" podStartSLOduration=2.08702348 podStartE2EDuration="2.08702348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:09:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:09:48.084518353 +0000 UTC m=+2238.792104649" watchObservedRunningTime="2026-10-31 13:09:48.08702348 +0000 UTC m=+2238.794609864" Oct 31 13:09:48.729763 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:09:48.729725 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:09:48.983724 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:48.982018 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" Oct 31 13:09:48.999420 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:48.999145 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" Oct 31 13:09:49.072944 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:49.072913 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" Oct 31 13:09:49.073784 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:09:49.073765 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" Oct 31 13:10:01.222670 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:01.222051 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98"] Oct 31 13:10:01.222670 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:01.222379 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" podUID="9d6372c7-26c3-452c-a48f-e181e70bffd5" containerName="istio-proxy" containerID="cri-o://53e222b54aa010658e9101021aa03e18b7f698153b53d8f2448c5144bf2492a9" gracePeriod=30 Oct 31 13:10:04.076129 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:04.076095 5693 patch_prober.go:29] interesting pod/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Oct 31 13:10:04.076634 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:04.076158 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" podUID="9d6372c7-26c3-452c-a48f-e181e70bffd5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:10:04.134276 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:04.134236 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv"] Oct 31 13:10:04.134480 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:04.134456 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" podUID="2ae02ff0-1e33-4007-a97a-63a61e8b4bf4" containerName="istio-proxy" containerID="cri-o://98ad7ee33e380455fc042048fdba1e1b9f1c9e0ff0aa2f2ee06f8385255781d0" gracePeriod=30 Oct 31 13:10:06.596201 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:06.596146 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" Oct 31 13:10:06.639848 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:06.639810 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d6372c7-26c3-452c-a48f-e181e70bffd5-kube-api-access-d4nxc\" (UniqueName: \"kubernetes.io/projected/9d6372c7-26c3-452c-a48f-e181e70bffd5-kube-api-access-d4nxc\") pod \"9d6372c7-26c3-452c-a48f-e181e70bffd5\" (UID: \"9d6372c7-26c3-452c-a48f-e181e70bffd5\") " Oct 31 13:10:06.640048 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:06.639974 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d6372c7-26c3-452c-a48f-e181e70bffd5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6372c7-26c3-452c-a48f-e181e70bffd5-workload-socket\") pod \"9d6372c7-26c3-452c-a48f-e181e70bffd5\" (UID: \"9d6372c7-26c3-452c-a48f-e181e70bffd5\") " Oct 31 13:10:06.640048 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:06.640035 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d6372c7-26c3-452c-a48f-e181e70bffd5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d6372c7-26c3-452c-a48f-e181e70bffd5-istio-envoy\") pod \"9d6372c7-26c3-452c-a48f-e181e70bffd5\" (UID: \"9d6372c7-26c3-452c-a48f-e181e70bffd5\") " Oct 31 13:10:06.640163 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:06.640144 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d6372c7-26c3-452c-a48f-e181e70bffd5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6372c7-26c3-452c-a48f-e181e70bffd5-credential-socket\") pod \"9d6372c7-26c3-452c-a48f-e181e70bffd5\" (UID: \"9d6372c7-26c3-452c-a48f-e181e70bffd5\") " Oct 31 13:10:06.640220 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:06.640187 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d6372c7-26c3-452c-a48f-e181e70bffd5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d6372c7-26c3-452c-a48f-e181e70bffd5-istio-podinfo\") pod \"9d6372c7-26c3-452c-a48f-e181e70bffd5\" (UID: \"9d6372c7-26c3-452c-a48f-e181e70bffd5\") " Oct 31 13:10:06.640311 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:06.640287 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d6372c7-26c3-452c-a48f-e181e70bffd5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d6372c7-26c3-452c-a48f-e181e70bffd5-istio-data\") pod \"9d6372c7-26c3-452c-a48f-e181e70bffd5\" (UID: \"9d6372c7-26c3-452c-a48f-e181e70bffd5\") " Oct 31 13:10:06.640572 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:06.640553 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d6372c7-26c3-452c-a48f-e181e70bffd5-istio-token\" (UniqueName: \"kubernetes.io/projected/9d6372c7-26c3-452c-a48f-e181e70bffd5-istio-token\") pod \"9d6372c7-26c3-452c-a48f-e181e70bffd5\" (UID: \"9d6372c7-26c3-452c-a48f-e181e70bffd5\") " Oct 31 13:10:06.640760 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:06.640742 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d6372c7-26c3-452c-a48f-e181e70bffd5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d6372c7-26c3-452c-a48f-e181e70bffd5-istiod-ca-cert\") pod \"9d6372c7-26c3-452c-a48f-e181e70bffd5\" (UID: \"9d6372c7-26c3-452c-a48f-e181e70bffd5\") " Oct 31 13:10:06.640906 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:06.640891 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d6372c7-26c3-452c-a48f-e181e70bffd5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d6372c7-26c3-452c-a48f-e181e70bffd5-workload-certs\") pod \"9d6372c7-26c3-452c-a48f-e181e70bffd5\" (UID: \"9d6372c7-26c3-452c-a48f-e181e70bffd5\") " Oct 31 13:10:06.641689 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:06.640441 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d6372c7-26c3-452c-a48f-e181e70bffd5-workload-socket" pod "9d6372c7-26c3-452c-a48f-e181e70bffd5" (UID: "9d6372c7-26c3-452c-a48f-e181e70bffd5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:10:06.641911 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:06.640748 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d6372c7-26c3-452c-a48f-e181e70bffd5-credential-socket" pod "9d6372c7-26c3-452c-a48f-e181e70bffd5" (UID: "9d6372c7-26c3-452c-a48f-e181e70bffd5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:10:06.642041 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:06.641281 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d6372c7-26c3-452c-a48f-e181e70bffd5-istio-data" pod "9d6372c7-26c3-452c-a48f-e181e70bffd5" (UID: "9d6372c7-26c3-452c-a48f-e181e70bffd5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:10:06.642162 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:06.642000 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6372c7-26c3-452c-a48f-e181e70bffd5-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:10:06.642162 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:06.641525 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d6372c7-26c3-452c-a48f-e181e70bffd5-workload-certs" pod "9d6372c7-26c3-452c-a48f-e181e70bffd5" (UID: "9d6372c7-26c3-452c-a48f-e181e70bffd5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:10:06.642162 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:06.641605 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d6372c7-26c3-452c-a48f-e181e70bffd5-istiod-ca-cert" pod "9d6372c7-26c3-452c-a48f-e181e70bffd5" (UID: "9d6372c7-26c3-452c-a48f-e181e70bffd5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:10:06.643444 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:06.643402 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d6372c7-26c3-452c-a48f-e181e70bffd5-kube-api-access-d4nxc" pod "9d6372c7-26c3-452c-a48f-e181e70bffd5" (UID: "9d6372c7-26c3-452c-a48f-e181e70bffd5"). InnerVolumeSpecName "kube-api-access-d4nxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:10:06.643910 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:06.643884 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d6372c7-26c3-452c-a48f-e181e70bffd5-istio-envoy" pod "9d6372c7-26c3-452c-a48f-e181e70bffd5" (UID: "9d6372c7-26c3-452c-a48f-e181e70bffd5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:10:06.647040 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:06.647014 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d6372c7-26c3-452c-a48f-e181e70bffd5-istio-token" pod "9d6372c7-26c3-452c-a48f-e181e70bffd5" (UID: "9d6372c7-26c3-452c-a48f-e181e70bffd5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:10:06.647296 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:06.647269 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d6372c7-26c3-452c-a48f-e181e70bffd5-istio-podinfo" pod "9d6372c7-26c3-452c-a48f-e181e70bffd5" (UID: "9d6372c7-26c3-452c-a48f-e181e70bffd5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:10:06.743024 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:06.742991 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d6372c7-26c3-452c-a48f-e181e70bffd5-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:10:06.743024 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:06.743017 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6372c7-26c3-452c-a48f-e181e70bffd5-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:10:06.743024 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:06.743027 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d6372c7-26c3-452c-a48f-e181e70bffd5-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:10:06.743024 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:06.743035 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d6372c7-26c3-452c-a48f-e181e70bffd5-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:10:06.743398 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:06.743044 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d6372c7-26c3-452c-a48f-e181e70bffd5-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:10:06.743398 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:06.743055 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d6372c7-26c3-452c-a48f-e181e70bffd5-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:10:06.743398 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:06.743064 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d6372c7-26c3-452c-a48f-e181e70bffd5-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:10:06.743398 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:06.743072 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4nxc\" (UniqueName: \"kubernetes.io/projected/9d6372c7-26c3-452c-a48f-e181e70bffd5-kube-api-access-d4nxc\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:10:07.239113 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:07.239078 5693 generic.go:356] "Generic (PLEG): container finished" podID="9d6372c7-26c3-452c-a48f-e181e70bffd5" containerID="53e222b54aa010658e9101021aa03e18b7f698153b53d8f2448c5144bf2492a9" exitCode=0 Oct 31 13:10:07.239336 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:07.239141 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" event={"ID":"9d6372c7-26c3-452c-a48f-e181e70bffd5","Type":"ContainerDied","Data":"53e222b54aa010658e9101021aa03e18b7f698153b53d8f2448c5144bf2492a9"} Oct 31 13:10:07.239336 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:07.239144 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" Oct 31 13:10:07.239336 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:07.239231 5693 scope.go:122] "RemoveContainer" containerID="53e222b54aa010658e9101021aa03e18b7f698153b53d8f2448c5144bf2492a9" Oct 31 13:10:07.239336 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:07.239218 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98" event={"ID":"9d6372c7-26c3-452c-a48f-e181e70bffd5","Type":"ContainerDied","Data":"617e23a0294845976b542a7f03bb10b6f42e36880f476a63654e31102140ef13"} Oct 31 13:10:07.252062 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:07.252032 5693 scope.go:122] "RemoveContainer" containerID="53e222b54aa010658e9101021aa03e18b7f698153b53d8f2448c5144bf2492a9" Oct 31 13:10:07.252420 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:10:07.252394 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53e222b54aa010658e9101021aa03e18b7f698153b53d8f2448c5144bf2492a9\": container with ID starting with 53e222b54aa010658e9101021aa03e18b7f698153b53d8f2448c5144bf2492a9 not found: ID does not exist" containerID="53e222b54aa010658e9101021aa03e18b7f698153b53d8f2448c5144bf2492a9" Oct 31 13:10:07.252522 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:07.252426 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53e222b54aa010658e9101021aa03e18b7f698153b53d8f2448c5144bf2492a9"} err="failed to get container status \"53e222b54aa010658e9101021aa03e18b7f698153b53d8f2448c5144bf2492a9\": rpc error: code = NotFound desc = could not find container \"53e222b54aa010658e9101021aa03e18b7f698153b53d8f2448c5144bf2492a9\": container with ID starting with 53e222b54aa010658e9101021aa03e18b7f698153b53d8f2448c5144bf2492a9 not found: ID does not exist" Oct 31 13:10:07.266599 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:07.266569 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98"] Oct 31 13:10:07.272240 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:07.272214 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-95dfcc-openshift-default-9764dfdb5-8dg98"] Oct 31 13:10:07.411600 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:07.411559 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d6372c7-26c3-452c-a48f-e181e70bffd5" path="/var/lib/kubelet/pods/9d6372c7-26c3-452c-a48f-e181e70bffd5/volumes" Oct 31 13:10:09.256138 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:09.256103 5693 generic.go:356] "Generic (PLEG): container finished" podID="2ae02ff0-1e33-4007-a97a-63a61e8b4bf4" containerID="98ad7ee33e380455fc042048fdba1e1b9f1c9e0ff0aa2f2ee06f8385255781d0" exitCode=0 Oct 31 13:10:09.256485 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:09.256156 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" event={"ID":"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4","Type":"ContainerDied","Data":"98ad7ee33e380455fc042048fdba1e1b9f1c9e0ff0aa2f2ee06f8385255781d0"} Oct 31 13:10:09.488858 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:09.488096 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" Oct 31 13:10:09.599272 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:09.599232 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-istiod-ca-cert\") pod \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\" (UID: \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\") " Oct 31 13:10:09.599458 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:09.599297 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-kube-api-access-5vjvb\" (UniqueName: \"kubernetes.io/projected/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-kube-api-access-5vjvb\") pod \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\" (UID: \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\") " Oct 31 13:10:09.599458 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:09.599337 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-istio-token\" (UniqueName: \"kubernetes.io/projected/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-istio-token\") pod \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\" (UID: \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\") " Oct 31 13:10:09.599458 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:09.599404 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-istio-data\") pod \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\" (UID: \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\") " Oct 31 13:10:09.599572 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:09.599515 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-workload-socket\") pod \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\" (UID: \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\") " Oct 31 13:10:09.599619 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:09.599587 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-credential-socket\") pod \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\" (UID: \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\") " Oct 31 13:10:09.599695 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:09.599656 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-istio-podinfo\") pod \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\" (UID: \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\") " Oct 31 13:10:09.599756 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:09.599701 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-istiod-ca-cert" pod "2ae02ff0-1e33-4007-a97a-63a61e8b4bf4" (UID: "2ae02ff0-1e33-4007-a97a-63a61e8b4bf4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:10:09.599756 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:09.599720 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-istio-data" pod "2ae02ff0-1e33-4007-a97a-63a61e8b4bf4" (UID: "2ae02ff0-1e33-4007-a97a-63a61e8b4bf4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:10:09.599882 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:09.599738 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-workload-certs\") pod \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\" (UID: \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\") " Oct 31 13:10:09.599882 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:09.599803 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-workload-socket" pod "2ae02ff0-1e33-4007-a97a-63a61e8b4bf4" (UID: "2ae02ff0-1e33-4007-a97a-63a61e8b4bf4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:10:09.599882 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:09.599828 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-istio-envoy\") pod \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\" (UID: \"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4\") " Oct 31 13:10:09.600033 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:09.599951 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-credential-socket" pod "2ae02ff0-1e33-4007-a97a-63a61e8b4bf4" (UID: "2ae02ff0-1e33-4007-a97a-63a61e8b4bf4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:10:09.600033 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:09.599983 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-workload-certs" pod "2ae02ff0-1e33-4007-a97a-63a61e8b4bf4" (UID: "2ae02ff0-1e33-4007-a97a-63a61e8b4bf4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:10:09.600761 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:09.600735 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:10:09.600905 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:09.600764 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:10:09.600905 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:09.600778 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:10:09.600905 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:09.600791 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:10:09.600905 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:09.600805 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:10:09.601865 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:09.601836 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-istio-token" pod "2ae02ff0-1e33-4007-a97a-63a61e8b4bf4" (UID: "2ae02ff0-1e33-4007-a97a-63a61e8b4bf4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:10:09.602022 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:09.602005 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-istio-podinfo" pod "2ae02ff0-1e33-4007-a97a-63a61e8b4bf4" (UID: "2ae02ff0-1e33-4007-a97a-63a61e8b4bf4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:10:09.602022 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:09.602007 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-kube-api-access-5vjvb" pod "2ae02ff0-1e33-4007-a97a-63a61e8b4bf4" (UID: "2ae02ff0-1e33-4007-a97a-63a61e8b4bf4"). InnerVolumeSpecName "kube-api-access-5vjvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:10:09.602274 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:09.602252 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-istio-envoy" pod "2ae02ff0-1e33-4007-a97a-63a61e8b4bf4" (UID: "2ae02ff0-1e33-4007-a97a-63a61e8b4bf4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:10:09.702449 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:09.702354 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:10:09.702449 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:09.702399 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vjvb\" (UniqueName: \"kubernetes.io/projected/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-kube-api-access-5vjvb\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:10:09.702449 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:09.702409 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:10:09.702449 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:09.702418 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:10:10.299611 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:10.299542 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" Oct 31 13:10:10.299611 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:10.299556 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv" event={"ID":"2ae02ff0-1e33-4007-a97a-63a61e8b4bf4","Type":"ContainerDied","Data":"d4d44ad53932b3b1f32b24881998a24fa335ca0dcd161bcade3ba935f1e36732"} Oct 31 13:10:10.299611 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:10.299615 5693 scope.go:122] "RemoveContainer" containerID="98ad7ee33e380455fc042048fdba1e1b9f1c9e0ff0aa2f2ee06f8385255781d0" Oct 31 13:10:10.327036 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:10.326991 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv"] Oct 31 13:10:10.334015 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:10.333984 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-rskhv"] Oct 31 13:10:11.410945 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:11.410908 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ae02ff0-1e33-4007-a97a-63a61e8b4bf4" path="/var/lib/kubelet/pods/2ae02ff0-1e33-4007-a97a-63a61e8b4bf4/volumes" Oct 31 13:10:18.706777 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:10:18.706735 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:10:48.691138 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:10:48.691091 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:10:54.945175 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:54.945128 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0612-b5b66c795-fdmgl"] Oct 31 13:10:54.945846 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:54.945830 5693 cpu_manager.go:414] "containerMap: removing container" podUID="2ae02ff0-1e33-4007-a97a-63a61e8b4bf4" containerName="istio-proxy" Oct 31 13:10:54.945896 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:54.945848 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ae02ff0-1e33-4007-a97a-63a61e8b4bf4" containerName="istio-proxy" Oct 31 13:10:54.945896 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:54.945859 5693 cpu_manager.go:414] "containerMap: removing container" podUID="9d6372c7-26c3-452c-a48f-e181e70bffd5" containerName="istio-proxy" Oct 31 13:10:54.945896 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:54.945864 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d6372c7-26c3-452c-a48f-e181e70bffd5" containerName="istio-proxy" Oct 31 13:10:54.946042 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:54.945974 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ae02ff0-1e33-4007-a97a-63a61e8b4bf4" containerName="istio-proxy" Oct 31 13:10:54.946042 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:54.945981 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d6372c7-26c3-452c-a48f-e181e70bffd5" containerName="istio-proxy" Oct 31 13:10:54.949711 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:54.949687 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0612-b5b66c795-fdmgl" Oct 31 13:10:54.953986 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:54.953962 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-6c0612-85c484d865-fvthd"] Oct 31 13:10:54.963473 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:54.963454 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0612-b5b66c795-fdmgl"] Oct 31 13:10:54.963624 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:54.963554 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c0612-85c484d865-fvthd" Oct 31 13:10:54.979475 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:54.979432 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-6c0612-85c484d865-fvthd"] Oct 31 13:10:54.990732 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:54.979593 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-6c0612\"" Oct 31 13:10:54.991532 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:54.991507 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32019af4-f819-4127-9d00-d2582affe985-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-6c0612-85c484d865-fvthd\" (UID: \"32019af4-f819-4127-9d00-d2582affe985\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c0612-85c484d865-fvthd" Oct 31 13:10:54.991765 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:54.991741 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf84d\" (UniqueName: \"kubernetes.io/projected/32019af4-f819-4127-9d00-d2582affe985-kube-api-access-zf84d\") pod \"payload-processing-e2e-shared-a-w4-6c0612-85c484d865-fvthd\" (UID: \"32019af4-f819-4127-9d00-d2582affe985\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c0612-85c484d865-fvthd" Oct 31 13:10:54.991826 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:54.991800 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcplm\" (UniqueName: \"kubernetes.io/projected/fba7a7df-1586-461c-b3d7-63ddcf53f218-kube-api-access-pcplm\") pod \"payload-pre-processing-e2e-shared-a-w4-6c0612-b5b66c795-fdmgl\" (UID: \"fba7a7df-1586-461c-b3d7-63ddcf53f218\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0612-b5b66c795-fdmgl" Oct 31 13:10:54.991973 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:54.991957 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fba7a7df-1586-461c-b3d7-63ddcf53f218-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-6c0612-b5b66c795-fdmgl\" (UID: \"fba7a7df-1586-461c-b3d7-63ddcf53f218\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0612-b5b66c795-fdmgl" Oct 31 13:10:55.014689 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:55.014609 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-6c0612-dockercfg-22pbp\"" Oct 31 13:10:55.097665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:55.096634 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fba7a7df-1586-461c-b3d7-63ddcf53f218-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-6c0612-b5b66c795-fdmgl\" (UID: \"fba7a7df-1586-461c-b3d7-63ddcf53f218\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0612-b5b66c795-fdmgl" Oct 31 13:10:55.097665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:55.097596 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fba7a7df-1586-461c-b3d7-63ddcf53f218-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-6c0612-b5b66c795-fdmgl\" (UID: \"fba7a7df-1586-461c-b3d7-63ddcf53f218\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0612-b5b66c795-fdmgl" Oct 31 13:10:55.101716 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:55.098317 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32019af4-f819-4127-9d00-d2582affe985-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-6c0612-85c484d865-fvthd\" (UID: \"32019af4-f819-4127-9d00-d2582affe985\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c0612-85c484d865-fvthd" Oct 31 13:10:55.101716 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:55.098441 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf84d\" (UniqueName: \"kubernetes.io/projected/32019af4-f819-4127-9d00-d2582affe985-kube-api-access-zf84d\") pod \"payload-processing-e2e-shared-a-w4-6c0612-85c484d865-fvthd\" (UID: \"32019af4-f819-4127-9d00-d2582affe985\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c0612-85c484d865-fvthd" Oct 31 13:10:55.101716 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:55.098476 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcplm\" (UniqueName: \"kubernetes.io/projected/fba7a7df-1586-461c-b3d7-63ddcf53f218-kube-api-access-pcplm\") pod \"payload-pre-processing-e2e-shared-a-w4-6c0612-b5b66c795-fdmgl\" (UID: \"fba7a7df-1586-461c-b3d7-63ddcf53f218\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0612-b5b66c795-fdmgl" Oct 31 13:10:55.120663 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:55.119709 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32019af4-f819-4127-9d00-d2582affe985-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-6c0612-85c484d865-fvthd\" (UID: \"32019af4-f819-4127-9d00-d2582affe985\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c0612-85c484d865-fvthd" Oct 31 13:10:55.129664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:55.127702 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcplm\" (UniqueName: \"kubernetes.io/projected/fba7a7df-1586-461c-b3d7-63ddcf53f218-kube-api-access-pcplm\") pod \"payload-pre-processing-e2e-shared-a-w4-6c0612-b5b66c795-fdmgl\" (UID: \"fba7a7df-1586-461c-b3d7-63ddcf53f218\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0612-b5b66c795-fdmgl" Oct 31 13:10:55.149489 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:55.149449 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf84d\" (UniqueName: \"kubernetes.io/projected/32019af4-f819-4127-9d00-d2582affe985-kube-api-access-zf84d\") pod \"payload-processing-e2e-shared-a-w4-6c0612-85c484d865-fvthd\" (UID: \"32019af4-f819-4127-9d00-d2582affe985\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c0612-85c484d865-fvthd" Oct 31 13:10:55.267953 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:55.267870 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0612-b5b66c795-fdmgl" Oct 31 13:10:55.294410 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:55.294379 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c0612-85c484d865-fvthd" Oct 31 13:10:55.575449 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:55.575425 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0612-b5b66c795-fdmgl"] Oct 31 13:10:55.578695 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:10:55.578630 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba7a7df_1586_461c_b3d7_63ddcf53f218.slice/crio-7cdcd77277bb1a3af661aa46fed8786b2fbfe401ce0dcd1e755bb247e5a5aa86 WatchSource:0}: Error finding container 7cdcd77277bb1a3af661aa46fed8786b2fbfe401ce0dcd1e755bb247e5a5aa86: Status 404 returned error can't find the container with id 7cdcd77277bb1a3af661aa46fed8786b2fbfe401ce0dcd1e755bb247e5a5aa86 Oct 31 13:10:55.632868 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:55.632828 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-6c0612-85c484d865-fvthd"] Oct 31 13:10:55.633118 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:10:55.633074 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32019af4_f819_4127_9d00_d2582affe985.slice/crio-98054c763cf9d482e1dad2ccd9567bb938c09205da9135527da8d7977f075bed WatchSource:0}: Error finding container 98054c763cf9d482e1dad2ccd9567bb938c09205da9135527da8d7977f075bed: Status 404 returned error can't find the container with id 98054c763cf9d482e1dad2ccd9567bb938c09205da9135527da8d7977f075bed Oct 31 13:10:55.634807 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:55.634776 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0612-b5b66c795-fdmgl" event={"ID":"fba7a7df-1586-461c-b3d7-63ddcf53f218","Type":"ContainerStarted","Data":"7cdcd77277bb1a3af661aa46fed8786b2fbfe401ce0dcd1e755bb247e5a5aa86"} Oct 31 13:10:56.643093 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:56.643055 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c0612-85c484d865-fvthd" event={"ID":"32019af4-f819-4127-9d00-d2582affe985","Type":"ContainerStarted","Data":"e91931e26bf74cfd83aa2fb254db66fa562192155e76391641fe4672e4188f00"} Oct 31 13:10:56.643544 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:56.643106 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c0612-85c484d865-fvthd" event={"ID":"32019af4-f819-4127-9d00-d2582affe985","Type":"ContainerStarted","Data":"98054c763cf9d482e1dad2ccd9567bb938c09205da9135527da8d7977f075bed"} Oct 31 13:10:56.643544 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:56.643145 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c0612-85c484d865-fvthd" Oct 31 13:10:56.644822 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:56.644796 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0612-b5b66c795-fdmgl" event={"ID":"fba7a7df-1586-461c-b3d7-63ddcf53f218","Type":"ContainerStarted","Data":"5a08050267cf453575846e766b0837b98554f1abbfe01bab12d04621c8f9d003"} Oct 31 13:10:56.644957 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:56.644926 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0612-b5b66c795-fdmgl" Oct 31 13:10:56.666549 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:56.666506 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c0612-85c484d865-fvthd" podStartSLOduration=2.666494315 podStartE2EDuration="2.666494315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:10:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:56.66217636 +0000 UTC m=+2307.369762666" watchObservedRunningTime="2026-10-31 13:10:56.666494315 +0000 UTC m=+2307.374080602" Oct 31 13:10:56.683293 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:10:56.683253 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0612-b5b66c795-fdmgl" podStartSLOduration=2.683241545 podStartE2EDuration="2.683241545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:10:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:56.681439779 +0000 UTC m=+2307.389026075" watchObservedRunningTime="2026-10-31 13:10:56.683241545 +0000 UTC m=+2307.390827840" Oct 31 13:11:07.654294 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:07.654265 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0612-b5b66c795-fdmgl" Oct 31 13:11:07.654753 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:07.654336 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c0612-85c484d865-fvthd" Oct 31 13:11:17.526495 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:17.526462 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6"] Oct 31 13:11:17.530885 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:17.530861 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" Oct 31 13:11:17.534707 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:17.534675 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-4574ec-openshift-default-dockercfg-kg8h4\"" Oct 31 13:11:17.544058 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:17.544022 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6"] Oct 31 13:11:17.662391 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:17.662354 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-istiod-ca-cert\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6\" (UID: \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" Oct 31 13:11:17.662391 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:17.662395 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-workload-certs\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6\" (UID: \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" Oct 31 13:11:17.662608 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:17.662549 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxsbc\" (UniqueName: \"kubernetes.io/projected/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-kube-api-access-sxsbc\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6\" (UID: \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" Oct 31 13:11:17.662608 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:17.662580 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-istio-data\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6\" (UID: \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" Oct 31 13:11:17.662778 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:17.662612 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-workload-socket\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6\" (UID: \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" Oct 31 13:11:17.662778 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:17.662671 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-istio-envoy\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6\" (UID: \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" Oct 31 13:11:17.662874 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:17.662765 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-istio-token\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6\" (UID: \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" Oct 31 13:11:17.662936 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:17.662917 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-credential-socket\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6\" (UID: \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" Oct 31 13:11:17.663001 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:17.662988 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-istio-podinfo\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6\" (UID: \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" Oct 31 13:11:17.764502 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:17.764465 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-workload-certs\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6\" (UID: \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" Oct 31 13:11:17.764754 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:17.764617 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxsbc\" (UniqueName: \"kubernetes.io/projected/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-kube-api-access-sxsbc\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6\" (UID: \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" Oct 31 13:11:17.764754 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:17.764663 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-istio-data\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6\" (UID: \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" Oct 31 13:11:17.764754 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:17.764705 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-workload-socket\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6\" (UID: \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" Oct 31 13:11:17.764970 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:17.764754 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-istio-envoy\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6\" (UID: \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" Oct 31 13:11:17.764970 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:17.764781 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-istio-token\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6\" (UID: \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" Oct 31 13:11:17.764970 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:17.764880 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-workload-certs\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6\" (UID: \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" Oct 31 13:11:17.764970 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:17.764888 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-credential-socket\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6\" (UID: \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" Oct 31 13:11:17.764970 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:17.764958 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-istio-podinfo\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6\" (UID: \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" Oct 31 13:11:17.765260 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:17.765010 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-istiod-ca-cert\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6\" (UID: \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" Oct 31 13:11:17.765260 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:17.765040 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-istio-data\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6\" (UID: \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" Oct 31 13:11:17.765260 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:17.765195 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-credential-socket\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6\" (UID: \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" Oct 31 13:11:17.765435 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:17.765408 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-workload-socket\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6\" (UID: \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" Oct 31 13:11:17.765797 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:17.765774 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-istiod-ca-cert\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6\" (UID: \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" Oct 31 13:11:17.767679 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:17.767617 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-istio-envoy\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6\" (UID: \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" Oct 31 13:11:17.767789 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:17.767633 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-istio-podinfo\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6\" (UID: \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" Oct 31 13:11:17.778299 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:17.778244 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-istio-token\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6\" (UID: \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" Oct 31 13:11:17.778430 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:17.778412 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxsbc\" (UniqueName: \"kubernetes.io/projected/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-kube-api-access-sxsbc\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6\" (UID: \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" Oct 31 13:11:17.849425 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:17.848726 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" Oct 31 13:11:18.155764 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:18.155734 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6"] Oct 31 13:11:18.156787 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:11:18.156762 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8afd9c8e_49db_4c62_8af3_4fbed8d2d954.slice/crio-b7df648282208532807f8099472f86141e695ab55b5d0d35c4c318806c14dbaa WatchSource:0}: Error finding container b7df648282208532807f8099472f86141e695ab55b5d0d35c4c318806c14dbaa: Status 404 returned error can't find the container with id b7df648282208532807f8099472f86141e695ab55b5d0d35c4c318806c14dbaa Oct 31 13:11:18.159747 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:18.159713 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:11:18.159838 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:18.159803 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:11:18.159838 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:18.159833 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:11:18.690437 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:11:18.690401 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:11:18.866028 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:18.865983 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" event={"ID":"8afd9c8e-49db-4c62-8af3-4fbed8d2d954","Type":"ContainerStarted","Data":"2dd0d980673356e52c7665d3ff54c25de45a867171a0ae907a1a844f8a6b6445"} Oct 31 13:11:18.866355 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:18.866058 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" event={"ID":"8afd9c8e-49db-4c62-8af3-4fbed8d2d954","Type":"ContainerStarted","Data":"b7df648282208532807f8099472f86141e695ab55b5d0d35c4c318806c14dbaa"} Oct 31 13:11:18.907862 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:18.907783 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" podStartSLOduration=1.907750975 podStartE2EDuration="1.907750975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:11:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:11:18.901864166 +0000 UTC m=+2329.609450462" watchObservedRunningTime="2026-10-31 13:11:18.907750975 +0000 UTC m=+2329.615337290" Oct 31 13:11:19.849128 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:19.849094 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" Oct 31 13:11:19.850353 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:19.850327 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 31 13:11:19.850485 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:19.850373 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" podUID="8afd9c8e-49db-4c62-8af3-4fbed8d2d954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 31 13:11:20.849135 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:20.849106 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 31 13:11:20.849516 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:20.849175 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" podUID="8afd9c8e-49db-4c62-8af3-4fbed8d2d954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 31 13:11:21.850072 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:21.850029 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 31 13:11:21.850572 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:21.850104 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" podUID="8afd9c8e-49db-4c62-8af3-4fbed8d2d954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 31 13:11:22.850130 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:22.850085 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 31 13:11:22.850708 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:22.850151 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" podUID="8afd9c8e-49db-4c62-8af3-4fbed8d2d954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 31 13:11:23.849304 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:23.849264 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 31 13:11:23.849478 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:23.849326 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" podUID="8afd9c8e-49db-4c62-8af3-4fbed8d2d954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 31 13:11:25.045014 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:25.044974 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" Oct 31 13:11:25.045820 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:25.045798 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" Oct 31 13:11:25.047012 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:25.046989 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" Oct 31 13:11:25.528419 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:25.528388 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-1e72f6-ddbd6777-k2fm6"] Oct 31 13:11:25.533069 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:25.533041 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1e72f6-ddbd6777-k2fm6" Oct 31 13:11:25.538147 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:25.538125 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-1e72f6\"" Oct 31 13:11:25.542576 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:25.542554 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-1e72f6-ddbd6777-k2fm6"] Oct 31 13:11:25.565691 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:25.565662 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1e72f6-64d95f46cf-wxgkn"] Oct 31 13:11:25.570281 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:25.570261 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1e72f6-64d95f46cf-wxgkn" Oct 31 13:11:25.578790 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:25.578676 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-1e72f6-dockercfg-nt4fw\"" Oct 31 13:11:25.583559 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:25.583510 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1e72f6-64d95f46cf-wxgkn"] Oct 31 13:11:25.596699 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:25.596664 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8f9a320-4770-43a7-9297-b1d659961376-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-1e72f6-ddbd6777-k2fm6\" (UID: \"f8f9a320-4770-43a7-9297-b1d659961376\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1e72f6-ddbd6777-k2fm6" Oct 31 13:11:25.596864 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:25.596803 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vk66\" (UniqueName: \"kubernetes.io/projected/f8f9a320-4770-43a7-9297-b1d659961376-kube-api-access-2vk66\") pod \"payload-processing-e2e-shared-b-w4-1e72f6-ddbd6777-k2fm6\" (UID: \"f8f9a320-4770-43a7-9297-b1d659961376\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1e72f6-ddbd6777-k2fm6" Oct 31 13:11:25.708009 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:25.700033 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6ff15bc-cbaa-4bbb-a634-415dc98bbb12-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-1e72f6-64d95f46cf-wxgkn\" (UID: \"c6ff15bc-cbaa-4bbb-a634-415dc98bbb12\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1e72f6-64d95f46cf-wxgkn" Oct 31 13:11:25.708009 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:25.700087 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7pmv\" (UniqueName: \"kubernetes.io/projected/c6ff15bc-cbaa-4bbb-a634-415dc98bbb12-kube-api-access-n7pmv\") pod \"payload-pre-processing-e2e-shared-b-w4-1e72f6-64d95f46cf-wxgkn\" (UID: \"c6ff15bc-cbaa-4bbb-a634-415dc98bbb12\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1e72f6-64d95f46cf-wxgkn" Oct 31 13:11:25.708009 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:25.700291 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8f9a320-4770-43a7-9297-b1d659961376-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-1e72f6-ddbd6777-k2fm6\" (UID: \"f8f9a320-4770-43a7-9297-b1d659961376\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1e72f6-ddbd6777-k2fm6" Oct 31 13:11:25.708009 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:25.700375 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vk66\" (UniqueName: \"kubernetes.io/projected/f8f9a320-4770-43a7-9297-b1d659961376-kube-api-access-2vk66\") pod \"payload-processing-e2e-shared-b-w4-1e72f6-ddbd6777-k2fm6\" (UID: \"f8f9a320-4770-43a7-9297-b1d659961376\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1e72f6-ddbd6777-k2fm6" Oct 31 13:11:25.708009 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:25.702605 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8f9a320-4770-43a7-9297-b1d659961376-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-1e72f6-ddbd6777-k2fm6\" (UID: \"f8f9a320-4770-43a7-9297-b1d659961376\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1e72f6-ddbd6777-k2fm6" Oct 31 13:11:25.771220 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:25.769468 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vk66\" (UniqueName: \"kubernetes.io/projected/f8f9a320-4770-43a7-9297-b1d659961376-kube-api-access-2vk66\") pod \"payload-processing-e2e-shared-b-w4-1e72f6-ddbd6777-k2fm6\" (UID: \"f8f9a320-4770-43a7-9297-b1d659961376\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1e72f6-ddbd6777-k2fm6" Oct 31 13:11:25.817722 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:25.810955 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6ff15bc-cbaa-4bbb-a634-415dc98bbb12-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-1e72f6-64d95f46cf-wxgkn\" (UID: \"c6ff15bc-cbaa-4bbb-a634-415dc98bbb12\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1e72f6-64d95f46cf-wxgkn" Oct 31 13:11:25.817722 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:25.811005 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7pmv\" (UniqueName: \"kubernetes.io/projected/c6ff15bc-cbaa-4bbb-a634-415dc98bbb12-kube-api-access-n7pmv\") pod \"payload-pre-processing-e2e-shared-b-w4-1e72f6-64d95f46cf-wxgkn\" (UID: \"c6ff15bc-cbaa-4bbb-a634-415dc98bbb12\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1e72f6-64d95f46cf-wxgkn" Oct 31 13:11:25.817722 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:25.812671 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6ff15bc-cbaa-4bbb-a634-415dc98bbb12-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-1e72f6-64d95f46cf-wxgkn\" (UID: \"c6ff15bc-cbaa-4bbb-a634-415dc98bbb12\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1e72f6-64d95f46cf-wxgkn" Oct 31 13:11:25.837677 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:25.832871 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7pmv\" (UniqueName: \"kubernetes.io/projected/c6ff15bc-cbaa-4bbb-a634-415dc98bbb12-kube-api-access-n7pmv\") pod \"payload-pre-processing-e2e-shared-b-w4-1e72f6-64d95f46cf-wxgkn\" (UID: \"c6ff15bc-cbaa-4bbb-a634-415dc98bbb12\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1e72f6-64d95f46cf-wxgkn" Oct 31 13:11:25.859271 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:25.857964 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1e72f6-ddbd6777-k2fm6" Oct 31 13:11:25.889952 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:25.889507 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1e72f6-64d95f46cf-wxgkn" Oct 31 13:11:26.161309 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:26.161281 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-1e72f6-ddbd6777-k2fm6"] Oct 31 13:11:26.161854 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:11:26.161832 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8f9a320_4770_43a7_9297_b1d659961376.slice/crio-57c61d8a664f9371df453084cd870f8d46f6e5a7428b5b0c15ca7b817d797cd9 WatchSource:0}: Error finding container 57c61d8a664f9371df453084cd870f8d46f6e5a7428b5b0c15ca7b817d797cd9: Status 404 returned error can't find the container with id 57c61d8a664f9371df453084cd870f8d46f6e5a7428b5b0c15ca7b817d797cd9 Oct 31 13:11:26.209941 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:26.209912 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1e72f6-64d95f46cf-wxgkn"] Oct 31 13:11:26.210536 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:11:26.210509 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6ff15bc_cbaa_4bbb_a634_415dc98bbb12.slice/crio-f2778342ddc724b66601353b7da97d4baa79138d70cafd34ff841c8b6d2c2bfc WatchSource:0}: Error finding container f2778342ddc724b66601353b7da97d4baa79138d70cafd34ff841c8b6d2c2bfc: Status 404 returned error can't find the container with id f2778342ddc724b66601353b7da97d4baa79138d70cafd34ff841c8b6d2c2bfc Oct 31 13:11:27.086631 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:27.086595 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1e72f6-ddbd6777-k2fm6" event={"ID":"f8f9a320-4770-43a7-9297-b1d659961376","Type":"ContainerStarted","Data":"5fb9ced1279d5809ac9f354b12bc23f7f1af6721b0212c96fa6fd54a4d8ba989"} Oct 31 13:11:27.090324 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:27.088513 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1e72f6-ddbd6777-k2fm6" Oct 31 13:11:27.090324 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:27.088554 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1e72f6-ddbd6777-k2fm6" event={"ID":"f8f9a320-4770-43a7-9297-b1d659961376","Type":"ContainerStarted","Data":"57c61d8a664f9371df453084cd870f8d46f6e5a7428b5b0c15ca7b817d797cd9"} Oct 31 13:11:27.093960 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:27.093926 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1e72f6-64d95f46cf-wxgkn" event={"ID":"c6ff15bc-cbaa-4bbb-a634-415dc98bbb12","Type":"ContainerStarted","Data":"b70b0035e6482367f73184c3110ea7e48ee0a538ad1899f3b4083988f550040b"} Oct 31 13:11:27.094125 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:27.093991 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1e72f6-64d95f46cf-wxgkn" event={"ID":"c6ff15bc-cbaa-4bbb-a634-415dc98bbb12","Type":"ContainerStarted","Data":"f2778342ddc724b66601353b7da97d4baa79138d70cafd34ff841c8b6d2c2bfc"} Oct 31 13:11:27.094173 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:27.094120 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1e72f6-64d95f46cf-wxgkn" Oct 31 13:11:27.112566 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:27.112516 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1e72f6-ddbd6777-k2fm6" podStartSLOduration=2.112504581 podStartE2EDuration="2.112504581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:11:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:11:27.109457812 +0000 UTC m=+2337.817044110" watchObservedRunningTime="2026-10-31 13:11:27.112504581 +0000 UTC m=+2337.820090876" Oct 31 13:11:30.330284 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:30.330245 5693 scope.go:122] "RemoveContainer" containerID="36ab8df3c04988a7464ac23f6b6b0eca69c10361abc7c5ea22ec9510151ffce6" Oct 31 13:11:38.104714 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:38.104680 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1e72f6-ddbd6777-k2fm6" Oct 31 13:11:38.105213 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:38.104885 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1e72f6-64d95f46cf-wxgkn" Oct 31 13:11:38.128019 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:38.127962 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1e72f6-64d95f46cf-wxgkn" podStartSLOduration=13.127948731 podStartE2EDuration="13.127948731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:11:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:11:27.13194241 +0000 UTC m=+2337.839528707" watchObservedRunningTime="2026-10-31 13:11:38.127948731 +0000 UTC m=+2348.835535026" Oct 31 13:11:48.689415 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:11:48.689333 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:11:55.505465 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:55.505427 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11397-85e626"] Oct 31 13:11:55.510767 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:55.510741 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11397-85e626" Oct 31 13:11:55.518342 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:55.518310 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11397-85e626"] Oct 31 13:11:55.524041 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:55.524018 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mn5fx\"" Oct 31 13:11:55.588096 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:55.588058 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjksr\" (UniqueName: \"kubernetes.io/projected/5d3b7f10-6969-447c-be69-8fff9c87155a-kube-api-access-jjksr\") pod \"test-curl-11397-85e626\" (UID: \"5d3b7f10-6969-447c-be69-8fff9c87155a\") " pod="opendatahub/test-curl-11397-85e626" Oct 31 13:11:55.689795 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:55.689754 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjksr\" (UniqueName: \"kubernetes.io/projected/5d3b7f10-6969-447c-be69-8fff9c87155a-kube-api-access-jjksr\") pod \"test-curl-11397-85e626\" (UID: \"5d3b7f10-6969-447c-be69-8fff9c87155a\") " pod="opendatahub/test-curl-11397-85e626" Oct 31 13:11:55.705203 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:55.705173 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjksr\" (UniqueName: \"kubernetes.io/projected/5d3b7f10-6969-447c-be69-8fff9c87155a-kube-api-access-jjksr\") pod \"test-curl-11397-85e626\" (UID: \"5d3b7f10-6969-447c-be69-8fff9c87155a\") " pod="opendatahub/test-curl-11397-85e626" Oct 31 13:11:55.831577 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:55.831487 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11397-85e626" Oct 31 13:11:56.145426 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:56.143010 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11397-85e626"] Oct 31 13:11:56.145426 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:11:56.144005 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d3b7f10_6969_447c_be69_8fff9c87155a.slice/crio-cfe63d22b6ec9d179dca7a473a01e3adf27c28b6de5afc00fe703bc0f324a81b WatchSource:0}: Error finding container cfe63d22b6ec9d179dca7a473a01e3adf27c28b6de5afc00fe703bc0f324a81b: Status 404 returned error can't find the container with id cfe63d22b6ec9d179dca7a473a01e3adf27c28b6de5afc00fe703bc0f324a81b Oct 31 13:11:56.147488 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:56.146781 5693 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 13:11:56.428867 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:56.428828 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11397-85e626" event={"ID":"5d3b7f10-6969-447c-be69-8fff9c87155a","Type":"ContainerStarted","Data":"cfe63d22b6ec9d179dca7a473a01e3adf27c28b6de5afc00fe703bc0f324a81b"} Oct 31 13:11:59.453364 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:59.453329 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11397-85e626" event={"ID":"5d3b7f10-6969-447c-be69-8fff9c87155a","Type":"ContainerStarted","Data":"2bcd23c32594561513aeaff89a062d2c73561b3e731fa8137fb5a088377c41da"} Oct 31 13:11:59.470787 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:11:59.470740 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11397-85e626" podStartSLOduration=2.094150794 podStartE2EDuration="4.470726466s" totalImagesPullingTime="2.376575672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:11:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:11:59.468848849 +0000 UTC m=+2370.176435146" watchObservedRunningTime="2026-10-31 13:11:59.470726466 +0000 UTC m=+2370.178312760" Oct 31 13:12:00.215112 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:00.215074 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11397-85e626"] Oct 31 13:12:00.219005 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:00.218979 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11397-85e626"] Oct 31 13:12:00.384367 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:00.384328 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11397-a62fc3"] Oct 31 13:12:00.385064 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:00.385048 5693 cpu_manager.go:414] "containerMap: removing container" podUID="5d3b7f10-6969-447c-be69-8fff9c87155a" containerName="test-curl-11397-85e626" Oct 31 13:12:00.385122 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:00.385066 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d3b7f10-6969-447c-be69-8fff9c87155a" containerName="test-curl-11397-85e626" Oct 31 13:12:00.385199 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:00.385190 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d3b7f10-6969-447c-be69-8fff9c87155a" containerName="test-curl-11397-85e626" Oct 31 13:12:00.389132 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:00.389116 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11397-a62fc3" Oct 31 13:12:00.405698 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:00.405669 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11397-a62fc3"] Oct 31 13:12:00.476884 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:00.476798 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t6gz\" (UniqueName: \"kubernetes.io/projected/c3cb7a67-8b22-43ab-a4ec-0bc0d0edb163-kube-api-access-9t6gz\") pod \"test-curl-11397-a62fc3\" (UID: \"c3cb7a67-8b22-43ab-a4ec-0bc0d0edb163\") " pod="opendatahub/test-curl-11397-a62fc3" Oct 31 13:12:00.577893 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:00.577858 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t6gz\" (UniqueName: \"kubernetes.io/projected/c3cb7a67-8b22-43ab-a4ec-0bc0d0edb163-kube-api-access-9t6gz\") pod \"test-curl-11397-a62fc3\" (UID: \"c3cb7a67-8b22-43ab-a4ec-0bc0d0edb163\") " pod="opendatahub/test-curl-11397-a62fc3" Oct 31 13:12:00.593476 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:00.593445 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t6gz\" (UniqueName: \"kubernetes.io/projected/c3cb7a67-8b22-43ab-a4ec-0bc0d0edb163-kube-api-access-9t6gz\") pod \"test-curl-11397-a62fc3\" (UID: \"c3cb7a67-8b22-43ab-a4ec-0bc0d0edb163\") " pod="opendatahub/test-curl-11397-a62fc3" Oct 31 13:12:00.702210 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:00.702169 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11397-a62fc3" Oct 31 13:12:01.011039 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:01.011007 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11397-a62fc3"] Oct 31 13:12:01.013106 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:12:01.013073 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3cb7a67_8b22_43ab_a4ec_0bc0d0edb163.slice/crio-9755554069ba24f7656a27a1b7e9cfa51cb5f6b73735a6b119ade4f3e65a8b93 WatchSource:0}: Error finding container 9755554069ba24f7656a27a1b7e9cfa51cb5f6b73735a6b119ade4f3e65a8b93: Status 404 returned error can't find the container with id 9755554069ba24f7656a27a1b7e9cfa51cb5f6b73735a6b119ade4f3e65a8b93 Oct 31 13:12:01.483816 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:01.483732 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11397-85e626" podUID="5d3b7f10-6969-447c-be69-8fff9c87155a" containerName="test-curl-11397-85e626" containerID="cri-o://2bcd23c32594561513aeaff89a062d2c73561b3e731fa8137fb5a088377c41da" gracePeriod=2 Oct 31 13:12:01.484328 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:01.484223 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11397-a62fc3" event={"ID":"c3cb7a67-8b22-43ab-a4ec-0bc0d0edb163","Type":"ContainerStarted","Data":"de81f0b6bb9d431ef500465d2ffb47f9c8d1d9fcc7d65030ae15c5a5b8bfcdb7"} Oct 31 13:12:01.484491 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:01.484417 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11397-a62fc3" event={"ID":"c3cb7a67-8b22-43ab-a4ec-0bc0d0edb163","Type":"ContainerStarted","Data":"9755554069ba24f7656a27a1b7e9cfa51cb5f6b73735a6b119ade4f3e65a8b93"} Oct 31 13:12:01.511126 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:01.511044 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11397-a62fc3" podStartSLOduration=1.5110231669999998 podStartE2EDuration="1.511023167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:01.506230278 +0000 UTC m=+2372.213816575" watchObservedRunningTime="2026-10-31 13:12:01.511023167 +0000 UTC m=+2372.218609463" Oct 31 13:12:02.238954 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:02.238917 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11397-a62fc3"] Oct 31 13:12:02.246589 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:02.246559 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11397-a62fc3"] Oct 31 13:12:02.402038 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:02.402007 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11397-0d95fd"] Oct 31 13:12:02.402803 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:02.402781 5693 cpu_manager.go:414] "containerMap: removing container" podUID="c3cb7a67-8b22-43ab-a4ec-0bc0d0edb163" containerName="test-curl-11397-a62fc3" Oct 31 13:12:02.402803 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:02.402802 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3cb7a67-8b22-43ab-a4ec-0bc0d0edb163" containerName="test-curl-11397-a62fc3" Oct 31 13:12:02.402996 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:02.402979 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3cb7a67-8b22-43ab-a4ec-0bc0d0edb163" containerName="test-curl-11397-a62fc3" Oct 31 13:12:02.406712 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:02.406691 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11397-0d95fd" Oct 31 13:12:02.412386 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:02.412358 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11397-0d95fd"] Oct 31 13:12:02.534907 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:02.534823 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vqrc\" (UniqueName: \"kubernetes.io/projected/7cddd68d-b734-4bf2-a70d-96f6bef4ead0-kube-api-access-2vqrc\") pod \"test-curl-11397-0d95fd\" (UID: \"7cddd68d-b734-4bf2-a70d-96f6bef4ead0\") " pod="opendatahub/test-curl-11397-0d95fd" Oct 31 13:12:02.636542 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:02.636505 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vqrc\" (UniqueName: \"kubernetes.io/projected/7cddd68d-b734-4bf2-a70d-96f6bef4ead0-kube-api-access-2vqrc\") pod \"test-curl-11397-0d95fd\" (UID: \"7cddd68d-b734-4bf2-a70d-96f6bef4ead0\") " pod="opendatahub/test-curl-11397-0d95fd" Oct 31 13:12:02.649413 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:02.649376 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vqrc\" (UniqueName: \"kubernetes.io/projected/7cddd68d-b734-4bf2-a70d-96f6bef4ead0-kube-api-access-2vqrc\") pod \"test-curl-11397-0d95fd\" (UID: \"7cddd68d-b734-4bf2-a70d-96f6bef4ead0\") " pod="opendatahub/test-curl-11397-0d95fd" Oct 31 13:12:02.724081 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:02.724041 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11397-0d95fd" Oct 31 13:12:03.029373 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:03.029342 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11397-0d95fd"] Oct 31 13:12:03.031712 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:12:03.031688 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7cddd68d_b734_4bf2_a70d_96f6bef4ead0.slice/crio-506abd42e088389a2d75d511afffb43dae37698c85e5c891b7c2f121af564bee WatchSource:0}: Error finding container 506abd42e088389a2d75d511afffb43dae37698c85e5c891b7c2f121af564bee: Status 404 returned error can't find the container with id 506abd42e088389a2d75d511afffb43dae37698c85e5c891b7c2f121af564bee Oct 31 13:12:03.505025 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:03.504997 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11397-0d95fd" event={"ID":"7cddd68d-b734-4bf2-a70d-96f6bef4ead0","Type":"ContainerStarted","Data":"56e78eff3df4a2e4f04ae26f84e3e251bb78b179e5a1230c6b1f6488bf8ac960"} Oct 31 13:12:03.505149 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:03.505038 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11397-0d95fd" event={"ID":"7cddd68d-b734-4bf2-a70d-96f6bef4ead0","Type":"ContainerStarted","Data":"506abd42e088389a2d75d511afffb43dae37698c85e5c891b7c2f121af564bee"} Oct 31 13:12:03.505270 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:03.505246 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11397-a62fc3" podUID="c3cb7a67-8b22-43ab-a4ec-0bc0d0edb163" containerName="test-curl-11397-a62fc3" containerID="cri-o://de81f0b6bb9d431ef500465d2ffb47f9c8d1d9fcc7d65030ae15c5a5b8bfcdb7" gracePeriod=2 Oct 31 13:12:03.522877 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:03.522826 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11397-0d95fd" podStartSLOduration=1.522810051 podStartE2EDuration="1.522810051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:03.521510493 +0000 UTC m=+2374.229096789" watchObservedRunningTime="2026-10-31 13:12:03.522810051 +0000 UTC m=+2374.230396345" Oct 31 13:12:03.708431 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:03.708407 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11397-85e626" Oct 31 13:12:03.873606 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:03.862356 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d3b7f10-6969-447c-be69-8fff9c87155a-kube-api-access-jjksr\" (UniqueName: \"kubernetes.io/projected/5d3b7f10-6969-447c-be69-8fff9c87155a-kube-api-access-jjksr\") pod \"5d3b7f10-6969-447c-be69-8fff9c87155a\" (UID: \"5d3b7f10-6969-447c-be69-8fff9c87155a\") " Oct 31 13:12:03.883180 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:03.882966 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d3b7f10-6969-447c-be69-8fff9c87155a-kube-api-access-jjksr" pod "5d3b7f10-6969-447c-be69-8fff9c87155a" (UID: "5d3b7f10-6969-447c-be69-8fff9c87155a"). InnerVolumeSpecName "kube-api-access-jjksr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:12:03.968889 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:03.968859 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjksr\" (UniqueName: \"kubernetes.io/projected/5d3b7f10-6969-447c-be69-8fff9c87155a-kube-api-access-jjksr\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:04.239399 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:04.239366 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11397-0d95fd"] Oct 31 13:12:04.244442 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:04.244412 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11397-0d95fd"] Oct 31 13:12:04.401500 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:04.401469 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11397-a7ca3d"] Oct 31 13:12:04.402199 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:04.402181 5693 cpu_manager.go:414] "containerMap: removing container" podUID="7cddd68d-b734-4bf2-a70d-96f6bef4ead0" containerName="test-curl-11397-0d95fd" Oct 31 13:12:04.402268 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:04.402201 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cddd68d-b734-4bf2-a70d-96f6bef4ead0" containerName="test-curl-11397-0d95fd" Oct 31 13:12:04.402323 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:04.402318 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cddd68d-b734-4bf2-a70d-96f6bef4ead0" containerName="test-curl-11397-0d95fd" Oct 31 13:12:04.405960 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:04.405939 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11397-a7ca3d" Oct 31 13:12:04.411094 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:04.411072 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11397-a7ca3d"] Oct 31 13:12:04.513463 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:04.513372 5693 generic.go:356] "Generic (PLEG): container finished" podID="5d3b7f10-6969-447c-be69-8fff9c87155a" containerID="2bcd23c32594561513aeaff89a062d2c73561b3e731fa8137fb5a088377c41da" exitCode=137 Oct 31 13:12:04.513463 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:04.513421 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11397-85e626" Oct 31 13:12:04.513719 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:04.513475 5693 scope.go:122] "RemoveContainer" containerID="2bcd23c32594561513aeaff89a062d2c73561b3e731fa8137fb5a088377c41da" Oct 31 13:12:04.524543 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:04.524526 5693 scope.go:122] "RemoveContainer" containerID="2bcd23c32594561513aeaff89a062d2c73561b3e731fa8137fb5a088377c41da" Oct 31 13:12:04.524855 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:12:04.524824 5693 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11397-85e626\" is forbidden: User \"system:node:ip-10-0-131-157.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-157.us-east-2.compute.internal' and this object" podUID="5d3b7f10-6969-447c-be69-8fff9c87155a" pod="opendatahub/test-curl-11397-85e626" Oct 31 13:12:04.524958 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:12:04.524849 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bcd23c32594561513aeaff89a062d2c73561b3e731fa8137fb5a088377c41da\": container with ID starting with 2bcd23c32594561513aeaff89a062d2c73561b3e731fa8137fb5a088377c41da not found: ID does not exist" containerID="2bcd23c32594561513aeaff89a062d2c73561b3e731fa8137fb5a088377c41da" Oct 31 13:12:04.524958 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:04.524875 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bcd23c32594561513aeaff89a062d2c73561b3e731fa8137fb5a088377c41da"} err="failed to get container status \"2bcd23c32594561513aeaff89a062d2c73561b3e731fa8137fb5a088377c41da\": rpc error: code = NotFound desc = could not find container \"2bcd23c32594561513aeaff89a062d2c73561b3e731fa8137fb5a088377c41da\": container with ID starting with 2bcd23c32594561513aeaff89a062d2c73561b3e731fa8137fb5a088377c41da not found: ID does not exist" Oct 31 13:12:04.577047 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:04.577012 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltz8h\" (UniqueName: \"kubernetes.io/projected/f4644bf9-4fb7-4924-933a-6946919a998f-kube-api-access-ltz8h\") pod \"test-curl-11397-a7ca3d\" (UID: \"f4644bf9-4fb7-4924-933a-6946919a998f\") " pod="opendatahub/test-curl-11397-a7ca3d" Oct 31 13:12:04.681169 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:04.681129 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltz8h\" (UniqueName: \"kubernetes.io/projected/f4644bf9-4fb7-4924-933a-6946919a998f-kube-api-access-ltz8h\") pod \"test-curl-11397-a7ca3d\" (UID: \"f4644bf9-4fb7-4924-933a-6946919a998f\") " pod="opendatahub/test-curl-11397-a7ca3d" Oct 31 13:12:04.694320 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:04.694290 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltz8h\" (UniqueName: \"kubernetes.io/projected/f4644bf9-4fb7-4924-933a-6946919a998f-kube-api-access-ltz8h\") pod \"test-curl-11397-a7ca3d\" (UID: \"f4644bf9-4fb7-4924-933a-6946919a998f\") " pod="opendatahub/test-curl-11397-a7ca3d" Oct 31 13:12:04.722214 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:04.722173 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11397-a7ca3d" Oct 31 13:12:05.011836 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:05.011794 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11397-a7ca3d"] Oct 31 13:12:05.012625 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:12:05.012596 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4644bf9_4fb7_4924_933a_6946919a998f.slice/crio-14655fe512c3b033cc42db5206f70822a6a62b559ff9df9ba405f11ddf45098f WatchSource:0}: Error finding container 14655fe512c3b033cc42db5206f70822a6a62b559ff9df9ba405f11ddf45098f: Status 404 returned error can't find the container with id 14655fe512c3b033cc42db5206f70822a6a62b559ff9df9ba405f11ddf45098f Oct 31 13:12:05.422514 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:05.422480 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d3b7f10-6969-447c-be69-8fff9c87155a" path="/var/lib/kubelet/pods/5d3b7f10-6969-447c-be69-8fff9c87155a/volumes" Oct 31 13:12:05.523928 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:05.523898 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11397-a7ca3d" event={"ID":"f4644bf9-4fb7-4924-933a-6946919a998f","Type":"ContainerStarted","Data":"52ace4f4d8f256605b82f604bdd816d5cf24f3783b73ab6eb3a16eef8ca3d668"} Oct 31 13:12:05.524088 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:05.523937 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11397-a7ca3d" event={"ID":"f4644bf9-4fb7-4924-933a-6946919a998f","Type":"ContainerStarted","Data":"14655fe512c3b033cc42db5206f70822a6a62b559ff9df9ba405f11ddf45098f"} Oct 31 13:12:05.524226 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:05.524191 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11397-0d95fd" podUID="7cddd68d-b734-4bf2-a70d-96f6bef4ead0" containerName="test-curl-11397-0d95fd" containerID="cri-o://56e78eff3df4a2e4f04ae26f84e3e251bb78b179e5a1230c6b1f6488bf8ac960" gracePeriod=2 Oct 31 13:12:05.542663 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:05.542592 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11397-a7ca3d" podStartSLOduration=1.542578663 podStartE2EDuration="1.542578663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:05.539189949 +0000 UTC m=+2376.246776244" watchObservedRunningTime="2026-10-31 13:12:05.542578663 +0000 UTC m=+2376.250164957" Oct 31 13:12:05.686075 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:05.686050 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11397-a62fc3" Oct 31 13:12:05.799957 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:05.799928 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3cb7a67-8b22-43ab-a4ec-0bc0d0edb163-kube-api-access-9t6gz\" (UniqueName: \"kubernetes.io/projected/c3cb7a67-8b22-43ab-a4ec-0bc0d0edb163-kube-api-access-9t6gz\") pod \"c3cb7a67-8b22-43ab-a4ec-0bc0d0edb163\" (UID: \"c3cb7a67-8b22-43ab-a4ec-0bc0d0edb163\") " Oct 31 13:12:05.802291 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:05.802246 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3cb7a67-8b22-43ab-a4ec-0bc0d0edb163-kube-api-access-9t6gz" pod "c3cb7a67-8b22-43ab-a4ec-0bc0d0edb163" (UID: "c3cb7a67-8b22-43ab-a4ec-0bc0d0edb163"). InnerVolumeSpecName "kube-api-access-9t6gz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:12:05.902237 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:05.902198 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t6gz\" (UniqueName: \"kubernetes.io/projected/c3cb7a67-8b22-43ab-a4ec-0bc0d0edb163-kube-api-access-9t6gz\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:06.292696 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:06.292661 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11397-a7ca3d"] Oct 31 13:12:06.317391 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:06.317356 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11397-a7ca3d"] Oct 31 13:12:06.535106 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:06.535074 5693 generic.go:356] "Generic (PLEG): container finished" podID="c3cb7a67-8b22-43ab-a4ec-0bc0d0edb163" containerID="de81f0b6bb9d431ef500465d2ffb47f9c8d1d9fcc7d65030ae15c5a5b8bfcdb7" exitCode=137 Oct 31 13:12:06.535324 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:06.535159 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11397-a62fc3" Oct 31 13:12:06.535324 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:06.535182 5693 scope.go:122] "RemoveContainer" containerID="de81f0b6bb9d431ef500465d2ffb47f9c8d1d9fcc7d65030ae15c5a5b8bfcdb7" Oct 31 13:12:06.554070 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:12:06.554014 5693 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11397-a62fc3\" is forbidden: User \"system:node:ip-10-0-131-157.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-157.us-east-2.compute.internal' and this object" podUID="c3cb7a67-8b22-43ab-a4ec-0bc0d0edb163" pod="opendatahub/test-curl-11397-a62fc3" Oct 31 13:12:06.556901 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:06.556881 5693 scope.go:122] "RemoveContainer" containerID="de81f0b6bb9d431ef500465d2ffb47f9c8d1d9fcc7d65030ae15c5a5b8bfcdb7" Oct 31 13:12:06.557217 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:12:06.557184 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de81f0b6bb9d431ef500465d2ffb47f9c8d1d9fcc7d65030ae15c5a5b8bfcdb7\": container with ID starting with de81f0b6bb9d431ef500465d2ffb47f9c8d1d9fcc7d65030ae15c5a5b8bfcdb7 not found: ID does not exist" containerID="de81f0b6bb9d431ef500465d2ffb47f9c8d1d9fcc7d65030ae15c5a5b8bfcdb7" Oct 31 13:12:06.557284 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:06.557231 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de81f0b6bb9d431ef500465d2ffb47f9c8d1d9fcc7d65030ae15c5a5b8bfcdb7"} err="failed to get container status \"de81f0b6bb9d431ef500465d2ffb47f9c8d1d9fcc7d65030ae15c5a5b8bfcdb7\": rpc error: code = NotFound desc = could not find container \"de81f0b6bb9d431ef500465d2ffb47f9c8d1d9fcc7d65030ae15c5a5b8bfcdb7\": container with ID starting with de81f0b6bb9d431ef500465d2ffb47f9c8d1d9fcc7d65030ae15c5a5b8bfcdb7 not found: ID does not exist" Oct 31 13:12:07.410554 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:07.410522 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3cb7a67-8b22-43ab-a4ec-0bc0d0edb163" path="/var/lib/kubelet/pods/c3cb7a67-8b22-43ab-a4ec-0bc0d0edb163/volumes" Oct 31 13:12:07.545030 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:07.544988 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11397-a7ca3d" podUID="f4644bf9-4fb7-4924-933a-6946919a998f" containerName="test-curl-11397-a7ca3d" containerID="cri-o://52ace4f4d8f256605b82f604bdd816d5cf24f3783b73ab6eb3a16eef8ca3d668" gracePeriod=2 Oct 31 13:12:07.739457 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:07.739432 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11397-0d95fd" Oct 31 13:12:07.930838 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:07.930805 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cddd68d-b734-4bf2-a70d-96f6bef4ead0-kube-api-access-2vqrc\" (UniqueName: \"kubernetes.io/projected/7cddd68d-b734-4bf2-a70d-96f6bef4ead0-kube-api-access-2vqrc\") pod \"7cddd68d-b734-4bf2-a70d-96f6bef4ead0\" (UID: \"7cddd68d-b734-4bf2-a70d-96f6bef4ead0\") " Oct 31 13:12:07.933259 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:07.933232 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cddd68d-b734-4bf2-a70d-96f6bef4ead0-kube-api-access-2vqrc" pod "7cddd68d-b734-4bf2-a70d-96f6bef4ead0" (UID: "7cddd68d-b734-4bf2-a70d-96f6bef4ead0"). InnerVolumeSpecName "kube-api-access-2vqrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:12:08.032543 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.032463 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vqrc\" (UniqueName: \"kubernetes.io/projected/7cddd68d-b734-4bf2-a70d-96f6bef4ead0-kube-api-access-2vqrc\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:08.487891 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.487856 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4"] Oct 31 13:12:08.488955 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.488926 5693 cpu_manager.go:414] "containerMap: removing container" podUID="f4644bf9-4fb7-4924-933a-6946919a998f" containerName="test-curl-11397-a7ca3d" Oct 31 13:12:08.488955 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.488954 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4644bf9-4fb7-4924-933a-6946919a998f" containerName="test-curl-11397-a7ca3d" Oct 31 13:12:08.489203 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.489185 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4644bf9-4fb7-4924-933a-6946919a998f" containerName="test-curl-11397-a7ca3d" Oct 31 13:12:08.492853 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.492836 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" Oct 31 13:12:08.514317 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.514292 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-842ac9-openshift-default-dockercfg-tdrsf\"" Oct 31 13:12:08.517162 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.517138 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4"] Oct 31 13:12:08.539201 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.539158 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a022831f-b584-43fe-91a7-7f541c066ace-istio-podinfo\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4\" (UID: \"a022831f-b584-43fe-91a7-7f541c066ace\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" Oct 31 13:12:08.539409 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.539273 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a022831f-b584-43fe-91a7-7f541c066ace-istio-envoy\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4\" (UID: \"a022831f-b584-43fe-91a7-7f541c066ace\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" Oct 31 13:12:08.539409 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.539372 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmt6x\" (UniqueName: \"kubernetes.io/projected/a022831f-b584-43fe-91a7-7f541c066ace-kube-api-access-cmt6x\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4\" (UID: \"a022831f-b584-43fe-91a7-7f541c066ace\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" Oct 31 13:12:08.539524 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.539433 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a022831f-b584-43fe-91a7-7f541c066ace-credential-socket\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4\" (UID: \"a022831f-b584-43fe-91a7-7f541c066ace\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" Oct 31 13:12:08.539524 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.539506 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a022831f-b584-43fe-91a7-7f541c066ace-workload-socket\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4\" (UID: \"a022831f-b584-43fe-91a7-7f541c066ace\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" Oct 31 13:12:08.539603 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.539539 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a022831f-b584-43fe-91a7-7f541c066ace-workload-certs\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4\" (UID: \"a022831f-b584-43fe-91a7-7f541c066ace\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" Oct 31 13:12:08.539603 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.539581 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a022831f-b584-43fe-91a7-7f541c066ace-istio-data\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4\" (UID: \"a022831f-b584-43fe-91a7-7f541c066ace\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" Oct 31 13:12:08.539715 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.539660 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a022831f-b584-43fe-91a7-7f541c066ace-istiod-ca-cert\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4\" (UID: \"a022831f-b584-43fe-91a7-7f541c066ace\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" Oct 31 13:12:08.539768 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.539746 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a022831f-b584-43fe-91a7-7f541c066ace-istio-token\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4\" (UID: \"a022831f-b584-43fe-91a7-7f541c066ace\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" Oct 31 13:12:08.554136 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.554100 5693 generic.go:356] "Generic (PLEG): container finished" podID="7cddd68d-b734-4bf2-a70d-96f6bef4ead0" containerID="56e78eff3df4a2e4f04ae26f84e3e251bb78b179e5a1230c6b1f6488bf8ac960" exitCode=137 Oct 31 13:12:08.554316 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.554147 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11397-0d95fd" Oct 31 13:12:08.554316 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.554209 5693 scope.go:122] "RemoveContainer" containerID="56e78eff3df4a2e4f04ae26f84e3e251bb78b179e5a1230c6b1f6488bf8ac960" Oct 31 13:12:08.616936 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.568762 5693 scope.go:122] "RemoveContainer" containerID="56e78eff3df4a2e4f04ae26f84e3e251bb78b179e5a1230c6b1f6488bf8ac960" Oct 31 13:12:08.616936 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:12:08.569105 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56e78eff3df4a2e4f04ae26f84e3e251bb78b179e5a1230c6b1f6488bf8ac960\": container with ID starting with 56e78eff3df4a2e4f04ae26f84e3e251bb78b179e5a1230c6b1f6488bf8ac960 not found: ID does not exist" containerID="56e78eff3df4a2e4f04ae26f84e3e251bb78b179e5a1230c6b1f6488bf8ac960" Oct 31 13:12:08.616936 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.569138 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56e78eff3df4a2e4f04ae26f84e3e251bb78b179e5a1230c6b1f6488bf8ac960"} err="failed to get container status \"56e78eff3df4a2e4f04ae26f84e3e251bb78b179e5a1230c6b1f6488bf8ac960\": rpc error: code = NotFound desc = could not find container \"56e78eff3df4a2e4f04ae26f84e3e251bb78b179e5a1230c6b1f6488bf8ac960\": container with ID starting with 56e78eff3df4a2e4f04ae26f84e3e251bb78b179e5a1230c6b1f6488bf8ac960 not found: ID does not exist" Oct 31 13:12:08.616936 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:12:08.570419 5693 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11397-0d95fd\" is forbidden: User \"system:node:ip-10-0-131-157.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-157.us-east-2.compute.internal' and this object" podUID="7cddd68d-b734-4bf2-a70d-96f6bef4ead0" pod="opendatahub/test-curl-11397-0d95fd" Oct 31 13:12:08.656675 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.653991 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a022831f-b584-43fe-91a7-7f541c066ace-workload-certs\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4\" (UID: \"a022831f-b584-43fe-91a7-7f541c066ace\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" Oct 31 13:12:08.656675 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.654064 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a022831f-b584-43fe-91a7-7f541c066ace-istio-data\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4\" (UID: \"a022831f-b584-43fe-91a7-7f541c066ace\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" Oct 31 13:12:08.656675 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.654137 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a022831f-b584-43fe-91a7-7f541c066ace-istiod-ca-cert\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4\" (UID: \"a022831f-b584-43fe-91a7-7f541c066ace\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" Oct 31 13:12:08.656675 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.654222 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a022831f-b584-43fe-91a7-7f541c066ace-istio-token\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4\" (UID: \"a022831f-b584-43fe-91a7-7f541c066ace\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" Oct 31 13:12:08.656675 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.654298 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a022831f-b584-43fe-91a7-7f541c066ace-istio-podinfo\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4\" (UID: \"a022831f-b584-43fe-91a7-7f541c066ace\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" Oct 31 13:12:08.656675 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.654395 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a022831f-b584-43fe-91a7-7f541c066ace-istio-envoy\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4\" (UID: \"a022831f-b584-43fe-91a7-7f541c066ace\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" Oct 31 13:12:08.656675 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.654574 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmt6x\" (UniqueName: \"kubernetes.io/projected/a022831f-b584-43fe-91a7-7f541c066ace-kube-api-access-cmt6x\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4\" (UID: \"a022831f-b584-43fe-91a7-7f541c066ace\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" Oct 31 13:12:08.656675 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.654667 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a022831f-b584-43fe-91a7-7f541c066ace-credential-socket\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4\" (UID: \"a022831f-b584-43fe-91a7-7f541c066ace\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" Oct 31 13:12:08.656675 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.654765 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a022831f-b584-43fe-91a7-7f541c066ace-workload-socket\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4\" (UID: \"a022831f-b584-43fe-91a7-7f541c066ace\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" Oct 31 13:12:08.656675 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.655948 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a022831f-b584-43fe-91a7-7f541c066ace-workload-certs\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4\" (UID: \"a022831f-b584-43fe-91a7-7f541c066ace\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" Oct 31 13:12:08.656675 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.656421 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a022831f-b584-43fe-91a7-7f541c066ace-istio-data\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4\" (UID: \"a022831f-b584-43fe-91a7-7f541c066ace\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" Oct 31 13:12:08.662195 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.657185 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a022831f-b584-43fe-91a7-7f541c066ace-istiod-ca-cert\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4\" (UID: \"a022831f-b584-43fe-91a7-7f541c066ace\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" Oct 31 13:12:08.662195 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.659627 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a022831f-b584-43fe-91a7-7f541c066ace-credential-socket\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4\" (UID: \"a022831f-b584-43fe-91a7-7f541c066ace\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" Oct 31 13:12:08.696676 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.668055 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a022831f-b584-43fe-91a7-7f541c066ace-workload-socket\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4\" (UID: \"a022831f-b584-43fe-91a7-7f541c066ace\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" Oct 31 13:12:08.696676 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.685875 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a022831f-b584-43fe-91a7-7f541c066ace-istio-podinfo\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4\" (UID: \"a022831f-b584-43fe-91a7-7f541c066ace\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" Oct 31 13:12:08.745714 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.745006 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a022831f-b584-43fe-91a7-7f541c066ace-istio-envoy\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4\" (UID: \"a022831f-b584-43fe-91a7-7f541c066ace\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" Oct 31 13:12:08.760674 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.754874 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmt6x\" (UniqueName: \"kubernetes.io/projected/a022831f-b584-43fe-91a7-7f541c066ace-kube-api-access-cmt6x\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4\" (UID: \"a022831f-b584-43fe-91a7-7f541c066ace\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" Oct 31 13:12:08.769058 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.764634 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a022831f-b584-43fe-91a7-7f541c066ace-istio-token\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4\" (UID: \"a022831f-b584-43fe-91a7-7f541c066ace\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" Oct 31 13:12:08.816673 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:08.806953 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" Oct 31 13:12:09.134427 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:09.134400 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4"] Oct 31 13:12:09.135171 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:12:09.135137 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda022831f_b584_43fe_91a7_7f541c066ace.slice/crio-d6626697337212525e6ef7dff7209bff00c943917572cc1641ef215b2d000e6f WatchSource:0}: Error finding container d6626697337212525e6ef7dff7209bff00c943917572cc1641ef215b2d000e6f: Status 404 returned error can't find the container with id d6626697337212525e6ef7dff7209bff00c943917572cc1641ef215b2d000e6f Oct 31 13:12:09.137931 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:09.137898 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:12:09.138002 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:09.137965 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:12:09.138002 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:09.137995 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:12:09.422933 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:09.422901 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cddd68d-b734-4bf2-a70d-96f6bef4ead0" path="/var/lib/kubelet/pods/7cddd68d-b734-4bf2-a70d-96f6bef4ead0/volumes" Oct 31 13:12:09.423861 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:12:09.423830 5693 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11397-0d95fd\" is forbidden: User \"system:node:ip-10-0-131-157.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-157.us-east-2.compute.internal' and this object" podUID="7cddd68d-b734-4bf2-a70d-96f6bef4ead0" pod="opendatahub/test-curl-11397-0d95fd" Oct 31 13:12:09.585069 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:09.585022 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" event={"ID":"a022831f-b584-43fe-91a7-7f541c066ace","Type":"ContainerStarted","Data":"c140569b2bd55c4082acba10ca650cd3758e8d04deb825997848a11005d1a9f8"} Oct 31 13:12:09.585530 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:09.585129 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" event={"ID":"a022831f-b584-43fe-91a7-7f541c066ace","Type":"ContainerStarted","Data":"d6626697337212525e6ef7dff7209bff00c943917572cc1641ef215b2d000e6f"} Oct 31 13:12:09.617184 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:09.617114 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" podStartSLOduration=1.617098338 podStartE2EDuration="1.617098338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:09.614281307 +0000 UTC m=+2380.321867641" watchObservedRunningTime="2026-10-31 13:12:09.617098338 +0000 UTC m=+2380.324684633" Oct 31 13:12:09.798091 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:09.798062 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11397-a7ca3d" Oct 31 13:12:09.807765 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:09.807729 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" Oct 31 13:12:09.898336 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:09.898298 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4644bf9-4fb7-4924-933a-6946919a998f-kube-api-access-ltz8h\" (UniqueName: \"kubernetes.io/projected/f4644bf9-4fb7-4924-933a-6946919a998f-kube-api-access-ltz8h\") pod \"f4644bf9-4fb7-4924-933a-6946919a998f\" (UID: \"f4644bf9-4fb7-4924-933a-6946919a998f\") " Oct 31 13:12:09.900872 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:09.900841 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4644bf9-4fb7-4924-933a-6946919a998f-kube-api-access-ltz8h" pod "f4644bf9-4fb7-4924-933a-6946919a998f" (UID: "f4644bf9-4fb7-4924-933a-6946919a998f"). InnerVolumeSpecName "kube-api-access-ltz8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:12:10.000218 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:10.000134 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltz8h\" (UniqueName: \"kubernetes.io/projected/f4644bf9-4fb7-4924-933a-6946919a998f-kube-api-access-ltz8h\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:10.594824 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:10.594793 5693 generic.go:356] "Generic (PLEG): container finished" podID="f4644bf9-4fb7-4924-933a-6946919a998f" containerID="52ace4f4d8f256605b82f604bdd816d5cf24f3783b73ab6eb3a16eef8ca3d668" exitCode=137 Oct 31 13:12:10.595298 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:10.594845 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11397-a7ca3d" Oct 31 13:12:10.595298 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:10.594898 5693 scope.go:122] "RemoveContainer" containerID="52ace4f4d8f256605b82f604bdd816d5cf24f3783b73ab6eb3a16eef8ca3d668" Oct 31 13:12:10.609721 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:12:10.609694 5693 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11397-a7ca3d\" is forbidden: User \"system:node:ip-10-0-131-157.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-157.us-east-2.compute.internal' and this object" podUID="f4644bf9-4fb7-4924-933a-6946919a998f" pod="opendatahub/test-curl-11397-a7ca3d" Oct 31 13:12:10.610377 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:10.610356 5693 scope.go:122] "RemoveContainer" containerID="52ace4f4d8f256605b82f604bdd816d5cf24f3783b73ab6eb3a16eef8ca3d668" Oct 31 13:12:10.610705 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:12:10.610685 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52ace4f4d8f256605b82f604bdd816d5cf24f3783b73ab6eb3a16eef8ca3d668\": container with ID starting with 52ace4f4d8f256605b82f604bdd816d5cf24f3783b73ab6eb3a16eef8ca3d668 not found: ID does not exist" containerID="52ace4f4d8f256605b82f604bdd816d5cf24f3783b73ab6eb3a16eef8ca3d668" Oct 31 13:12:10.610764 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:10.610715 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52ace4f4d8f256605b82f604bdd816d5cf24f3783b73ab6eb3a16eef8ca3d668"} err="failed to get container status \"52ace4f4d8f256605b82f604bdd816d5cf24f3783b73ab6eb3a16eef8ca3d668\": rpc error: code = NotFound desc = could not find container \"52ace4f4d8f256605b82f604bdd816d5cf24f3783b73ab6eb3a16eef8ca3d668\": container with ID starting with 52ace4f4d8f256605b82f604bdd816d5cf24f3783b73ab6eb3a16eef8ca3d668 not found: ID does not exist" Oct 31 13:12:10.814141 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:10.814108 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" Oct 31 13:12:10.842323 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:12:10.842272 5693 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11397-a7ca3d\" is forbidden: User \"system:node:ip-10-0-131-157.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-157.us-east-2.compute.internal' and this object" podUID="f4644bf9-4fb7-4924-933a-6946919a998f" pod="opendatahub/test-curl-11397-a7ca3d" Oct 31 13:12:11.411674 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:11.411608 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4644bf9-4fb7-4924-933a-6946919a998f" path="/var/lib/kubelet/pods/f4644bf9-4fb7-4924-933a-6946919a998f/volumes" Oct 31 13:12:11.603721 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:11.603691 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" Oct 31 13:12:11.604799 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:11.604776 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" Oct 31 13:12:13.243937 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:13.243901 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-842ac9-5d74f7f789-w2rqn"] Oct 31 13:12:13.249912 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:13.249882 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-842ac9-5d74f7f789-w2rqn" Oct 31 13:12:13.259137 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:13.259111 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-842ac9-5d74f7f789-w2rqn"] Oct 31 13:12:13.334665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:13.330956 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-842ac9\"" Oct 31 13:12:13.334665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:13.331091 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-842ac9-dockercfg-vdsbn\"" Oct 31 13:12:13.371816 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:13.369800 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efeec4f0-9411-4add-8d49-586201474004-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-842ac9-5d74f7f789-w2rqn\" (UID: \"efeec4f0-9411-4add-8d49-586201474004\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-842ac9-5d74f7f789-w2rqn" Oct 31 13:12:13.390667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:13.387896 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbh56\" (UniqueName: \"kubernetes.io/projected/efeec4f0-9411-4add-8d49-586201474004-kube-api-access-gbh56\") pod \"payload-processing-e2e-extmod-a-842ac9-5d74f7f789-w2rqn\" (UID: \"efeec4f0-9411-4add-8d49-586201474004\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-842ac9-5d74f7f789-w2rqn" Oct 31 13:12:13.498249 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:13.498156 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efeec4f0-9411-4add-8d49-586201474004-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-842ac9-5d74f7f789-w2rqn\" (UID: \"efeec4f0-9411-4add-8d49-586201474004\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-842ac9-5d74f7f789-w2rqn" Oct 31 13:12:13.500237 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:13.500085 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efeec4f0-9411-4add-8d49-586201474004-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-842ac9-5d74f7f789-w2rqn\" (UID: \"efeec4f0-9411-4add-8d49-586201474004\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-842ac9-5d74f7f789-w2rqn" Oct 31 13:12:13.501216 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:13.501198 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbh56\" (UniqueName: \"kubernetes.io/projected/efeec4f0-9411-4add-8d49-586201474004-kube-api-access-gbh56\") pod \"payload-processing-e2e-extmod-a-842ac9-5d74f7f789-w2rqn\" (UID: \"efeec4f0-9411-4add-8d49-586201474004\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-842ac9-5d74f7f789-w2rqn" Oct 31 13:12:13.514698 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:13.514245 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbh56\" (UniqueName: \"kubernetes.io/projected/efeec4f0-9411-4add-8d49-586201474004-kube-api-access-gbh56\") pod \"payload-processing-e2e-extmod-a-842ac9-5d74f7f789-w2rqn\" (UID: \"efeec4f0-9411-4add-8d49-586201474004\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-842ac9-5d74f7f789-w2rqn" Oct 31 13:12:13.567612 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:13.567582 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-842ac9-5d74f7f789-w2rqn" Oct 31 13:12:13.866543 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:13.866508 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-842ac9-5d74f7f789-w2rqn"] Oct 31 13:12:13.866849 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:12:13.866821 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefeec4f0_9411_4add_8d49_586201474004.slice/crio-62efc5f4520528c9d82c8f60d135b4f15ad24df3de8546348a86af24821fb6fe WatchSource:0}: Error finding container 62efc5f4520528c9d82c8f60d135b4f15ad24df3de8546348a86af24821fb6fe: Status 404 returned error can't find the container with id 62efc5f4520528c9d82c8f60d135b4f15ad24df3de8546348a86af24821fb6fe Oct 31 13:12:14.628429 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:14.628394 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-842ac9-5d74f7f789-w2rqn" event={"ID":"efeec4f0-9411-4add-8d49-586201474004","Type":"ContainerStarted","Data":"1214bde53139fc4ca67020a820b5c6c27ad78a6dfd75ea99ca6dde757018424b"} Oct 31 13:12:14.628906 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:14.628452 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-842ac9-5d74f7f789-w2rqn" event={"ID":"efeec4f0-9411-4add-8d49-586201474004","Type":"ContainerStarted","Data":"62efc5f4520528c9d82c8f60d135b4f15ad24df3de8546348a86af24821fb6fe"} Oct 31 13:12:14.628906 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:14.628473 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-842ac9-5d74f7f789-w2rqn" Oct 31 13:12:14.650970 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:14.650913 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-842ac9-5d74f7f789-w2rqn" podStartSLOduration=1.650898886 podStartE2EDuration="1.650898886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:14.646514045 +0000 UTC m=+2385.354100337" watchObservedRunningTime="2026-10-31 13:12:14.650898886 +0000 UTC m=+2385.358485181" Oct 31 13:12:18.689505 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:12:18.689468 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:12:25.639581 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:25.639550 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-842ac9-5d74f7f789-w2rqn" Oct 31 13:12:38.429715 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:38.429678 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9ef744-75b9b8c75c-hzlhr"] Oct 31 13:12:38.436077 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:38.436037 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9ef744-75b9b8c75c-hzlhr" Oct 31 13:12:38.461544 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:38.461514 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9ef744-79656647f-g5cd2"] Oct 31 13:12:38.466531 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:38.466500 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ef744-79656647f-g5cd2" Oct 31 13:12:38.470301 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:38.468704 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-9ef744\"" Oct 31 13:12:38.482662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:38.482618 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9ef744-75b9b8c75c-hzlhr"] Oct 31 13:12:38.490417 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:38.490389 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-9ef744-dockercfg-5vgjg\"" Oct 31 13:12:38.513104 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:38.513057 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9ef744-79656647f-g5cd2"] Oct 31 13:12:38.552856 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:38.552825 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf6026f7-85c0-4659-8888-e4532b55f73a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9ef744-79656647f-g5cd2\" (UID: \"bf6026f7-85c0-4659-8888-e4532b55f73a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ef744-79656647f-g5cd2" Oct 31 13:12:38.553017 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:38.552889 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tq8m\" (UniqueName: \"kubernetes.io/projected/430e7981-8421-4898-8bef-fba2efcb1676-kube-api-access-4tq8m\") pod \"payload-processing-e2e-extmod-a-9ef744-75b9b8c75c-hzlhr\" (UID: \"430e7981-8421-4898-8bef-fba2efcb1676\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9ef744-75b9b8c75c-hzlhr" Oct 31 13:12:38.553059 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:38.553012 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/430e7981-8421-4898-8bef-fba2efcb1676-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-9ef744-75b9b8c75c-hzlhr\" (UID: \"430e7981-8421-4898-8bef-fba2efcb1676\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9ef744-75b9b8c75c-hzlhr" Oct 31 13:12:38.553059 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:38.553050 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsj4j\" (UniqueName: \"kubernetes.io/projected/bf6026f7-85c0-4659-8888-e4532b55f73a-kube-api-access-gsj4j\") pod \"payload-pre-processing-e2e-extmod-a-9ef744-79656647f-g5cd2\" (UID: \"bf6026f7-85c0-4659-8888-e4532b55f73a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ef744-79656647f-g5cd2" Oct 31 13:12:38.657944 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:38.657900 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tq8m\" (UniqueName: \"kubernetes.io/projected/430e7981-8421-4898-8bef-fba2efcb1676-kube-api-access-4tq8m\") pod \"payload-processing-e2e-extmod-a-9ef744-75b9b8c75c-hzlhr\" (UID: \"430e7981-8421-4898-8bef-fba2efcb1676\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9ef744-75b9b8c75c-hzlhr" Oct 31 13:12:38.658884 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:38.658855 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/430e7981-8421-4898-8bef-fba2efcb1676-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-9ef744-75b9b8c75c-hzlhr\" (UID: \"430e7981-8421-4898-8bef-fba2efcb1676\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9ef744-75b9b8c75c-hzlhr" Oct 31 13:12:38.659093 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:38.659076 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsj4j\" (UniqueName: \"kubernetes.io/projected/bf6026f7-85c0-4659-8888-e4532b55f73a-kube-api-access-gsj4j\") pod \"payload-pre-processing-e2e-extmod-a-9ef744-79656647f-g5cd2\" (UID: \"bf6026f7-85c0-4659-8888-e4532b55f73a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ef744-79656647f-g5cd2" Oct 31 13:12:38.659490 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:38.659469 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf6026f7-85c0-4659-8888-e4532b55f73a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9ef744-79656647f-g5cd2\" (UID: \"bf6026f7-85c0-4659-8888-e4532b55f73a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ef744-79656647f-g5cd2" Oct 31 13:12:38.662204 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:38.662169 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf6026f7-85c0-4659-8888-e4532b55f73a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9ef744-79656647f-g5cd2\" (UID: \"bf6026f7-85c0-4659-8888-e4532b55f73a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ef744-79656647f-g5cd2" Oct 31 13:12:38.665345 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:38.665322 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/430e7981-8421-4898-8bef-fba2efcb1676-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-9ef744-75b9b8c75c-hzlhr\" (UID: \"430e7981-8421-4898-8bef-fba2efcb1676\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9ef744-75b9b8c75c-hzlhr" Oct 31 13:12:38.729706 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:38.727369 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsj4j\" (UniqueName: \"kubernetes.io/projected/bf6026f7-85c0-4659-8888-e4532b55f73a-kube-api-access-gsj4j\") pod \"payload-pre-processing-e2e-extmod-a-9ef744-79656647f-g5cd2\" (UID: \"bf6026f7-85c0-4659-8888-e4532b55f73a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ef744-79656647f-g5cd2" Oct 31 13:12:38.809710 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:38.788897 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ef744-79656647f-g5cd2" Oct 31 13:12:38.812837 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:38.812804 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tq8m\" (UniqueName: \"kubernetes.io/projected/430e7981-8421-4898-8bef-fba2efcb1676-kube-api-access-4tq8m\") pod \"payload-processing-e2e-extmod-a-9ef744-75b9b8c75c-hzlhr\" (UID: \"430e7981-8421-4898-8bef-fba2efcb1676\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9ef744-75b9b8c75c-hzlhr" Oct 31 13:12:39.058846 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:39.058760 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9ef744-75b9b8c75c-hzlhr" Oct 31 13:12:39.150863 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:12:39.141719 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf6026f7_85c0_4659_8888_e4532b55f73a.slice/crio-154a47949befe794a7c91824f2158e616b8fc398abb85861e86252255c27b743 WatchSource:0}: Error finding container 154a47949befe794a7c91824f2158e616b8fc398abb85861e86252255c27b743: Status 404 returned error can't find the container with id 154a47949befe794a7c91824f2158e616b8fc398abb85861e86252255c27b743 Oct 31 13:12:39.150863 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:39.147000 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9ef744-79656647f-g5cd2"] Oct 31 13:12:39.376986 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:39.376904 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9ef744-75b9b8c75c-hzlhr"] Oct 31 13:12:39.382153 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:12:39.382122 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod430e7981_8421_4898_8bef_fba2efcb1676.slice/crio-2f75eff167c10f97af31400fa88bddee77e7be66d6469ee8d520ba24682e2869 WatchSource:0}: Error finding container 2f75eff167c10f97af31400fa88bddee77e7be66d6469ee8d520ba24682e2869: Status 404 returned error can't find the container with id 2f75eff167c10f97af31400fa88bddee77e7be66d6469ee8d520ba24682e2869 Oct 31 13:12:39.880998 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:39.880955 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ef744-79656647f-g5cd2" event={"ID":"bf6026f7-85c0-4659-8888-e4532b55f73a","Type":"ContainerStarted","Data":"97bfe515a0ec732f88b382caaf2e7ec62632bb226654fbd3913f940da26f911d"} Oct 31 13:12:39.881480 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:39.881155 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ef744-79656647f-g5cd2" Oct 31 13:12:39.881480 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:39.881209 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ef744-79656647f-g5cd2" event={"ID":"bf6026f7-85c0-4659-8888-e4532b55f73a","Type":"ContainerStarted","Data":"154a47949befe794a7c91824f2158e616b8fc398abb85861e86252255c27b743"} Oct 31 13:12:39.882595 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:39.882572 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9ef744-75b9b8c75c-hzlhr" event={"ID":"430e7981-8421-4898-8bef-fba2efcb1676","Type":"ContainerStarted","Data":"01e5d6e29d41a9b9417fe7ced7bf79c95b660975d2d5bc429125bd6a577ce467"} Oct 31 13:12:39.882730 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:39.882609 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9ef744-75b9b8c75c-hzlhr" event={"ID":"430e7981-8421-4898-8bef-fba2efcb1676","Type":"ContainerStarted","Data":"2f75eff167c10f97af31400fa88bddee77e7be66d6469ee8d520ba24682e2869"} Oct 31 13:12:39.882799 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:39.882730 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-9ef744-75b9b8c75c-hzlhr" Oct 31 13:12:39.906270 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:39.906229 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ef744-79656647f-g5cd2" podStartSLOduration=1.906215702 podStartE2EDuration="1.906215702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:39.902818674 +0000 UTC m=+2410.610404970" watchObservedRunningTime="2026-10-31 13:12:39.906215702 +0000 UTC m=+2410.613802032" Oct 31 13:12:39.930043 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:39.929989 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-9ef744-75b9b8c75c-hzlhr" podStartSLOduration=1.929976994 podStartE2EDuration="1.929976994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:39.924556262 +0000 UTC m=+2410.632142535" watchObservedRunningTime="2026-10-31 13:12:39.929976994 +0000 UTC m=+2410.637563289" Oct 31 13:12:48.689580 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:12:48.689535 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:12:50.892096 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:50.892065 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ef744-79656647f-g5cd2" Oct 31 13:12:50.892457 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:50.892139 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-9ef744-75b9b8c75c-hzlhr" Oct 31 13:12:58.372066 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:58.372031 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df"] Oct 31 13:12:58.376277 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:58.376255 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" Oct 31 13:12:58.384473 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:58.384440 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-569075-openshift-default-dockercfg-lwqkn\"" Oct 31 13:12:58.401283 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:58.401255 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df"] Oct 31 13:12:58.446652 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:58.446598 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-istio-podinfo\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-562df\" (UID: \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" Oct 31 13:12:58.446853 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:58.446826 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-workload-socket\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-562df\" (UID: \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" Oct 31 13:12:58.447033 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:58.447014 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-workload-certs\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-562df\" (UID: \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" Oct 31 13:12:58.447095 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:58.447074 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-istio-data\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-562df\" (UID: \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" Oct 31 13:12:58.447224 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:58.447201 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-istio-envoy\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-562df\" (UID: \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" Oct 31 13:12:58.447290 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:58.447262 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-credential-socket\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-562df\" (UID: \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" Oct 31 13:12:58.447341 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:58.447323 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-istio-token\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-562df\" (UID: \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" Oct 31 13:12:58.447438 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:58.447420 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-istiod-ca-cert\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-562df\" (UID: \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" Oct 31 13:12:58.447491 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:58.447472 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg7z7\" (UniqueName: \"kubernetes.io/projected/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-kube-api-access-pg7z7\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-562df\" (UID: \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" Oct 31 13:12:58.549431 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:58.549397 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-workload-certs\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-562df\" (UID: \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" Oct 31 13:12:58.549619 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:58.549450 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-istio-data\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-562df\" (UID: \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" Oct 31 13:12:58.549619 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:58.549530 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-istio-envoy\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-562df\" (UID: \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" Oct 31 13:12:58.549619 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:58.549577 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-credential-socket\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-562df\" (UID: \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" Oct 31 13:12:58.549619 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:58.549599 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-istio-token\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-562df\" (UID: \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" Oct 31 13:12:58.549930 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:58.549633 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-istiod-ca-cert\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-562df\" (UID: \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" Oct 31 13:12:58.549930 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:58.549687 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg7z7\" (UniqueName: \"kubernetes.io/projected/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-kube-api-access-pg7z7\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-562df\" (UID: \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" Oct 31 13:12:58.549930 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:58.549756 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-istio-podinfo\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-562df\" (UID: \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" Oct 31 13:12:58.549930 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:58.549881 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-workload-socket\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-562df\" (UID: \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" Oct 31 13:12:58.550141 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:58.550050 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-credential-socket\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-562df\" (UID: \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" Oct 31 13:12:58.550386 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:58.550306 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-workload-certs\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-562df\" (UID: \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" Oct 31 13:12:58.550386 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:58.550368 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-istio-data\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-562df\" (UID: \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" Oct 31 13:12:58.550528 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:58.550446 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-workload-socket\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-562df\" (UID: \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" Oct 31 13:12:58.550528 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:58.550455 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-istiod-ca-cert\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-562df\" (UID: \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" Oct 31 13:12:58.552234 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:58.552211 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-istio-envoy\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-562df\" (UID: \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" Oct 31 13:12:58.552599 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:58.552581 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-istio-podinfo\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-562df\" (UID: \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" Oct 31 13:12:58.562678 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:58.562617 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-istio-token\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-562df\" (UID: \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" Oct 31 13:12:58.562884 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:58.562866 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg7z7\" (UniqueName: \"kubernetes.io/projected/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-kube-api-access-pg7z7\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-562df\" (UID: \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" Oct 31 13:12:58.691885 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:58.691845 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" Oct 31 13:12:59.011176 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:59.011145 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df"] Oct 31 13:12:59.012081 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:12:59.012048 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2f6ff04_3b39_4e0e_bb85_f41e4aac55ab.slice/crio-988fe92c11a2fe8487af03b8ec52666d8680b388b7e0109cbda51e2167d25ff3 WatchSource:0}: Error finding container 988fe92c11a2fe8487af03b8ec52666d8680b388b7e0109cbda51e2167d25ff3: Status 404 returned error can't find the container with id 988fe92c11a2fe8487af03b8ec52666d8680b388b7e0109cbda51e2167d25ff3 Oct 31 13:12:59.015753 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:59.015708 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:12:59.015891 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:59.015790 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:12:59.015891 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:59.015835 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:12:59.048461 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:12:59.048431 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" event={"ID":"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab","Type":"ContainerStarted","Data":"988fe92c11a2fe8487af03b8ec52666d8680b388b7e0109cbda51e2167d25ff3"} Oct 31 13:13:00.058693 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:00.058659 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" event={"ID":"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab","Type":"ContainerStarted","Data":"61c01bf4b1dabc339d8fbf113111f0c6be8374695a537a4d7a39824c1549d19f"} Oct 31 13:13:00.704856 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:00.700080 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" Oct 31 13:13:00.733617 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:00.732069 5693 patch_prober.go:29] interesting pod/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 31 13:13:00.733617 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:00.732151 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" podUID="b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 31 13:13:01.699667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:01.698047 5693 patch_prober.go:29] interesting pod/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 31 13:13:01.699667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:01.698106 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" podUID="b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 31 13:13:01.784667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:01.776438 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" podStartSLOduration=3.776418322 podStartE2EDuration="3.776418322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:00.102144434 +0000 UTC m=+2430.809730731" watchObservedRunningTime="2026-10-31 13:13:01.776418322 +0000 UTC m=+2432.484004617" Oct 31 13:13:01.807669 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:01.806836 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-6cef79/auth-test-model-6cef79-kserve-7645c89c97-8kjxg"] Oct 31 13:13:02.702776 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:02.702730 5693 patch_prober.go:29] interesting pod/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 31 13:13:02.703409 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:02.702789 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" podUID="b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 31 13:13:03.699666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:03.698154 5693 patch_prober.go:29] interesting pod/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 31 13:13:03.699666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:03.698220 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" podUID="b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 31 13:13:04.700670 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:04.696100 5693 patch_prober.go:29] interesting pod/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 31 13:13:04.700670 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:04.696167 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" podUID="b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 31 13:13:05.806664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:05.806131 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" Oct 31 13:13:05.808310 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:05.808282 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" Oct 31 13:13:05.809563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:05.809535 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" Oct 31 13:13:07.238722 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:07.238676 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4574ec/auth-test-model-4574ec-kserve-85d54f44b9-nzpcc"] Oct 31 13:13:12.505805 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:12.505718 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp"] Oct 31 13:13:12.520568 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:12.520535 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp" Oct 31 13:13:12.531130 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:12.529963 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-15a6a2-serving-cert\"" Oct 31 13:13:12.535468 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:12.535433 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp"] Oct 31 13:13:12.661421 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:12.661377 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a509360d-4d23-4078-b1d8-053338d5de5f-metrics-certs\") pod \"maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp\" (UID: \"a509360d-4d23-4078-b1d8-053338d5de5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp" Oct 31 13:13:12.661691 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:12.661631 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a509360d-4d23-4078-b1d8-053338d5de5f-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp\" (UID: \"a509360d-4d23-4078-b1d8-053338d5de5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp" Oct 31 13:13:12.661841 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:12.661735 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a509360d-4d23-4078-b1d8-053338d5de5f-maas-api-tls\") pod \"maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp\" (UID: \"a509360d-4d23-4078-b1d8-053338d5de5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp" Oct 31 13:13:12.661945 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:12.661913 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs9vr\" (UniqueName: \"kubernetes.io/projected/a509360d-4d23-4078-b1d8-053338d5de5f-kube-api-access-hs9vr\") pod \"maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp\" (UID: \"a509360d-4d23-4078-b1d8-053338d5de5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp" Oct 31 13:13:12.766708 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:12.764433 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a509360d-4d23-4078-b1d8-053338d5de5f-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp\" (UID: \"a509360d-4d23-4078-b1d8-053338d5de5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp" Oct 31 13:13:12.766708 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:12.764526 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a509360d-4d23-4078-b1d8-053338d5de5f-maas-api-tls\") pod \"maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp\" (UID: \"a509360d-4d23-4078-b1d8-053338d5de5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp" Oct 31 13:13:12.766708 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:12.764627 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs9vr\" (UniqueName: \"kubernetes.io/projected/a509360d-4d23-4078-b1d8-053338d5de5f-kube-api-access-hs9vr\") pod \"maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp\" (UID: \"a509360d-4d23-4078-b1d8-053338d5de5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp" Oct 31 13:13:12.766708 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:12.764742 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a509360d-4d23-4078-b1d8-053338d5de5f-metrics-certs\") pod \"maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp\" (UID: \"a509360d-4d23-4078-b1d8-053338d5de5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp" Oct 31 13:13:12.766708 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:12.765622 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a509360d-4d23-4078-b1d8-053338d5de5f-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp\" (UID: \"a509360d-4d23-4078-b1d8-053338d5de5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp" Oct 31 13:13:12.805165 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:12.804669 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a509360d-4d23-4078-b1d8-053338d5de5f-metrics-certs\") pod \"maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp\" (UID: \"a509360d-4d23-4078-b1d8-053338d5de5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp" Oct 31 13:13:12.809867 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:12.809792 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs9vr\" (UniqueName: \"kubernetes.io/projected/a509360d-4d23-4078-b1d8-053338d5de5f-kube-api-access-hs9vr\") pod \"maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp\" (UID: \"a509360d-4d23-4078-b1d8-053338d5de5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp" Oct 31 13:13:12.826342 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:12.825136 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a509360d-4d23-4078-b1d8-053338d5de5f-maas-api-tls\") pod \"maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp\" (UID: \"a509360d-4d23-4078-b1d8-053338d5de5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp" Oct 31 13:13:12.849444 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:12.849416 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp" Oct 31 13:13:13.171077 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:13.171049 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp"] Oct 31 13:13:13.172746 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:13:13.172712 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda509360d_4d23_4078_b1d8_053338d5de5f.slice/crio-329c1dda45e7f8960a0394b82c5472bb1939383bd22881ec911cdf359d95df8c WatchSource:0}: Error finding container 329c1dda45e7f8960a0394b82c5472bb1939383bd22881ec911cdf359d95df8c: Status 404 returned error can't find the container with id 329c1dda45e7f8960a0394b82c5472bb1939383bd22881ec911cdf359d95df8c Oct 31 13:13:13.210958 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:13.210920 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp" event={"ID":"a509360d-4d23-4078-b1d8-053338d5de5f","Type":"ContainerStarted","Data":"329c1dda45e7f8960a0394b82c5472bb1939383bd22881ec911cdf359d95df8c"} Oct 31 13:13:14.276834 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:14.271532 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp" event={"ID":"a509360d-4d23-4078-b1d8-053338d5de5f","Type":"ContainerStarted","Data":"cb4ca31a80de8933ba82b5ee6a176133aab9c55c551172c0be254c082533e260"} Oct 31 13:13:14.276834 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:14.271726 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp" Oct 31 13:13:14.312663 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:14.303106 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp" podStartSLOduration=1.654381591 podStartE2EDuration="2.303083966s" totalImagesPullingTime="648.702375ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:14.299287379 +0000 UTC m=+2445.006873677" watchObservedRunningTime="2026-10-31 13:13:14.303083966 +0000 UTC m=+2445.010670261" Oct 31 13:13:15.768435 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:15.768402 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf"] Oct 31 13:13:15.768978 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:15.768612 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" podUID="52b4bc93-af99-4c3e-bddb-c087f6c0247c" containerName="istio-proxy" containerID="cri-o://5f331ad908d23f77d04b95af2c98dbcb5a505fb0e1361c1097526f4088481afa" gracePeriod=30 Oct 31 13:13:16.841406 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:16.841360 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-6cef79/auth-test-model-6cef79-kserve-7645c89c97-8kjxg" podUID="8997c531-623c-4b8a-b446-79db29e64bbb" containerName="main" containerID="cri-o://d8ae1cf1a256a0aa4cfd3a65bdf521ddc8794dff1cbd5f70e81a876bcdb34595" gracePeriod=45 Oct 31 13:13:17.130003 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:17.129971 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-6cef79/auth-test-model-6cef79-kserve-7645c89c97-8kjxg" Oct 31 13:13:17.221060 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:17.221023 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8997c531-623c-4b8a-b446-79db29e64bbb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8997c531-623c-4b8a-b446-79db29e64bbb-model-cache\") pod \"8997c531-623c-4b8a-b446-79db29e64bbb\" (UID: \"8997c531-623c-4b8a-b446-79db29e64bbb\") " Oct 31 13:13:17.221241 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:17.221124 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8997c531-623c-4b8a-b446-79db29e64bbb-tls-certs\" (UniqueName: \"kubernetes.io/secret/8997c531-623c-4b8a-b446-79db29e64bbb-tls-certs\") pod \"8997c531-623c-4b8a-b446-79db29e64bbb\" (UID: \"8997c531-623c-4b8a-b446-79db29e64bbb\") " Oct 31 13:13:17.221320 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:17.221307 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8997c531-623c-4b8a-b446-79db29e64bbb-kube-api-access-nz8kx\" (UniqueName: \"kubernetes.io/projected/8997c531-623c-4b8a-b446-79db29e64bbb-kube-api-access-nz8kx\") pod \"8997c531-623c-4b8a-b446-79db29e64bbb\" (UID: \"8997c531-623c-4b8a-b446-79db29e64bbb\") " Oct 31 13:13:17.221368 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:17.221314 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8997c531-623c-4b8a-b446-79db29e64bbb-model-cache" pod "8997c531-623c-4b8a-b446-79db29e64bbb" (UID: "8997c531-623c-4b8a-b446-79db29e64bbb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:17.221368 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:17.221349 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8997c531-623c-4b8a-b446-79db29e64bbb-home\" (UniqueName: \"kubernetes.io/empty-dir/8997c531-623c-4b8a-b446-79db29e64bbb-home\") pod \"8997c531-623c-4b8a-b446-79db29e64bbb\" (UID: \"8997c531-623c-4b8a-b446-79db29e64bbb\") " Oct 31 13:13:17.221514 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:17.221492 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8997c531-623c-4b8a-b446-79db29e64bbb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8997c531-623c-4b8a-b446-79db29e64bbb-dshm\") pod \"8997c531-623c-4b8a-b446-79db29e64bbb\" (UID: \"8997c531-623c-4b8a-b446-79db29e64bbb\") " Oct 31 13:13:17.221573 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:17.221558 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8997c531-623c-4b8a-b446-79db29e64bbb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8997c531-623c-4b8a-b446-79db29e64bbb-tmp-dir\") pod \"8997c531-623c-4b8a-b446-79db29e64bbb\" (UID: \"8997c531-623c-4b8a-b446-79db29e64bbb\") " Oct 31 13:13:17.221631 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:17.221596 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8997c531-623c-4b8a-b446-79db29e64bbb-home" pod "8997c531-623c-4b8a-b446-79db29e64bbb" (UID: "8997c531-623c-4b8a-b446-79db29e64bbb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:17.221946 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:17.221917 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8997c531-623c-4b8a-b446-79db29e64bbb-tmp-dir" pod "8997c531-623c-4b8a-b446-79db29e64bbb" (UID: "8997c531-623c-4b8a-b446-79db29e64bbb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:17.222622 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:17.222599 5693 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8997c531-623c-4b8a-b446-79db29e64bbb-home\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:17.222822 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:17.222625 5693 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8997c531-623c-4b8a-b446-79db29e64bbb-tmp-dir\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:17.222822 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:17.222658 5693 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8997c531-623c-4b8a-b446-79db29e64bbb-model-cache\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:17.224947 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:17.224919 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8997c531-623c-4b8a-b446-79db29e64bbb-tls-certs" pod "8997c531-623c-4b8a-b446-79db29e64bbb" (UID: "8997c531-623c-4b8a-b446-79db29e64bbb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:13:17.225065 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:17.224980 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8997c531-623c-4b8a-b446-79db29e64bbb-kube-api-access-nz8kx" pod "8997c531-623c-4b8a-b446-79db29e64bbb" (UID: "8997c531-623c-4b8a-b446-79db29e64bbb"). InnerVolumeSpecName "kube-api-access-nz8kx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:13:17.225690 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:17.225627 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8997c531-623c-4b8a-b446-79db29e64bbb-dshm" pod "8997c531-623c-4b8a-b446-79db29e64bbb" (UID: "8997c531-623c-4b8a-b446-79db29e64bbb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:17.294969 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:17.294935 5693 generic.go:356] "Generic (PLEG): container finished" podID="8997c531-623c-4b8a-b446-79db29e64bbb" containerID="d8ae1cf1a256a0aa4cfd3a65bdf521ddc8794dff1cbd5f70e81a876bcdb34595" exitCode=0 Oct 31 13:13:17.295135 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:17.294988 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-6cef79/auth-test-model-6cef79-kserve-7645c89c97-8kjxg" Oct 31 13:13:17.295135 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:17.295021 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-6cef79/auth-test-model-6cef79-kserve-7645c89c97-8kjxg" event={"ID":"8997c531-623c-4b8a-b446-79db29e64bbb","Type":"ContainerDied","Data":"d8ae1cf1a256a0aa4cfd3a65bdf521ddc8794dff1cbd5f70e81a876bcdb34595"} Oct 31 13:13:17.295135 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:17.295072 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-6cef79/auth-test-model-6cef79-kserve-7645c89c97-8kjxg" event={"ID":"8997c531-623c-4b8a-b446-79db29e64bbb","Type":"ContainerDied","Data":"9b84a41780842db83d804217f2196f977e46ccf3a8dd9e4eac6881d908ea723d"} Oct 31 13:13:17.295135 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:17.295090 5693 scope.go:122] "RemoveContainer" containerID="d8ae1cf1a256a0aa4cfd3a65bdf521ddc8794dff1cbd5f70e81a876bcdb34595" Oct 31 13:13:17.308934 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:17.308915 5693 scope.go:122] "RemoveContainer" containerID="d8ae1cf1a256a0aa4cfd3a65bdf521ddc8794dff1cbd5f70e81a876bcdb34595" Oct 31 13:13:17.309285 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:13:17.309264 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8ae1cf1a256a0aa4cfd3a65bdf521ddc8794dff1cbd5f70e81a876bcdb34595\": container with ID starting with d8ae1cf1a256a0aa4cfd3a65bdf521ddc8794dff1cbd5f70e81a876bcdb34595 not found: ID does not exist" containerID="d8ae1cf1a256a0aa4cfd3a65bdf521ddc8794dff1cbd5f70e81a876bcdb34595" Oct 31 13:13:17.309388 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:17.309293 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8ae1cf1a256a0aa4cfd3a65bdf521ddc8794dff1cbd5f70e81a876bcdb34595"} err="failed to get container status \"d8ae1cf1a256a0aa4cfd3a65bdf521ddc8794dff1cbd5f70e81a876bcdb34595\": rpc error: code = NotFound desc = could not find container \"d8ae1cf1a256a0aa4cfd3a65bdf521ddc8794dff1cbd5f70e81a876bcdb34595\": container with ID starting with d8ae1cf1a256a0aa4cfd3a65bdf521ddc8794dff1cbd5f70e81a876bcdb34595 not found: ID does not exist" Oct 31 13:13:17.324356 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:17.324322 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz8kx\" (UniqueName: \"kubernetes.io/projected/8997c531-623c-4b8a-b446-79db29e64bbb-kube-api-access-nz8kx\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:17.324511 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:17.324359 5693 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8997c531-623c-4b8a-b446-79db29e64bbb-dshm\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:17.324511 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:17.324375 5693 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8997c531-623c-4b8a-b446-79db29e64bbb-tls-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:17.347670 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:17.347579 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-6cef79/auth-test-model-6cef79-kserve-7645c89c97-8kjxg"] Oct 31 13:13:17.360582 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:17.360536 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-6cef79/auth-test-model-6cef79-kserve-7645c89c97-8kjxg"] Oct 31 13:13:17.411766 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:17.411734 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8997c531-623c-4b8a-b446-79db29e64bbb" path="/var/lib/kubelet/pods/8997c531-623c-4b8a-b446-79db29e64bbb/volumes" Oct 31 13:13:18.139415 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:18.139387 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5dc570-5cfd7479bc-csb46"] Oct 31 13:13:18.140142 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:18.140128 5693 cpu_manager.go:414] "containerMap: removing container" podUID="8997c531-623c-4b8a-b446-79db29e64bbb" containerName="main" Oct 31 13:13:18.140199 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:18.140143 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8997c531-623c-4b8a-b446-79db29e64bbb" containerName="main" Oct 31 13:13:18.140298 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:18.140290 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="8997c531-623c-4b8a-b446-79db29e64bbb" containerName="main" Oct 31 13:13:18.145100 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:18.145084 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5dc570-5cfd7479bc-csb46" Oct 31 13:13:18.169626 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:18.169600 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-5dc570\"" Oct 31 13:13:18.172574 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:18.172549 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5dc570-5cfd7479bc-csb46"] Oct 31 13:13:18.265697 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:18.255971 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-5dc570-dockercfg-ld59t\"" Oct 31 13:13:18.302668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:18.301734 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ad184ee-c710-47b0-9d44-b4bbcb5487e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-5dc570-5cfd7479bc-csb46\" (UID: \"1ad184ee-c710-47b0-9d44-b4bbcb5487e5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5dc570-5cfd7479bc-csb46" Oct 31 13:13:18.302668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:18.301789 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks8kq\" (UniqueName: \"kubernetes.io/projected/1ad184ee-c710-47b0-9d44-b4bbcb5487e5-kube-api-access-ks8kq\") pod \"payload-pre-processing-e2e-extmod-a-5dc570-5cfd7479bc-csb46\" (UID: \"1ad184ee-c710-47b0-9d44-b4bbcb5487e5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5dc570-5cfd7479bc-csb46" Oct 31 13:13:18.409688 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:18.409546 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ad184ee-c710-47b0-9d44-b4bbcb5487e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-5dc570-5cfd7479bc-csb46\" (UID: \"1ad184ee-c710-47b0-9d44-b4bbcb5487e5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5dc570-5cfd7479bc-csb46" Oct 31 13:13:18.409688 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:18.409624 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ad184ee-c710-47b0-9d44-b4bbcb5487e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-5dc570-5cfd7479bc-csb46\" (UID: \"1ad184ee-c710-47b0-9d44-b4bbcb5487e5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5dc570-5cfd7479bc-csb46" Oct 31 13:13:18.409977 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:18.409726 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks8kq\" (UniqueName: \"kubernetes.io/projected/1ad184ee-c710-47b0-9d44-b4bbcb5487e5-kube-api-access-ks8kq\") pod \"payload-pre-processing-e2e-extmod-a-5dc570-5cfd7479bc-csb46\" (UID: \"1ad184ee-c710-47b0-9d44-b4bbcb5487e5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5dc570-5cfd7479bc-csb46" Oct 31 13:13:18.432500 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:18.432469 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks8kq\" (UniqueName: \"kubernetes.io/projected/1ad184ee-c710-47b0-9d44-b4bbcb5487e5-kube-api-access-ks8kq\") pod \"payload-pre-processing-e2e-extmod-a-5dc570-5cfd7479bc-csb46\" (UID: \"1ad184ee-c710-47b0-9d44-b4bbcb5487e5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5dc570-5cfd7479bc-csb46" Oct 31 13:13:18.460683 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:18.460021 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5dc570-5cfd7479bc-csb46" Oct 31 13:13:18.690501 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:13:18.690419 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:13:18.773240 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:13:18.773196 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ad184ee_c710_47b0_9d44_b4bbcb5487e5.slice/crio-7e9aa40fa9606a611cee6afb8907f61a89e741f826bf41e5eeace828d3252821 WatchSource:0}: Error finding container 7e9aa40fa9606a611cee6afb8907f61a89e741f826bf41e5eeace828d3252821: Status 404 returned error can't find the container with id 7e9aa40fa9606a611cee6afb8907f61a89e741f826bf41e5eeace828d3252821 Oct 31 13:13:18.773630 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:18.773602 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5dc570-5cfd7479bc-csb46"] Oct 31 13:13:19.421131 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:19.420970 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5dc570-5cfd7479bc-csb46" Oct 31 13:13:19.421131 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:19.421031 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5dc570-5cfd7479bc-csb46" event={"ID":"1ad184ee-c710-47b0-9d44-b4bbcb5487e5","Type":"ContainerStarted","Data":"f82d62c9bb362858d5fad6d0f1ff1e6b390ddd1614f0483aa73b95bda97ed116"} Oct 31 13:13:19.421131 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:19.421062 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5dc570-5cfd7479bc-csb46" event={"ID":"1ad184ee-c710-47b0-9d44-b4bbcb5487e5","Type":"ContainerStarted","Data":"7e9aa40fa9606a611cee6afb8907f61a89e741f826bf41e5eeace828d3252821"} Oct 31 13:13:19.448222 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:19.448169 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5dc570-5cfd7479bc-csb46" podStartSLOduration=1.448153753 podStartE2EDuration="1.448153753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:19.446978903 +0000 UTC m=+2450.154565198" watchObservedRunningTime="2026-10-31 13:13:19.448153753 +0000 UTC m=+2450.155740046" Oct 31 13:13:20.285596 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:20.285567 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp" Oct 31 13:13:21.128888 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:21.128818 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" Oct 31 13:13:21.318248 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:21.318210 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52b4bc93-af99-4c3e-bddb-c087f6c0247c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52b4bc93-af99-4c3e-bddb-c087f6c0247c-istiod-ca-cert\") pod \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\" (UID: \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\") " Oct 31 13:13:21.318447 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:21.318274 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52b4bc93-af99-4c3e-bddb-c087f6c0247c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52b4bc93-af99-4c3e-bddb-c087f6c0247c-istio-envoy\") pod \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\" (UID: \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\") " Oct 31 13:13:21.318447 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:21.318336 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52b4bc93-af99-4c3e-bddb-c087f6c0247c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b4bc93-af99-4c3e-bddb-c087f6c0247c-credential-socket\") pod \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\" (UID: \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\") " Oct 31 13:13:21.318447 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:21.318392 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52b4bc93-af99-4c3e-bddb-c087f6c0247c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52b4bc93-af99-4c3e-bddb-c087f6c0247c-istio-podinfo\") pod \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\" (UID: \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\") " Oct 31 13:13:21.318617 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:21.318500 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52b4bc93-af99-4c3e-bddb-c087f6c0247c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52b4bc93-af99-4c3e-bddb-c087f6c0247c-workload-certs\") pod \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\" (UID: \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\") " Oct 31 13:13:21.318617 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:21.318573 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52b4bc93-af99-4c3e-bddb-c087f6c0247c-kube-api-access-p5lxt\" (UniqueName: \"kubernetes.io/projected/52b4bc93-af99-4c3e-bddb-c087f6c0247c-kube-api-access-p5lxt\") pod \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\" (UID: \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\") " Oct 31 13:13:21.318788 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:21.318625 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52b4bc93-af99-4c3e-bddb-c087f6c0247c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b4bc93-af99-4c3e-bddb-c087f6c0247c-workload-socket\") pod \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\" (UID: \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\") " Oct 31 13:13:21.318788 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:21.318570 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52b4bc93-af99-4c3e-bddb-c087f6c0247c-istiod-ca-cert" pod "52b4bc93-af99-4c3e-bddb-c087f6c0247c" (UID: "52b4bc93-af99-4c3e-bddb-c087f6c0247c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:13:21.318788 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:21.318680 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52b4bc93-af99-4c3e-bddb-c087f6c0247c-credential-socket" pod "52b4bc93-af99-4c3e-bddb-c087f6c0247c" (UID: "52b4bc93-af99-4c3e-bddb-c087f6c0247c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:21.318958 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:21.318807 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52b4bc93-af99-4c3e-bddb-c087f6c0247c-workload-certs" pod "52b4bc93-af99-4c3e-bddb-c087f6c0247c" (UID: "52b4bc93-af99-4c3e-bddb-c087f6c0247c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:21.318958 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:21.318820 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52b4bc93-af99-4c3e-bddb-c087f6c0247c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52b4bc93-af99-4c3e-bddb-c087f6c0247c-istio-data\") pod \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\" (UID: \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\") " Oct 31 13:13:21.318958 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:21.318859 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52b4bc93-af99-4c3e-bddb-c087f6c0247c-istio-token\" (UniqueName: \"kubernetes.io/projected/52b4bc93-af99-4c3e-bddb-c087f6c0247c-istio-token\") pod \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\" (UID: \"52b4bc93-af99-4c3e-bddb-c087f6c0247c\") " Oct 31 13:13:21.318958 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:21.318914 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52b4bc93-af99-4c3e-bddb-c087f6c0247c-workload-socket" pod "52b4bc93-af99-4c3e-bddb-c087f6c0247c" (UID: "52b4bc93-af99-4c3e-bddb-c087f6c0247c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:21.319123 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:21.319007 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52b4bc93-af99-4c3e-bddb-c087f6c0247c-istio-data" pod "52b4bc93-af99-4c3e-bddb-c087f6c0247c" (UID: "52b4bc93-af99-4c3e-bddb-c087f6c0247c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:21.319547 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:21.319524 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52b4bc93-af99-4c3e-bddb-c087f6c0247c-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:21.319547 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:21.319545 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b4bc93-af99-4c3e-bddb-c087f6c0247c-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:21.320034 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:21.319556 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52b4bc93-af99-4c3e-bddb-c087f6c0247c-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:21.320034 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:21.319565 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b4bc93-af99-4c3e-bddb-c087f6c0247c-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:21.320034 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:21.319574 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52b4bc93-af99-4c3e-bddb-c087f6c0247c-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:21.321240 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:21.321214 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52b4bc93-af99-4c3e-bddb-c087f6c0247c-kube-api-access-p5lxt" pod "52b4bc93-af99-4c3e-bddb-c087f6c0247c" (UID: "52b4bc93-af99-4c3e-bddb-c087f6c0247c"). InnerVolumeSpecName "kube-api-access-p5lxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:13:21.321361 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:21.321284 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52b4bc93-af99-4c3e-bddb-c087f6c0247c-istio-token" pod "52b4bc93-af99-4c3e-bddb-c087f6c0247c" (UID: "52b4bc93-af99-4c3e-bddb-c087f6c0247c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:13:21.321361 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:21.321347 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52b4bc93-af99-4c3e-bddb-c087f6c0247c-istio-envoy" pod "52b4bc93-af99-4c3e-bddb-c087f6c0247c" (UID: "52b4bc93-af99-4c3e-bddb-c087f6c0247c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:21.321361 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:21.321345 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52b4bc93-af99-4c3e-bddb-c087f6c0247c-istio-podinfo" pod "52b4bc93-af99-4c3e-bddb-c087f6c0247c" (UID: "52b4bc93-af99-4c3e-bddb-c087f6c0247c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:13:21.429692 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:21.427585 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52b4bc93-af99-4c3e-bddb-c087f6c0247c-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:21.429692 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:21.427617 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52b4bc93-af99-4c3e-bddb-c087f6c0247c-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:21.429692 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:21.427630 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52b4bc93-af99-4c3e-bddb-c087f6c0247c-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:21.429692 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:21.427662 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5lxt\" (UniqueName: \"kubernetes.io/projected/52b4bc93-af99-4c3e-bddb-c087f6c0247c-kube-api-access-p5lxt\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:21.447105 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:21.447071 5693 generic.go:356] "Generic (PLEG): container finished" podID="52b4bc93-af99-4c3e-bddb-c087f6c0247c" containerID="5f331ad908d23f77d04b95af2c98dbcb5a505fb0e1361c1097526f4088481afa" exitCode=0 Oct 31 13:13:21.447294 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:21.447190 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" event={"ID":"52b4bc93-af99-4c3e-bddb-c087f6c0247c","Type":"ContainerDied","Data":"5f331ad908d23f77d04b95af2c98dbcb5a505fb0e1361c1097526f4088481afa"} Oct 31 13:13:21.447370 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:21.447303 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" event={"ID":"52b4bc93-af99-4c3e-bddb-c087f6c0247c","Type":"ContainerDied","Data":"8ea044d3f0163207bd72d3b9db2791c59fc04a586b8b38c642c8b80f8ea14d44"} Oct 31 13:13:21.447370 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:21.447337 5693 scope.go:122] "RemoveContainer" containerID="5f331ad908d23f77d04b95af2c98dbcb5a505fb0e1361c1097526f4088481afa" Oct 31 13:13:21.447507 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:21.447155 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf" Oct 31 13:13:21.462678 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:21.462624 5693 scope.go:122] "RemoveContainer" containerID="5f331ad908d23f77d04b95af2c98dbcb5a505fb0e1361c1097526f4088481afa" Oct 31 13:13:21.463033 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:13:21.463008 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f331ad908d23f77d04b95af2c98dbcb5a505fb0e1361c1097526f4088481afa\": container with ID starting with 5f331ad908d23f77d04b95af2c98dbcb5a505fb0e1361c1097526f4088481afa not found: ID does not exist" containerID="5f331ad908d23f77d04b95af2c98dbcb5a505fb0e1361c1097526f4088481afa" Oct 31 13:13:21.463107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:21.463042 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f331ad908d23f77d04b95af2c98dbcb5a505fb0e1361c1097526f4088481afa"} err="failed to get container status \"5f331ad908d23f77d04b95af2c98dbcb5a505fb0e1361c1097526f4088481afa\": rpc error: code = NotFound desc = could not find container \"5f331ad908d23f77d04b95af2c98dbcb5a505fb0e1361c1097526f4088481afa\": container with ID starting with 5f331ad908d23f77d04b95af2c98dbcb5a505fb0e1361c1097526f4088481afa not found: ID does not exist" Oct 31 13:13:21.480531 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:21.480502 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf"] Oct 31 13:13:21.488419 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:21.488389 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-gt5rf"] Oct 31 13:13:22.268873 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:22.268824 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-4574ec/auth-test-model-4574ec-kserve-85d54f44b9-nzpcc" podUID="a2085991-6765-4533-aec8-b0f40de40642" containerName="main" containerID="cri-o://c3e4fca749625bd6a320e80c9874ed83d6c86a402be95d6d1674d7d5d576ae38" gracePeriod=45 Oct 31 13:13:22.460928 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:22.460895 5693 generic.go:356] "Generic (PLEG): container finished" podID="a2085991-6765-4533-aec8-b0f40de40642" containerID="c3e4fca749625bd6a320e80c9874ed83d6c86a402be95d6d1674d7d5d576ae38" exitCode=0 Oct 31 13:13:22.461058 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:22.460988 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4574ec/auth-test-model-4574ec-kserve-85d54f44b9-nzpcc" event={"ID":"a2085991-6765-4533-aec8-b0f40de40642","Type":"ContainerDied","Data":"c3e4fca749625bd6a320e80c9874ed83d6c86a402be95d6d1674d7d5d576ae38"} Oct 31 13:13:22.577720 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:22.577695 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4574ec/auth-test-model-4574ec-kserve-85d54f44b9-nzpcc" Oct 31 13:13:22.650381 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:22.650352 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2085991-6765-4533-aec8-b0f40de40642-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2085991-6765-4533-aec8-b0f40de40642-model-cache\") pod \"a2085991-6765-4533-aec8-b0f40de40642\" (UID: \"a2085991-6765-4533-aec8-b0f40de40642\") " Oct 31 13:13:22.650576 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:22.650396 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2085991-6765-4533-aec8-b0f40de40642-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2085991-6765-4533-aec8-b0f40de40642-tmp-dir\") pod \"a2085991-6765-4533-aec8-b0f40de40642\" (UID: \"a2085991-6765-4533-aec8-b0f40de40642\") " Oct 31 13:13:22.650576 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:22.650512 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2085991-6765-4533-aec8-b0f40de40642-home\" (UniqueName: \"kubernetes.io/empty-dir/a2085991-6765-4533-aec8-b0f40de40642-home\") pod \"a2085991-6765-4533-aec8-b0f40de40642\" (UID: \"a2085991-6765-4533-aec8-b0f40de40642\") " Oct 31 13:13:22.650833 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:22.650583 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2085991-6765-4533-aec8-b0f40de40642-tls-certs\" (UniqueName: \"kubernetes.io/secret/a2085991-6765-4533-aec8-b0f40de40642-tls-certs\") pod \"a2085991-6765-4533-aec8-b0f40de40642\" (UID: \"a2085991-6765-4533-aec8-b0f40de40642\") " Oct 31 13:13:22.651000 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:22.650976 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2085991-6765-4533-aec8-b0f40de40642-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2085991-6765-4533-aec8-b0f40de40642-dshm\") pod \"a2085991-6765-4533-aec8-b0f40de40642\" (UID: \"a2085991-6765-4533-aec8-b0f40de40642\") " Oct 31 13:13:22.651097 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:22.651078 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2085991-6765-4533-aec8-b0f40de40642-kube-api-access-66wjh\" (UniqueName: \"kubernetes.io/projected/a2085991-6765-4533-aec8-b0f40de40642-kube-api-access-66wjh\") pod \"a2085991-6765-4533-aec8-b0f40de40642\" (UID: \"a2085991-6765-4533-aec8-b0f40de40642\") " Oct 31 13:13:22.651471 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:22.651446 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2085991-6765-4533-aec8-b0f40de40642-model-cache" pod "a2085991-6765-4533-aec8-b0f40de40642" (UID: "a2085991-6765-4533-aec8-b0f40de40642"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:22.652017 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:22.651994 5693 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2085991-6765-4533-aec8-b0f40de40642-model-cache\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:22.652613 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:22.652589 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2085991-6765-4533-aec8-b0f40de40642-tmp-dir" pod "a2085991-6765-4533-aec8-b0f40de40642" (UID: "a2085991-6765-4533-aec8-b0f40de40642"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:22.653119 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:22.653099 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2085991-6765-4533-aec8-b0f40de40642-home" pod "a2085991-6765-4533-aec8-b0f40de40642" (UID: "a2085991-6765-4533-aec8-b0f40de40642"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:22.687446 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:22.657831 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2085991-6765-4533-aec8-b0f40de40642-kube-api-access-66wjh" pod "a2085991-6765-4533-aec8-b0f40de40642" (UID: "a2085991-6765-4533-aec8-b0f40de40642"). InnerVolumeSpecName "kube-api-access-66wjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:13:22.687446 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:22.657934 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2085991-6765-4533-aec8-b0f40de40642-dshm" pod "a2085991-6765-4533-aec8-b0f40de40642" (UID: "a2085991-6765-4533-aec8-b0f40de40642"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:22.687446 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:22.658020 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2085991-6765-4533-aec8-b0f40de40642-tls-certs" pod "a2085991-6765-4533-aec8-b0f40de40642" (UID: "a2085991-6765-4533-aec8-b0f40de40642"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:13:22.756700 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:22.753925 5693 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2085991-6765-4533-aec8-b0f40de40642-home\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:22.756700 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:22.753959 5693 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2085991-6765-4533-aec8-b0f40de40642-tls-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:22.756700 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:22.753972 5693 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2085991-6765-4533-aec8-b0f40de40642-dshm\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:22.756700 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:22.753986 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66wjh\" (UniqueName: \"kubernetes.io/projected/a2085991-6765-4533-aec8-b0f40de40642-kube-api-access-66wjh\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:22.756700 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:22.754001 5693 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2085991-6765-4533-aec8-b0f40de40642-tmp-dir\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:23.410093 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:23.410061 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52b4bc93-af99-4c3e-bddb-c087f6c0247c" path="/var/lib/kubelet/pods/52b4bc93-af99-4c3e-bddb-c087f6c0247c/volumes" Oct 31 13:13:23.469500 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:23.469475 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4574ec/auth-test-model-4574ec-kserve-85d54f44b9-nzpcc" Oct 31 13:13:23.469736 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:23.469513 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4574ec/auth-test-model-4574ec-kserve-85d54f44b9-nzpcc" event={"ID":"a2085991-6765-4533-aec8-b0f40de40642","Type":"ContainerDied","Data":"460e36cb64c3a117e7eb5a9c5ebebba6b041fc70ae4e75a2ff5684ace051a9d7"} Oct 31 13:13:23.469736 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:23.469570 5693 scope.go:122] "RemoveContainer" containerID="c3e4fca749625bd6a320e80c9874ed83d6c86a402be95d6d1674d7d5d576ae38" Oct 31 13:13:23.505226 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:23.505191 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4574ec/auth-test-model-4574ec-kserve-85d54f44b9-nzpcc"] Oct 31 13:13:23.512018 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:23.511990 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4574ec/auth-test-model-4574ec-kserve-85d54f44b9-nzpcc"] Oct 31 13:13:23.546930 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:23.546889 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-4574ec/auth-test-model-4574ec-kserve-85d54f44b9-nzpcc" podUID="a2085991-6765-4533-aec8-b0f40de40642" containerName="main" probeResult="failure" output="Get \"https://10.133.0.152:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:13:24.097485 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:24.097439 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-fa8c3f-78f95764bc-p8tx5"] Oct 31 13:13:24.098662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:24.098621 5693 cpu_manager.go:414] "containerMap: removing container" podUID="a2085991-6765-4533-aec8-b0f40de40642" containerName="main" Oct 31 13:13:24.098729 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:24.098667 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2085991-6765-4533-aec8-b0f40de40642" containerName="main" Oct 31 13:13:24.098729 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:24.098705 5693 cpu_manager.go:414] "containerMap: removing container" podUID="52b4bc93-af99-4c3e-bddb-c087f6c0247c" containerName="istio-proxy" Oct 31 13:13:24.098729 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:24.098713 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52b4bc93-af99-4c3e-bddb-c087f6c0247c" containerName="istio-proxy" Oct 31 13:13:24.098916 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:24.098901 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2085991-6765-4533-aec8-b0f40de40642" containerName="main" Oct 31 13:13:24.098977 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:24.098924 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="52b4bc93-af99-4c3e-bddb-c087f6c0247c" containerName="istio-proxy" Oct 31 13:13:24.104143 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:24.104122 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-fa8c3f-78f95764bc-p8tx5" Oct 31 13:13:24.114719 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:24.114682 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-fa8c3f-78f95764bc-p8tx5"] Oct 31 13:13:24.115294 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:24.115273 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-fa8c3f\"" Oct 31 13:13:24.283811 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:24.283763 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9264e5f3-1234-47de-97f9-020dfb0b03f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-fa8c3f-78f95764bc-p8tx5\" (UID: \"9264e5f3-1234-47de-97f9-020dfb0b03f1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-fa8c3f-78f95764bc-p8tx5" Oct 31 13:13:24.284458 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:24.284363 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzrlt\" (UniqueName: \"kubernetes.io/projected/9264e5f3-1234-47de-97f9-020dfb0b03f1-kube-api-access-mzrlt\") pod \"payload-pre-processing-e2e-autores-fa8c3f-78f95764bc-p8tx5\" (UID: \"9264e5f3-1234-47de-97f9-020dfb0b03f1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-fa8c3f-78f95764bc-p8tx5" Oct 31 13:13:24.336683 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:24.336626 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-fa8c3f-6bc56f9c6c-r4ws4"] Oct 31 13:13:24.391502 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:24.389817 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9264e5f3-1234-47de-97f9-020dfb0b03f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-fa8c3f-78f95764bc-p8tx5\" (UID: \"9264e5f3-1234-47de-97f9-020dfb0b03f1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-fa8c3f-78f95764bc-p8tx5" Oct 31 13:13:24.391502 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:24.389974 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzrlt\" (UniqueName: \"kubernetes.io/projected/9264e5f3-1234-47de-97f9-020dfb0b03f1-kube-api-access-mzrlt\") pod \"payload-pre-processing-e2e-autores-fa8c3f-78f95764bc-p8tx5\" (UID: \"9264e5f3-1234-47de-97f9-020dfb0b03f1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-fa8c3f-78f95764bc-p8tx5" Oct 31 13:13:24.391502 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:24.391371 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9264e5f3-1234-47de-97f9-020dfb0b03f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-fa8c3f-78f95764bc-p8tx5\" (UID: \"9264e5f3-1234-47de-97f9-020dfb0b03f1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-fa8c3f-78f95764bc-p8tx5" Oct 31 13:13:24.430633 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:24.412975 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-fa8c3f-6bc56f9c6c-r4ws4" Oct 31 13:13:24.432392 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:24.432371 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-fa8c3f-6bc56f9c6c-r4ws4"] Oct 31 13:13:24.439399 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:24.438538 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-fa8c3f-dockercfg-l42tv\"" Oct 31 13:13:24.458207 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:24.458184 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzrlt\" (UniqueName: \"kubernetes.io/projected/9264e5f3-1234-47de-97f9-020dfb0b03f1-kube-api-access-mzrlt\") pod \"payload-pre-processing-e2e-autores-fa8c3f-78f95764bc-p8tx5\" (UID: \"9264e5f3-1234-47de-97f9-020dfb0b03f1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-fa8c3f-78f95764bc-p8tx5" Oct 31 13:13:24.592263 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:24.592221 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46d567ad-1dde-4d69-9ed1-8a2cab56e2db-plugins-config-volume\") pod \"payload-processing-e2e-autores-fa8c3f-6bc56f9c6c-r4ws4\" (UID: \"46d567ad-1dde-4d69-9ed1-8a2cab56e2db\") " pod="openshift-ingress/payload-processing-e2e-autores-fa8c3f-6bc56f9c6c-r4ws4" Oct 31 13:13:24.592480 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:24.592457 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77b65\" (UniqueName: \"kubernetes.io/projected/46d567ad-1dde-4d69-9ed1-8a2cab56e2db-kube-api-access-77b65\") pod \"payload-processing-e2e-autores-fa8c3f-6bc56f9c6c-r4ws4\" (UID: \"46d567ad-1dde-4d69-9ed1-8a2cab56e2db\") " pod="openshift-ingress/payload-processing-e2e-autores-fa8c3f-6bc56f9c6c-r4ws4" Oct 31 13:13:24.694010 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:24.693975 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46d567ad-1dde-4d69-9ed1-8a2cab56e2db-plugins-config-volume\") pod \"payload-processing-e2e-autores-fa8c3f-6bc56f9c6c-r4ws4\" (UID: \"46d567ad-1dde-4d69-9ed1-8a2cab56e2db\") " pod="openshift-ingress/payload-processing-e2e-autores-fa8c3f-6bc56f9c6c-r4ws4" Oct 31 13:13:24.694209 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:24.694177 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77b65\" (UniqueName: \"kubernetes.io/projected/46d567ad-1dde-4d69-9ed1-8a2cab56e2db-kube-api-access-77b65\") pod \"payload-processing-e2e-autores-fa8c3f-6bc56f9c6c-r4ws4\" (UID: \"46d567ad-1dde-4d69-9ed1-8a2cab56e2db\") " pod="openshift-ingress/payload-processing-e2e-autores-fa8c3f-6bc56f9c6c-r4ws4" Oct 31 13:13:24.694766 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:24.694745 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46d567ad-1dde-4d69-9ed1-8a2cab56e2db-plugins-config-volume\") pod \"payload-processing-e2e-autores-fa8c3f-6bc56f9c6c-r4ws4\" (UID: \"46d567ad-1dde-4d69-9ed1-8a2cab56e2db\") " pod="openshift-ingress/payload-processing-e2e-autores-fa8c3f-6bc56f9c6c-r4ws4" Oct 31 13:13:24.715454 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:24.715421 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77b65\" (UniqueName: \"kubernetes.io/projected/46d567ad-1dde-4d69-9ed1-8a2cab56e2db-kube-api-access-77b65\") pod \"payload-processing-e2e-autores-fa8c3f-6bc56f9c6c-r4ws4\" (UID: \"46d567ad-1dde-4d69-9ed1-8a2cab56e2db\") " pod="openshift-ingress/payload-processing-e2e-autores-fa8c3f-6bc56f9c6c-r4ws4" Oct 31 13:13:24.745090 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:24.745058 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-fa8c3f-78f95764bc-p8tx5" Oct 31 13:13:24.769822 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:24.769773 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-fa8c3f-6bc56f9c6c-r4ws4" Oct 31 13:13:25.056484 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:13:25.056439 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9264e5f3_1234_47de_97f9_020dfb0b03f1.slice/crio-43e64baecd174305d98340cd15f65dce59765cab4d7c740d710b04aed22055ec WatchSource:0}: Error finding container 43e64baecd174305d98340cd15f65dce59765cab4d7c740d710b04aed22055ec: Status 404 returned error can't find the container with id 43e64baecd174305d98340cd15f65dce59765cab4d7c740d710b04aed22055ec Oct 31 13:13:25.057027 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:25.056890 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-fa8c3f-78f95764bc-p8tx5"] Oct 31 13:13:25.096367 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:25.096338 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-fa8c3f-6bc56f9c6c-r4ws4"] Oct 31 13:13:25.100223 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:13:25.100187 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46d567ad_1dde_4d69_9ed1_8a2cab56e2db.slice/crio-4c424d822d3faab13c825a33a8d50d955e87963ff90022e686e86efe68f39730 WatchSource:0}: Error finding container 4c424d822d3faab13c825a33a8d50d955e87963ff90022e686e86efe68f39730: Status 404 returned error can't find the container with id 4c424d822d3faab13c825a33a8d50d955e87963ff90022e686e86efe68f39730 Oct 31 13:13:25.341131 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:25.341047 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-29e3a0-b4ccb7d5f-2z6dt"] Oct 31 13:13:25.345987 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:25.345963 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-29e3a0-b4ccb7d5f-2z6dt" Oct 31 13:13:25.355399 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:25.354297 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-29e3a0\"" Oct 31 13:13:25.355399 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:25.355075 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-29e3a0-dockercfg-gc7xm\"" Oct 31 13:13:25.359229 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:25.359200 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-29e3a0-b4ccb7d5f-2z6dt"] Oct 31 13:13:25.386227 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:25.385816 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-29e3a0-59874b6cdf-7hvmk"] Oct 31 13:13:25.391816 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:25.391792 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-29e3a0-59874b6cdf-7hvmk" Oct 31 13:13:25.406126 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:25.405951 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zksb\" (UniqueName: \"kubernetes.io/projected/e18dd291-4694-41ad-922c-8852d7c12960-kube-api-access-6zksb\") pod \"payload-pre-processing-e2e-extmod-b-29e3a0-b4ccb7d5f-2z6dt\" (UID: \"e18dd291-4694-41ad-922c-8852d7c12960\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-29e3a0-b4ccb7d5f-2z6dt" Oct 31 13:13:25.406126 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:25.406081 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e18dd291-4694-41ad-922c-8852d7c12960-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-29e3a0-b4ccb7d5f-2z6dt\" (UID: \"e18dd291-4694-41ad-922c-8852d7c12960\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-29e3a0-b4ccb7d5f-2z6dt" Oct 31 13:13:25.406343 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:25.406241 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtlj4\" (UniqueName: \"kubernetes.io/projected/5458e7f2-d785-4a4e-9b45-e5a9736b6758-kube-api-access-qtlj4\") pod \"payload-processing-e2e-extmod-b-29e3a0-59874b6cdf-7hvmk\" (UID: \"5458e7f2-d785-4a4e-9b45-e5a9736b6758\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-29e3a0-59874b6cdf-7hvmk" Oct 31 13:13:25.406440 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:25.406418 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5458e7f2-d785-4a4e-9b45-e5a9736b6758-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-29e3a0-59874b6cdf-7hvmk\" (UID: \"5458e7f2-d785-4a4e-9b45-e5a9736b6758\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-29e3a0-59874b6cdf-7hvmk" Oct 31 13:13:25.420575 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:25.420538 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2085991-6765-4533-aec8-b0f40de40642" path="/var/lib/kubelet/pods/a2085991-6765-4533-aec8-b0f40de40642/volumes" Oct 31 13:13:25.421073 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:25.421049 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-29e3a0-59874b6cdf-7hvmk"] Oct 31 13:13:25.531666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:25.530744 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e18dd291-4694-41ad-922c-8852d7c12960-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-29e3a0-b4ccb7d5f-2z6dt\" (UID: \"e18dd291-4694-41ad-922c-8852d7c12960\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-29e3a0-b4ccb7d5f-2z6dt" Oct 31 13:13:25.531666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:25.530857 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtlj4\" (UniqueName: \"kubernetes.io/projected/5458e7f2-d785-4a4e-9b45-e5a9736b6758-kube-api-access-qtlj4\") pod \"payload-processing-e2e-extmod-b-29e3a0-59874b6cdf-7hvmk\" (UID: \"5458e7f2-d785-4a4e-9b45-e5a9736b6758\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-29e3a0-59874b6cdf-7hvmk" Oct 31 13:13:25.531666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:25.531008 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5458e7f2-d785-4a4e-9b45-e5a9736b6758-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-29e3a0-59874b6cdf-7hvmk\" (UID: \"5458e7f2-d785-4a4e-9b45-e5a9736b6758\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-29e3a0-59874b6cdf-7hvmk" Oct 31 13:13:25.531666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:25.531067 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zksb\" (UniqueName: \"kubernetes.io/projected/e18dd291-4694-41ad-922c-8852d7c12960-kube-api-access-6zksb\") pod \"payload-pre-processing-e2e-extmod-b-29e3a0-b4ccb7d5f-2z6dt\" (UID: \"e18dd291-4694-41ad-922c-8852d7c12960\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-29e3a0-b4ccb7d5f-2z6dt" Oct 31 13:13:25.556728 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:25.533012 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e18dd291-4694-41ad-922c-8852d7c12960-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-29e3a0-b4ccb7d5f-2z6dt\" (UID: \"e18dd291-4694-41ad-922c-8852d7c12960\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-29e3a0-b4ccb7d5f-2z6dt" Oct 31 13:13:25.556728 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:25.555952 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5458e7f2-d785-4a4e-9b45-e5a9736b6758-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-29e3a0-59874b6cdf-7hvmk\" (UID: \"5458e7f2-d785-4a4e-9b45-e5a9736b6758\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-29e3a0-59874b6cdf-7hvmk" Oct 31 13:13:25.668826 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:25.637392 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zksb\" (UniqueName: \"kubernetes.io/projected/e18dd291-4694-41ad-922c-8852d7c12960-kube-api-access-6zksb\") pod \"payload-pre-processing-e2e-extmod-b-29e3a0-b4ccb7d5f-2z6dt\" (UID: \"e18dd291-4694-41ad-922c-8852d7c12960\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-29e3a0-b4ccb7d5f-2z6dt" Oct 31 13:13:25.703462 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:25.703407 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtlj4\" (UniqueName: \"kubernetes.io/projected/5458e7f2-d785-4a4e-9b45-e5a9736b6758-kube-api-access-qtlj4\") pod \"payload-processing-e2e-extmod-b-29e3a0-59874b6cdf-7hvmk\" (UID: \"5458e7f2-d785-4a4e-9b45-e5a9736b6758\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-29e3a0-59874b6cdf-7hvmk" Oct 31 13:13:25.719687 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:25.714630 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-fa8c3f-78f95764bc-p8tx5" event={"ID":"9264e5f3-1234-47de-97f9-020dfb0b03f1","Type":"ContainerStarted","Data":"75fb75e159216359bba5cd7733a4c01140f24da8c222dcbfcbc389d735eb12ef"} Oct 31 13:13:25.719687 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:25.714718 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-fa8c3f-78f95764bc-p8tx5" event={"ID":"9264e5f3-1234-47de-97f9-020dfb0b03f1","Type":"ContainerStarted","Data":"43e64baecd174305d98340cd15f65dce59765cab4d7c740d710b04aed22055ec"} Oct 31 13:13:25.719687 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:25.714855 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-fa8c3f-78f95764bc-p8tx5" Oct 31 13:13:25.719687 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:25.717420 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-fa8c3f-6bc56f9c6c-r4ws4" event={"ID":"46d567ad-1dde-4d69-9ed1-8a2cab56e2db","Type":"ContainerStarted","Data":"69eddbaf46faf46106eeac3aa5e0c3850c8812f14a09cacdace290bfe2f8e9d6"} Oct 31 13:13:25.719687 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:25.717465 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-fa8c3f-6bc56f9c6c-r4ws4" event={"ID":"46d567ad-1dde-4d69-9ed1-8a2cab56e2db","Type":"ContainerStarted","Data":"4c424d822d3faab13c825a33a8d50d955e87963ff90022e686e86efe68f39730"} Oct 31 13:13:25.719687 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:25.717587 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-fa8c3f-6bc56f9c6c-r4ws4" Oct 31 13:13:25.724811 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:25.724783 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-29e3a0-59874b6cdf-7hvmk" Oct 31 13:13:25.733624 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:25.733577 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-fa8c3f-78f95764bc-p8tx5" podStartSLOduration=1.733560254 podStartE2EDuration="1.733560254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:25.732574735 +0000 UTC m=+2456.440161025" watchObservedRunningTime="2026-10-31 13:13:25.733560254 +0000 UTC m=+2456.441146552" Oct 31 13:13:25.965359 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:25.965281 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-29e3a0-b4ccb7d5f-2z6dt" Oct 31 13:13:26.097005 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:26.096957 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-fa8c3f-6bc56f9c6c-r4ws4" podStartSLOduration=2.096943071 podStartE2EDuration="2.096943071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:25.761719656 +0000 UTC m=+2456.469305950" watchObservedRunningTime="2026-10-31 13:13:26.096943071 +0000 UTC m=+2456.804529366" Oct 31 13:13:26.099523 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:26.099499 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-29e3a0-59874b6cdf-7hvmk"] Oct 31 13:13:26.296916 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:13:26.296888 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode18dd291_4694_41ad_922c_8852d7c12960.slice/crio-4f6263121667e2914c5ef6834ab71cbf5721f2ac4cc16320483637e7237f6e55 WatchSource:0}: Error finding container 4f6263121667e2914c5ef6834ab71cbf5721f2ac4cc16320483637e7237f6e55: Status 404 returned error can't find the container with id 4f6263121667e2914c5ef6834ab71cbf5721f2ac4cc16320483637e7237f6e55 Oct 31 13:13:26.297248 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:26.297219 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-29e3a0-b4ccb7d5f-2z6dt"] Oct 31 13:13:26.726869 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:26.726830 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-29e3a0-59874b6cdf-7hvmk" event={"ID":"5458e7f2-d785-4a4e-9b45-e5a9736b6758","Type":"ContainerStarted","Data":"1f8334b7769d22a882a0c98a3775133bd6b8ffbdc89ec526a5e6baaa59d93770"} Oct 31 13:13:26.727735 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:26.726894 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-29e3a0-59874b6cdf-7hvmk" event={"ID":"5458e7f2-d785-4a4e-9b45-e5a9736b6758","Type":"ContainerStarted","Data":"a53aaed21c1374659e5917b16f912b42c6b096f98032193194c7ac078dfb568d"} Oct 31 13:13:26.727735 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:26.726923 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-29e3a0-59874b6cdf-7hvmk" Oct 31 13:13:26.728490 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:26.728466 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-29e3a0-b4ccb7d5f-2z6dt" event={"ID":"e18dd291-4694-41ad-922c-8852d7c12960","Type":"ContainerStarted","Data":"01ddeac2d9492d695ddba4145d70f9361789604246fa9c985b9d40c4685ec1e9"} Oct 31 13:13:26.728601 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:26.728510 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-29e3a0-b4ccb7d5f-2z6dt" event={"ID":"e18dd291-4694-41ad-922c-8852d7c12960","Type":"ContainerStarted","Data":"4f6263121667e2914c5ef6834ab71cbf5721f2ac4cc16320483637e7237f6e55"} Oct 31 13:13:26.728773 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:26.728751 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-29e3a0-b4ccb7d5f-2z6dt" Oct 31 13:13:26.758941 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:26.758898 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-29e3a0-59874b6cdf-7hvmk" podStartSLOduration=1.758885499 podStartE2EDuration="1.758885499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:26.757038723 +0000 UTC m=+2457.464625044" watchObservedRunningTime="2026-10-31 13:13:26.758885499 +0000 UTC m=+2457.466471848" Oct 31 13:13:26.783352 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:26.783287 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-29e3a0-b4ccb7d5f-2z6dt" podStartSLOduration=1.783268837 podStartE2EDuration="1.783268837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:26.782854146 +0000 UTC m=+2457.490440443" watchObservedRunningTime="2026-10-31 13:13:26.783268837 +0000 UTC m=+2457.490855133" Oct 31 13:13:30.437189 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:30.437165 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5dc570-5cfd7479bc-csb46" Oct 31 13:13:36.506443 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:36.506412 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz"] Oct 31 13:13:36.506893 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:36.506658 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" podUID="dbec326f-9f8a-4e82-82de-21436c94736e" containerName="istio-proxy" containerID="cri-o://399af08f4b98592ca3b7f4841bba2e6ef9106a2961e29adbb01422c42339a459" gracePeriod=30 Oct 31 13:13:36.731709 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:36.731678 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-fa8c3f-6bc56f9c6c-r4ws4" Oct 31 13:13:36.731900 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:36.731759 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-fa8c3f-78f95764bc-p8tx5" Oct 31 13:13:37.739498 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:37.739466 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-29e3a0-b4ccb7d5f-2z6dt" Oct 31 13:13:37.740021 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:37.739537 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-29e3a0-59874b6cdf-7hvmk" Oct 31 13:13:41.800743 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:41.800720 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" Oct 31 13:13:41.820001 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:41.819928 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbec326f-9f8a-4e82-82de-21436c94736e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbec326f-9f8a-4e82-82de-21436c94736e-workload-certs\") pod \"dbec326f-9f8a-4e82-82de-21436c94736e\" (UID: \"dbec326f-9f8a-4e82-82de-21436c94736e\") " Oct 31 13:13:41.820001 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:41.819994 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbec326f-9f8a-4e82-82de-21436c94736e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbec326f-9f8a-4e82-82de-21436c94736e-istio-data\") pod \"dbec326f-9f8a-4e82-82de-21436c94736e\" (UID: \"dbec326f-9f8a-4e82-82de-21436c94736e\") " Oct 31 13:13:41.820179 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:41.820030 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbec326f-9f8a-4e82-82de-21436c94736e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbec326f-9f8a-4e82-82de-21436c94736e-istio-envoy\") pod \"dbec326f-9f8a-4e82-82de-21436c94736e\" (UID: \"dbec326f-9f8a-4e82-82de-21436c94736e\") " Oct 31 13:13:41.820179 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:41.820096 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dbec326f-9f8a-4e82-82de-21436c94736e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbec326f-9f8a-4e82-82de-21436c94736e-istio-podinfo\") pod \"dbec326f-9f8a-4e82-82de-21436c94736e\" (UID: \"dbec326f-9f8a-4e82-82de-21436c94736e\") " Oct 31 13:13:41.820179 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:41.820163 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbec326f-9f8a-4e82-82de-21436c94736e-kube-api-access-mqzxx\" (UniqueName: \"kubernetes.io/projected/dbec326f-9f8a-4e82-82de-21436c94736e-kube-api-access-mqzxx\") pod \"dbec326f-9f8a-4e82-82de-21436c94736e\" (UID: \"dbec326f-9f8a-4e82-82de-21436c94736e\") " Oct 31 13:13:41.820329 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:41.820202 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbec326f-9f8a-4e82-82de-21436c94736e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbec326f-9f8a-4e82-82de-21436c94736e-credential-socket\") pod \"dbec326f-9f8a-4e82-82de-21436c94736e\" (UID: \"dbec326f-9f8a-4e82-82de-21436c94736e\") " Oct 31 13:13:41.820329 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:41.820248 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbec326f-9f8a-4e82-82de-21436c94736e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbec326f-9f8a-4e82-82de-21436c94736e-workload-socket\") pod \"dbec326f-9f8a-4e82-82de-21436c94736e\" (UID: \"dbec326f-9f8a-4e82-82de-21436c94736e\") " Oct 31 13:13:41.820440 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:41.820350 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbec326f-9f8a-4e82-82de-21436c94736e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbec326f-9f8a-4e82-82de-21436c94736e-istiod-ca-cert\") pod \"dbec326f-9f8a-4e82-82de-21436c94736e\" (UID: \"dbec326f-9f8a-4e82-82de-21436c94736e\") " Oct 31 13:13:41.820498 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:41.820478 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbec326f-9f8a-4e82-82de-21436c94736e-istio-token\" (UniqueName: \"kubernetes.io/projected/dbec326f-9f8a-4e82-82de-21436c94736e-istio-token\") pod \"dbec326f-9f8a-4e82-82de-21436c94736e\" (UID: \"dbec326f-9f8a-4e82-82de-21436c94736e\") " Oct 31 13:13:41.820580 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:41.820243 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbec326f-9f8a-4e82-82de-21436c94736e-workload-certs" pod "dbec326f-9f8a-4e82-82de-21436c94736e" (UID: "dbec326f-9f8a-4e82-82de-21436c94736e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:41.820696 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:41.820580 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbec326f-9f8a-4e82-82de-21436c94736e-workload-socket" pod "dbec326f-9f8a-4e82-82de-21436c94736e" (UID: "dbec326f-9f8a-4e82-82de-21436c94736e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:41.820696 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:41.820256 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbec326f-9f8a-4e82-82de-21436c94736e-istio-data" pod "dbec326f-9f8a-4e82-82de-21436c94736e" (UID: "dbec326f-9f8a-4e82-82de-21436c94736e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:41.820696 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:41.820543 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbec326f-9f8a-4e82-82de-21436c94736e-credential-socket" pod "dbec326f-9f8a-4e82-82de-21436c94736e" (UID: "dbec326f-9f8a-4e82-82de-21436c94736e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:41.821012 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:41.820943 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbec326f-9f8a-4e82-82de-21436c94736e-istiod-ca-cert" pod "dbec326f-9f8a-4e82-82de-21436c94736e" (UID: "dbec326f-9f8a-4e82-82de-21436c94736e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:13:41.821327 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:41.821304 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbec326f-9f8a-4e82-82de-21436c94736e-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:41.821478 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:41.821335 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbec326f-9f8a-4e82-82de-21436c94736e-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:41.821478 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:41.821350 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbec326f-9f8a-4e82-82de-21436c94736e-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:41.821478 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:41.821364 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbec326f-9f8a-4e82-82de-21436c94736e-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:41.821478 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:41.821380 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbec326f-9f8a-4e82-82de-21436c94736e-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:41.822786 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:41.822761 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbec326f-9f8a-4e82-82de-21436c94736e-istio-envoy" pod "dbec326f-9f8a-4e82-82de-21436c94736e" (UID: "dbec326f-9f8a-4e82-82de-21436c94736e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:41.822879 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:41.822831 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbec326f-9f8a-4e82-82de-21436c94736e-kube-api-access-mqzxx" pod "dbec326f-9f8a-4e82-82de-21436c94736e" (UID: "dbec326f-9f8a-4e82-82de-21436c94736e"). InnerVolumeSpecName "kube-api-access-mqzxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:13:41.822921 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:41.822901 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbec326f-9f8a-4e82-82de-21436c94736e-istio-token" pod "dbec326f-9f8a-4e82-82de-21436c94736e" (UID: "dbec326f-9f8a-4e82-82de-21436c94736e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:13:41.822957 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:41.822943 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbec326f-9f8a-4e82-82de-21436c94736e-istio-podinfo" pod "dbec326f-9f8a-4e82-82de-21436c94736e" (UID: "dbec326f-9f8a-4e82-82de-21436c94736e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:13:41.873070 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:41.873032 5693 generic.go:356] "Generic (PLEG): container finished" podID="dbec326f-9f8a-4e82-82de-21436c94736e" containerID="399af08f4b98592ca3b7f4841bba2e6ef9106a2961e29adbb01422c42339a459" exitCode=0 Oct 31 13:13:41.873269 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:41.873111 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" Oct 31 13:13:41.873269 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:41.873128 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" event={"ID":"dbec326f-9f8a-4e82-82de-21436c94736e","Type":"ContainerDied","Data":"399af08f4b98592ca3b7f4841bba2e6ef9106a2961e29adbb01422c42339a459"} Oct 31 13:13:41.873269 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:41.873258 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz" event={"ID":"dbec326f-9f8a-4e82-82de-21436c94736e","Type":"ContainerDied","Data":"74d2fba0fb7a72eca4bba245164133319c997a3b596042eda99e2ec1c01ca7ca"} Oct 31 13:13:41.873403 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:41.873325 5693 scope.go:122] "RemoveContainer" containerID="399af08f4b98592ca3b7f4841bba2e6ef9106a2961e29adbb01422c42339a459" Oct 31 13:13:41.888512 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:41.888482 5693 scope.go:122] "RemoveContainer" containerID="399af08f4b98592ca3b7f4841bba2e6ef9106a2961e29adbb01422c42339a459" Oct 31 13:13:41.888891 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:13:41.888863 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"399af08f4b98592ca3b7f4841bba2e6ef9106a2961e29adbb01422c42339a459\": container with ID starting with 399af08f4b98592ca3b7f4841bba2e6ef9106a2961e29adbb01422c42339a459 not found: ID does not exist" containerID="399af08f4b98592ca3b7f4841bba2e6ef9106a2961e29adbb01422c42339a459" Oct 31 13:13:41.888978 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:41.888903 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"399af08f4b98592ca3b7f4841bba2e6ef9106a2961e29adbb01422c42339a459"} err="failed to get container status \"399af08f4b98592ca3b7f4841bba2e6ef9106a2961e29adbb01422c42339a459\": rpc error: code = NotFound desc = could not find container \"399af08f4b98592ca3b7f4841bba2e6ef9106a2961e29adbb01422c42339a459\": container with ID starting with 399af08f4b98592ca3b7f4841bba2e6ef9106a2961e29adbb01422c42339a459 not found: ID does not exist" Oct 31 13:13:41.906035 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:41.906004 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz"] Oct 31 13:13:41.912141 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:41.912110 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-h4kvz"] Oct 31 13:13:41.923256 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:41.923217 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbec326f-9f8a-4e82-82de-21436c94736e-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:41.923256 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:41.923255 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbec326f-9f8a-4e82-82de-21436c94736e-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:41.923426 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:41.923288 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqzxx\" (UniqueName: \"kubernetes.io/projected/dbec326f-9f8a-4e82-82de-21436c94736e-kube-api-access-mqzxx\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:41.923426 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:41.923298 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbec326f-9f8a-4e82-82de-21436c94736e-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:43.195990 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.195717 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg"] Oct 31 13:13:43.197367 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.197002 5693 cpu_manager.go:414] "containerMap: removing container" podUID="dbec326f-9f8a-4e82-82de-21436c94736e" containerName="istio-proxy" Oct 31 13:13:43.197367 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.197027 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbec326f-9f8a-4e82-82de-21436c94736e" containerName="istio-proxy" Oct 31 13:13:43.197367 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.197236 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbec326f-9f8a-4e82-82de-21436c94736e" containerName="istio-proxy" Oct 31 13:13:43.206137 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.206112 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg"] Oct 31 13:13:43.206978 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.206249 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg" Oct 31 13:13:43.211116 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.210233 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-15a6a2\"/\"openshift-service-ca.crt\"" Oct 31 13:13:43.212810 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.212439 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-15a6a2\"/\"kube-root-ca.crt\"" Oct 31 13:13:43.212810 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.212595 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-15a6a2\"/\"agc-cred-49713eea7cf3\"" Oct 31 13:13:43.212810 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.212710 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-15a6a2\"/\"routing-overlay\"" Oct 31 13:13:43.212810 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.212750 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-15a6a2\"/\"payload-processing-external-model-plugins-e2e-extmod-b-15a6a2\"" Oct 31 13:13:43.281081 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.281044 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80106c11-89ff-40fb-9222-d2e431273541-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg\" (UID: \"80106c11-89ff-40fb-9222-d2e431273541\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg" Oct 31 13:13:43.281281 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.281177 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/80106c11-89ff-40fb-9222-d2e431273541-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg\" (UID: \"80106c11-89ff-40fb-9222-d2e431273541\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg" Oct 31 13:13:43.281281 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.281244 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80106c11-89ff-40fb-9222-d2e431273541-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg\" (UID: \"80106c11-89ff-40fb-9222-d2e431273541\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg" Oct 31 13:13:43.281281 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.281268 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln8x8\" (UniqueName: \"kubernetes.io/projected/80106c11-89ff-40fb-9222-d2e431273541-kube-api-access-ln8x8\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg\" (UID: \"80106c11-89ff-40fb-9222-d2e431273541\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg" Oct 31 13:13:43.383824 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.383787 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80106c11-89ff-40fb-9222-d2e431273541-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg\" (UID: \"80106c11-89ff-40fb-9222-d2e431273541\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg" Oct 31 13:13:43.384165 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.384145 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/80106c11-89ff-40fb-9222-d2e431273541-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg\" (UID: \"80106c11-89ff-40fb-9222-d2e431273541\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg" Oct 31 13:13:43.402868 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.402829 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80106c11-89ff-40fb-9222-d2e431273541-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg\" (UID: \"80106c11-89ff-40fb-9222-d2e431273541\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg" Oct 31 13:13:43.403119 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.403103 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln8x8\" (UniqueName: \"kubernetes.io/projected/80106c11-89ff-40fb-9222-d2e431273541-kube-api-access-ln8x8\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg\" (UID: \"80106c11-89ff-40fb-9222-d2e431273541\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg" Oct 31 13:13:43.403789 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.385010 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80106c11-89ff-40fb-9222-d2e431273541-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg\" (UID: \"80106c11-89ff-40fb-9222-d2e431273541\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg" Oct 31 13:13:43.404791 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.404767 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80106c11-89ff-40fb-9222-d2e431273541-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg\" (UID: \"80106c11-89ff-40fb-9222-d2e431273541\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg" Oct 31 13:13:43.424233 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.423212 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/80106c11-89ff-40fb-9222-d2e431273541-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg\" (UID: \"80106c11-89ff-40fb-9222-d2e431273541\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg" Oct 31 13:13:43.464547 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.464468 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln8x8\" (UniqueName: \"kubernetes.io/projected/80106c11-89ff-40fb-9222-d2e431273541-kube-api-access-ln8x8\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg\" (UID: \"80106c11-89ff-40fb-9222-d2e431273541\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg" Oct 31 13:13:43.471448 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.471409 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbec326f-9f8a-4e82-82de-21436c94736e" path="/var/lib/kubelet/pods/dbec326f-9f8a-4e82-82de-21436c94736e/volumes" Oct 31 13:13:43.533123 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.533082 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg" Oct 31 13:13:43.684885 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.684853 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b"] Oct 31 13:13:43.691852 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.691826 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" Oct 31 13:13:43.702434 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.701508 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-45bcad-openshift-default-dockercfg-gfb47\"" Oct 31 13:13:43.702434 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.702560 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b"] Oct 31 13:13:43.744983 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.742729 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffe4ec4f-e426-4764-9347-34369d72854e-istio-envoy\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b\" (UID: \"ffe4ec4f-e426-4764-9347-34369d72854e\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" Oct 31 13:13:43.744983 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.742820 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffe4ec4f-e426-4764-9347-34369d72854e-workload-certs\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b\" (UID: \"ffe4ec4f-e426-4764-9347-34369d72854e\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" Oct 31 13:13:43.744983 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.742893 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffe4ec4f-e426-4764-9347-34369d72854e-istio-token\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b\" (UID: \"ffe4ec4f-e426-4764-9347-34369d72854e\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" Oct 31 13:13:43.744983 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.742953 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe4ec4f-e426-4764-9347-34369d72854e-credential-socket\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b\" (UID: \"ffe4ec4f-e426-4764-9347-34369d72854e\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" Oct 31 13:13:43.744983 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.742997 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe4ec4f-e426-4764-9347-34369d72854e-workload-socket\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b\" (UID: \"ffe4ec4f-e426-4764-9347-34369d72854e\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" Oct 31 13:13:43.744983 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.743148 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfm9f\" (UniqueName: \"kubernetes.io/projected/ffe4ec4f-e426-4764-9347-34369d72854e-kube-api-access-bfm9f\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b\" (UID: \"ffe4ec4f-e426-4764-9347-34369d72854e\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" Oct 31 13:13:43.744983 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.743196 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffe4ec4f-e426-4764-9347-34369d72854e-istio-data\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b\" (UID: \"ffe4ec4f-e426-4764-9347-34369d72854e\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" Oct 31 13:13:43.744983 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.743282 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffe4ec4f-e426-4764-9347-34369d72854e-istiod-ca-cert\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b\" (UID: \"ffe4ec4f-e426-4764-9347-34369d72854e\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" Oct 31 13:13:43.744983 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.743336 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffe4ec4f-e426-4764-9347-34369d72854e-istio-podinfo\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b\" (UID: \"ffe4ec4f-e426-4764-9347-34369d72854e\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" Oct 31 13:13:43.834252 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.834033 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg"] Oct 31 13:13:43.844668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.844622 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffe4ec4f-e426-4764-9347-34369d72854e-istio-token\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b\" (UID: \"ffe4ec4f-e426-4764-9347-34369d72854e\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" Oct 31 13:13:43.844892 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.844873 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe4ec4f-e426-4764-9347-34369d72854e-credential-socket\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b\" (UID: \"ffe4ec4f-e426-4764-9347-34369d72854e\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" Oct 31 13:13:43.845039 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.845014 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe4ec4f-e426-4764-9347-34369d72854e-workload-socket\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b\" (UID: \"ffe4ec4f-e426-4764-9347-34369d72854e\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" Oct 31 13:13:43.845175 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.845152 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfm9f\" (UniqueName: \"kubernetes.io/projected/ffe4ec4f-e426-4764-9347-34369d72854e-kube-api-access-bfm9f\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b\" (UID: \"ffe4ec4f-e426-4764-9347-34369d72854e\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" Oct 31 13:13:43.845240 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.845191 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffe4ec4f-e426-4764-9347-34369d72854e-istio-data\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b\" (UID: \"ffe4ec4f-e426-4764-9347-34369d72854e\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" Oct 31 13:13:43.845300 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.845264 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffe4ec4f-e426-4764-9347-34369d72854e-istiod-ca-cert\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b\" (UID: \"ffe4ec4f-e426-4764-9347-34369d72854e\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" Oct 31 13:13:43.845300 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.845272 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe4ec4f-e426-4764-9347-34369d72854e-credential-socket\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b\" (UID: \"ffe4ec4f-e426-4764-9347-34369d72854e\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" Oct 31 13:13:43.845417 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.845332 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffe4ec4f-e426-4764-9347-34369d72854e-istio-podinfo\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b\" (UID: \"ffe4ec4f-e426-4764-9347-34369d72854e\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" Oct 31 13:13:43.845417 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.845366 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe4ec4f-e426-4764-9347-34369d72854e-workload-socket\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b\" (UID: \"ffe4ec4f-e426-4764-9347-34369d72854e\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" Oct 31 13:13:43.845417 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.845384 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffe4ec4f-e426-4764-9347-34369d72854e-istio-envoy\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b\" (UID: \"ffe4ec4f-e426-4764-9347-34369d72854e\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" Oct 31 13:13:43.845593 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.845466 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffe4ec4f-e426-4764-9347-34369d72854e-workload-certs\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b\" (UID: \"ffe4ec4f-e426-4764-9347-34369d72854e\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" Oct 31 13:13:43.845907 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.845878 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffe4ec4f-e426-4764-9347-34369d72854e-workload-certs\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b\" (UID: \"ffe4ec4f-e426-4764-9347-34369d72854e\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" Oct 31 13:13:43.846012 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.845942 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffe4ec4f-e426-4764-9347-34369d72854e-istio-data\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b\" (UID: \"ffe4ec4f-e426-4764-9347-34369d72854e\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" Oct 31 13:13:43.846012 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.845948 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffe4ec4f-e426-4764-9347-34369d72854e-istiod-ca-cert\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b\" (UID: \"ffe4ec4f-e426-4764-9347-34369d72854e\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" Oct 31 13:13:43.848527 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.848449 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffe4ec4f-e426-4764-9347-34369d72854e-istio-envoy\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b\" (UID: \"ffe4ec4f-e426-4764-9347-34369d72854e\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" Oct 31 13:13:43.848747 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.848729 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffe4ec4f-e426-4764-9347-34369d72854e-istio-podinfo\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b\" (UID: \"ffe4ec4f-e426-4764-9347-34369d72854e\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" Oct 31 13:13:43.859145 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.859124 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffe4ec4f-e426-4764-9347-34369d72854e-istio-token\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b\" (UID: \"ffe4ec4f-e426-4764-9347-34369d72854e\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" Oct 31 13:13:43.859915 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.859894 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfm9f\" (UniqueName: \"kubernetes.io/projected/ffe4ec4f-e426-4764-9347-34369d72854e-kube-api-access-bfm9f\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b\" (UID: \"ffe4ec4f-e426-4764-9347-34369d72854e\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" Oct 31 13:13:43.935985 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.935937 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg" event={"ID":"80106c11-89ff-40fb-9222-d2e431273541","Type":"ContainerStarted","Data":"c777877c9c91fa6bf12e9f54cd8eb2bf958136aec81b0a3023bf9af0d2afb727"} Oct 31 13:13:43.936144 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.936006 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg" event={"ID":"80106c11-89ff-40fb-9222-d2e431273541","Type":"ContainerStarted","Data":"aee01a9143d3237c8bf8940c8afe478574419901c95aa9e2c939c81d8b52eb26"} Oct 31 13:13:43.936199 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.936156 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg" Oct 31 13:13:43.957838 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:43.957787 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg" podStartSLOduration=0.957772838 podStartE2EDuration="957.772838ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:43.95440726 +0000 UTC m=+2474.661993555" watchObservedRunningTime="2026-10-31 13:13:43.957772838 +0000 UTC m=+2474.665359133" Oct 31 13:13:44.082818 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:44.081281 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" Oct 31 13:13:44.383817 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:44.383773 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b"] Oct 31 13:13:44.384953 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:13:44.384919 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffe4ec4f_e426_4764_9347_34369d72854e.slice/crio-c8529e835b4f79d828c31bc297631c23564e76668bca5830dd3e9c575f91cdce WatchSource:0}: Error finding container c8529e835b4f79d828c31bc297631c23564e76668bca5830dd3e9c575f91cdce: Status 404 returned error can't find the container with id c8529e835b4f79d828c31bc297631c23564e76668bca5830dd3e9c575f91cdce Oct 31 13:13:44.388075 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:44.388041 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:13:44.388160 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:44.388125 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:13:44.388160 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:44.388157 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:13:44.947313 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:44.947276 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" event={"ID":"ffe4ec4f-e426-4764-9347-34369d72854e","Type":"ContainerStarted","Data":"68af4afba643056662ab77775a87487c3890f6af466fcda8bd85e494c856e714"} Oct 31 13:13:44.947490 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:44.947357 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" event={"ID":"ffe4ec4f-e426-4764-9347-34369d72854e","Type":"ContainerStarted","Data":"c8529e835b4f79d828c31bc297631c23564e76668bca5830dd3e9c575f91cdce"} Oct 31 13:13:44.975274 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:44.975228 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" podStartSLOduration=1.975213881 podStartE2EDuration="1.975213881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:44.971409736 +0000 UTC m=+2475.678996043" watchObservedRunningTime="2026-10-31 13:13:44.975213881 +0000 UTC m=+2475.682800175" Oct 31 13:13:45.081460 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:45.081428 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" Oct 31 13:13:45.083022 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:45.082993 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 31 13:13:45.083179 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:45.083040 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" podUID="ffe4ec4f-e426-4764-9347-34369d72854e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 31 13:13:46.097690 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:46.094979 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 31 13:13:46.097690 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:46.095047 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" podUID="ffe4ec4f-e426-4764-9347-34369d72854e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 31 13:13:47.098668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:47.092007 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 31 13:13:47.098668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:47.092071 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" podUID="ffe4ec4f-e426-4764-9347-34369d72854e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 31 13:13:48.085081 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:48.085041 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 31 13:13:48.085386 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:48.085356 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" podUID="ffe4ec4f-e426-4764-9347-34369d72854e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 31 13:13:48.707803 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:13:48.707758 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:13:49.091796 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:49.085626 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 31 13:13:49.091796 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:49.085706 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" podUID="ffe4ec4f-e426-4764-9347-34369d72854e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 31 13:13:50.090667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:50.089067 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 31 13:13:50.090667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:50.089139 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" podUID="ffe4ec4f-e426-4764-9347-34369d72854e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 31 13:13:51.097690 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:51.097027 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 31 13:13:51.097690 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:51.097091 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" podUID="ffe4ec4f-e426-4764-9347-34369d72854e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 31 13:13:52.087437 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:52.087407 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" Oct 31 13:13:52.087919 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:52.087899 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" Oct 31 13:13:52.088541 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:52.088521 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" Oct 31 13:13:54.958910 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:13:54.958873 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg" Oct 31 13:14:02.535061 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:02.535028 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0c1fbb-9c8cfbcf-rthxp"] Oct 31 13:14:02.541187 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:02.541166 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0c1fbb-9c8cfbcf-rthxp" Oct 31 13:14:02.547614 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:02.547589 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-0c1fbb\"" Oct 31 13:14:02.552093 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:02.552069 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0c1fbb-9c8cfbcf-rthxp"] Oct 31 13:14:02.573208 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:02.573180 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0c1fbb-5dcd8b5b65-qlkdw"] Oct 31 13:14:02.574748 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:02.574724 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-0c1fbb-dockercfg-mgdzc\"" Oct 31 13:14:02.577956 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:02.577937 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0c1fbb-5dcd8b5b65-qlkdw" Oct 31 13:14:02.585212 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:02.585172 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0c1fbb-5dcd8b5b65-qlkdw"] Oct 31 13:14:02.643938 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:02.643855 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v6zn\" (UniqueName: \"kubernetes.io/projected/e87aa817-1a1d-4267-bd3f-0d74c9a229fd-kube-api-access-4v6zn\") pod \"payload-pre-processing-e2e-extmod-b-0c1fbb-9c8cfbcf-rthxp\" (UID: \"e87aa817-1a1d-4267-bd3f-0d74c9a229fd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0c1fbb-9c8cfbcf-rthxp" Oct 31 13:14:02.648882 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:02.648823 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e87aa817-1a1d-4267-bd3f-0d74c9a229fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0c1fbb-9c8cfbcf-rthxp\" (UID: \"e87aa817-1a1d-4267-bd3f-0d74c9a229fd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0c1fbb-9c8cfbcf-rthxp" Oct 31 13:14:02.801403 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:02.783359 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d850f787-64ff-4a6b-8534-a93329e45929-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0c1fbb-5dcd8b5b65-qlkdw\" (UID: \"d850f787-64ff-4a6b-8534-a93329e45929\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0c1fbb-5dcd8b5b65-qlkdw" Oct 31 13:14:02.816672 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:02.811346 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e87aa817-1a1d-4267-bd3f-0d74c9a229fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0c1fbb-9c8cfbcf-rthxp\" (UID: \"e87aa817-1a1d-4267-bd3f-0d74c9a229fd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0c1fbb-9c8cfbcf-rthxp" Oct 31 13:14:02.816672 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:02.811562 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6kgz\" (UniqueName: \"kubernetes.io/projected/d850f787-64ff-4a6b-8534-a93329e45929-kube-api-access-k6kgz\") pod \"payload-processing-e2e-extmod-b-0c1fbb-5dcd8b5b65-qlkdw\" (UID: \"d850f787-64ff-4a6b-8534-a93329e45929\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0c1fbb-5dcd8b5b65-qlkdw" Oct 31 13:14:02.816672 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:02.811609 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v6zn\" (UniqueName: \"kubernetes.io/projected/e87aa817-1a1d-4267-bd3f-0d74c9a229fd-kube-api-access-4v6zn\") pod \"payload-pre-processing-e2e-extmod-b-0c1fbb-9c8cfbcf-rthxp\" (UID: \"e87aa817-1a1d-4267-bd3f-0d74c9a229fd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0c1fbb-9c8cfbcf-rthxp" Oct 31 13:14:02.816672 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:02.813390 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e87aa817-1a1d-4267-bd3f-0d74c9a229fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0c1fbb-9c8cfbcf-rthxp\" (UID: \"e87aa817-1a1d-4267-bd3f-0d74c9a229fd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0c1fbb-9c8cfbcf-rthxp" Oct 31 13:14:02.853399 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:02.853312 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v6zn\" (UniqueName: \"kubernetes.io/projected/e87aa817-1a1d-4267-bd3f-0d74c9a229fd-kube-api-access-4v6zn\") pod \"payload-pre-processing-e2e-extmod-b-0c1fbb-9c8cfbcf-rthxp\" (UID: \"e87aa817-1a1d-4267-bd3f-0d74c9a229fd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0c1fbb-9c8cfbcf-rthxp" Oct 31 13:14:02.860783 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:02.860737 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0c1fbb-9c8cfbcf-rthxp" Oct 31 13:14:02.913321 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:02.913271 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6kgz\" (UniqueName: \"kubernetes.io/projected/d850f787-64ff-4a6b-8534-a93329e45929-kube-api-access-k6kgz\") pod \"payload-processing-e2e-extmod-b-0c1fbb-5dcd8b5b65-qlkdw\" (UID: \"d850f787-64ff-4a6b-8534-a93329e45929\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0c1fbb-5dcd8b5b65-qlkdw" Oct 31 13:14:02.913514 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:02.913442 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d850f787-64ff-4a6b-8534-a93329e45929-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0c1fbb-5dcd8b5b65-qlkdw\" (UID: \"d850f787-64ff-4a6b-8534-a93329e45929\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0c1fbb-5dcd8b5b65-qlkdw" Oct 31 13:14:02.914199 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:02.914169 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d850f787-64ff-4a6b-8534-a93329e45929-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0c1fbb-5dcd8b5b65-qlkdw\" (UID: \"d850f787-64ff-4a6b-8534-a93329e45929\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0c1fbb-5dcd8b5b65-qlkdw" Oct 31 13:14:02.928687 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:02.928620 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6kgz\" (UniqueName: \"kubernetes.io/projected/d850f787-64ff-4a6b-8534-a93329e45929-kube-api-access-k6kgz\") pod \"payload-processing-e2e-extmod-b-0c1fbb-5dcd8b5b65-qlkdw\" (UID: \"d850f787-64ff-4a6b-8534-a93329e45929\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0c1fbb-5dcd8b5b65-qlkdw" Oct 31 13:14:03.024711 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:03.024673 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0c1fbb-5dcd8b5b65-qlkdw" Oct 31 13:14:03.173870 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:03.173836 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0c1fbb-9c8cfbcf-rthxp"] Oct 31 13:14:03.179508 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:14:03.179465 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode87aa817_1a1d_4267_bd3f_0d74c9a229fd.slice/crio-0f6430ced51562c3e48d24ab65006fadf944bf14c23fe6195acca0ccbb89cf1d WatchSource:0}: Error finding container 0f6430ced51562c3e48d24ab65006fadf944bf14c23fe6195acca0ccbb89cf1d: Status 404 returned error can't find the container with id 0f6430ced51562c3e48d24ab65006fadf944bf14c23fe6195acca0ccbb89cf1d Oct 31 13:14:03.199595 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:03.199565 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0c1fbb-9c8cfbcf-rthxp" event={"ID":"e87aa817-1a1d-4267-bd3f-0d74c9a229fd","Type":"ContainerStarted","Data":"0f6430ced51562c3e48d24ab65006fadf944bf14c23fe6195acca0ccbb89cf1d"} Oct 31 13:14:03.355901 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:03.355865 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0c1fbb-5dcd8b5b65-qlkdw"] Oct 31 13:14:03.356174 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:14:03.356147 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd850f787_64ff_4a6b_8534_a93329e45929.slice/crio-362f4e46983cb41893a13ece9f090c2c8829f93096574d9bdc8b66671ec65408 WatchSource:0}: Error finding container 362f4e46983cb41893a13ece9f090c2c8829f93096574d9bdc8b66671ec65408: Status 404 returned error can't find the container with id 362f4e46983cb41893a13ece9f090c2c8829f93096574d9bdc8b66671ec65408 Oct 31 13:14:04.213939 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:04.213903 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0c1fbb-9c8cfbcf-rthxp" event={"ID":"e87aa817-1a1d-4267-bd3f-0d74c9a229fd","Type":"ContainerStarted","Data":"c2be1995485ae123473c10352ae83e6fb2869e33672901d14f29b19ed462f727"} Oct 31 13:14:04.214606 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:04.213976 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0c1fbb-9c8cfbcf-rthxp" Oct 31 13:14:04.216006 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:04.215976 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0c1fbb-5dcd8b5b65-qlkdw" event={"ID":"d850f787-64ff-4a6b-8534-a93329e45929","Type":"ContainerStarted","Data":"01a2320f95ffe6947a8e5d1cd2f648c812669cb213ce097c006d8ec15ca75b75"} Oct 31 13:14:04.216140 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:04.216030 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-0c1fbb-5dcd8b5b65-qlkdw" Oct 31 13:14:04.216140 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:04.216052 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0c1fbb-5dcd8b5b65-qlkdw" event={"ID":"d850f787-64ff-4a6b-8534-a93329e45929","Type":"ContainerStarted","Data":"362f4e46983cb41893a13ece9f090c2c8829f93096574d9bdc8b66671ec65408"} Oct 31 13:14:04.246501 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:04.246455 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0c1fbb-9c8cfbcf-rthxp" podStartSLOduration=2.246441856 podStartE2EDuration="2.246441856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:14:04.243770599 +0000 UTC m=+2494.951356894" watchObservedRunningTime="2026-10-31 13:14:04.246441856 +0000 UTC m=+2494.954028195" Oct 31 13:14:04.283302 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:04.283204 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-0c1fbb-5dcd8b5b65-qlkdw" podStartSLOduration=2.28318345 podStartE2EDuration="2.28318345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:14:04.280131063 +0000 UTC m=+2494.987717359" watchObservedRunningTime="2026-10-31 13:14:04.28318345 +0000 UTC m=+2494.990769747" Oct 31 13:14:10.049114 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:10.049080 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c2c98b-755946465f-m2g26"] Oct 31 13:14:10.054457 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:10.054433 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c2c98b-755946465f-m2g26" Oct 31 13:14:10.062572 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:10.062548 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-c2c98b-dockercfg-pdrmw\"" Oct 31 13:14:10.064989 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:10.064883 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-c2c98b\"" Oct 31 13:14:10.071038 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:10.071004 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c2c98b-755946465f-m2g26"] Oct 31 13:14:10.159366 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:10.159330 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08846306-817b-4ccf-8ec3-470e1bba6257-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-c2c98b-755946465f-m2g26\" (UID: \"08846306-817b-4ccf-8ec3-470e1bba6257\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c2c98b-755946465f-m2g26" Oct 31 13:14:10.159560 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:10.159386 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rcjt\" (UniqueName: \"kubernetes.io/projected/08846306-817b-4ccf-8ec3-470e1bba6257-kube-api-access-6rcjt\") pod \"payload-processing-e2e-extmod-b-c2c98b-755946465f-m2g26\" (UID: \"08846306-817b-4ccf-8ec3-470e1bba6257\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c2c98b-755946465f-m2g26" Oct 31 13:14:10.270691 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:10.267338 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08846306-817b-4ccf-8ec3-470e1bba6257-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-c2c98b-755946465f-m2g26\" (UID: \"08846306-817b-4ccf-8ec3-470e1bba6257\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c2c98b-755946465f-m2g26" Oct 31 13:14:10.270691 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:10.267426 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rcjt\" (UniqueName: \"kubernetes.io/projected/08846306-817b-4ccf-8ec3-470e1bba6257-kube-api-access-6rcjt\") pod \"payload-processing-e2e-extmod-b-c2c98b-755946465f-m2g26\" (UID: \"08846306-817b-4ccf-8ec3-470e1bba6257\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c2c98b-755946465f-m2g26" Oct 31 13:14:10.290675 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:10.276403 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08846306-817b-4ccf-8ec3-470e1bba6257-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-c2c98b-755946465f-m2g26\" (UID: \"08846306-817b-4ccf-8ec3-470e1bba6257\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c2c98b-755946465f-m2g26" Oct 31 13:14:10.290675 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:10.286759 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rcjt\" (UniqueName: \"kubernetes.io/projected/08846306-817b-4ccf-8ec3-470e1bba6257-kube-api-access-6rcjt\") pod \"payload-processing-e2e-extmod-b-c2c98b-755946465f-m2g26\" (UID: \"08846306-817b-4ccf-8ec3-470e1bba6257\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c2c98b-755946465f-m2g26" Oct 31 13:14:10.378024 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:10.377919 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c2c98b-755946465f-m2g26" Oct 31 13:14:10.712231 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:10.712201 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c2c98b-755946465f-m2g26"] Oct 31 13:14:10.712790 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:14:10.712761 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08846306_817b_4ccf_8ec3_470e1bba6257.slice/crio-1701db71ff9c1eb9384c11e47406f0846ca5886ab2b84af9131456f831f2b2e1 WatchSource:0}: Error finding container 1701db71ff9c1eb9384c11e47406f0846ca5886ab2b84af9131456f831f2b2e1: Status 404 returned error can't find the container with id 1701db71ff9c1eb9384c11e47406f0846ca5886ab2b84af9131456f831f2b2e1 Oct 31 13:14:11.354274 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:11.354235 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c2c98b-755946465f-m2g26" event={"ID":"08846306-817b-4ccf-8ec3-470e1bba6257","Type":"ContainerStarted","Data":"f020ac917a90d181f8441089aa3cad7751032e9cb1a8d171d22dec9b16de1536"} Oct 31 13:14:11.354819 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:11.354302 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c2c98b-755946465f-m2g26" event={"ID":"08846306-817b-4ccf-8ec3-470e1bba6257","Type":"ContainerStarted","Data":"1701db71ff9c1eb9384c11e47406f0846ca5886ab2b84af9131456f831f2b2e1"} Oct 31 13:14:11.354819 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:11.354435 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-c2c98b-755946465f-m2g26" Oct 31 13:14:11.406560 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:11.406505 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-c2c98b-755946465f-m2g26" podStartSLOduration=1.40648801 podStartE2EDuration="1.40648801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:14:11.403803524 +0000 UTC m=+2502.111389801" watchObservedRunningTime="2026-10-31 13:14:11.40648801 +0000 UTC m=+2502.114074306" Oct 31 13:14:11.792468 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:11.792435 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb"] Oct 31 13:14:11.798465 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:11.798440 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" Oct 31 13:14:11.817304 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:11.817280 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-5dc570-openshift-default-dockercfg-dvsb2\"" Oct 31 13:14:11.827510 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:11.827478 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb"] Oct 31 13:14:11.900745 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:11.900703 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0109302a-8951-450d-a280-98f4ab4fd093-workload-certs\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb\" (UID: \"0109302a-8951-450d-a280-98f4ab4fd093\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" Oct 31 13:14:11.900914 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:11.900777 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0109302a-8951-450d-a280-98f4ab4fd093-istio-podinfo\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb\" (UID: \"0109302a-8951-450d-a280-98f4ab4fd093\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" Oct 31 13:14:11.900914 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:11.900870 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0109302a-8951-450d-a280-98f4ab4fd093-istio-token\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb\" (UID: \"0109302a-8951-450d-a280-98f4ab4fd093\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" Oct 31 13:14:11.900914 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:11.900892 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0109302a-8951-450d-a280-98f4ab4fd093-istiod-ca-cert\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb\" (UID: \"0109302a-8951-450d-a280-98f4ab4fd093\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" Oct 31 13:14:11.901086 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:11.901014 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0109302a-8951-450d-a280-98f4ab4fd093-istio-data\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb\" (UID: \"0109302a-8951-450d-a280-98f4ab4fd093\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" Oct 31 13:14:11.901178 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:11.901157 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0109302a-8951-450d-a280-98f4ab4fd093-credential-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb\" (UID: \"0109302a-8951-450d-a280-98f4ab4fd093\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" Oct 31 13:14:11.901273 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:11.901258 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pk9j\" (UniqueName: \"kubernetes.io/projected/0109302a-8951-450d-a280-98f4ab4fd093-kube-api-access-4pk9j\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb\" (UID: \"0109302a-8951-450d-a280-98f4ab4fd093\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" Oct 31 13:14:11.901323 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:11.901310 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0109302a-8951-450d-a280-98f4ab4fd093-workload-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb\" (UID: \"0109302a-8951-450d-a280-98f4ab4fd093\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" Oct 31 13:14:11.901375 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:11.901329 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0109302a-8951-450d-a280-98f4ab4fd093-istio-envoy\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb\" (UID: \"0109302a-8951-450d-a280-98f4ab4fd093\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" Oct 31 13:14:12.002870 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.002832 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0109302a-8951-450d-a280-98f4ab4fd093-credential-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb\" (UID: \"0109302a-8951-450d-a280-98f4ab4fd093\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" Oct 31 13:14:12.003042 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.002910 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pk9j\" (UniqueName: \"kubernetes.io/projected/0109302a-8951-450d-a280-98f4ab4fd093-kube-api-access-4pk9j\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb\" (UID: \"0109302a-8951-450d-a280-98f4ab4fd093\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" Oct 31 13:14:12.003042 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.002968 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0109302a-8951-450d-a280-98f4ab4fd093-workload-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb\" (UID: \"0109302a-8951-450d-a280-98f4ab4fd093\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" Oct 31 13:14:12.003042 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.002996 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0109302a-8951-450d-a280-98f4ab4fd093-istio-envoy\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb\" (UID: \"0109302a-8951-450d-a280-98f4ab4fd093\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" Oct 31 13:14:12.003202 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.003083 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0109302a-8951-450d-a280-98f4ab4fd093-workload-certs\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb\" (UID: \"0109302a-8951-450d-a280-98f4ab4fd093\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" Oct 31 13:14:12.003202 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.003142 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0109302a-8951-450d-a280-98f4ab4fd093-istio-podinfo\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb\" (UID: \"0109302a-8951-450d-a280-98f4ab4fd093\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" Oct 31 13:14:12.003631 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.003272 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0109302a-8951-450d-a280-98f4ab4fd093-istio-token\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb\" (UID: \"0109302a-8951-450d-a280-98f4ab4fd093\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" Oct 31 13:14:12.003631 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.003312 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0109302a-8951-450d-a280-98f4ab4fd093-istiod-ca-cert\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb\" (UID: \"0109302a-8951-450d-a280-98f4ab4fd093\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" Oct 31 13:14:12.003631 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.003316 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0109302a-8951-450d-a280-98f4ab4fd093-credential-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb\" (UID: \"0109302a-8951-450d-a280-98f4ab4fd093\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" Oct 31 13:14:12.003631 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.003372 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0109302a-8951-450d-a280-98f4ab4fd093-workload-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb\" (UID: \"0109302a-8951-450d-a280-98f4ab4fd093\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" Oct 31 13:14:12.003631 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.003446 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0109302a-8951-450d-a280-98f4ab4fd093-istio-data\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb\" (UID: \"0109302a-8951-450d-a280-98f4ab4fd093\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" Oct 31 13:14:12.003985 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.003907 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0109302a-8951-450d-a280-98f4ab4fd093-istio-data\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb\" (UID: \"0109302a-8951-450d-a280-98f4ab4fd093\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" Oct 31 13:14:12.004049 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.003994 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0109302a-8951-450d-a280-98f4ab4fd093-workload-certs\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb\" (UID: \"0109302a-8951-450d-a280-98f4ab4fd093\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" Oct 31 13:14:12.004372 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.004352 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0109302a-8951-450d-a280-98f4ab4fd093-istiod-ca-cert\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb\" (UID: \"0109302a-8951-450d-a280-98f4ab4fd093\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" Oct 31 13:14:12.006429 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.006404 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0109302a-8951-450d-a280-98f4ab4fd093-istio-podinfo\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb\" (UID: \"0109302a-8951-450d-a280-98f4ab4fd093\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" Oct 31 13:14:12.006573 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.006556 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0109302a-8951-450d-a280-98f4ab4fd093-istio-envoy\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb\" (UID: \"0109302a-8951-450d-a280-98f4ab4fd093\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" Oct 31 13:14:12.016873 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.016847 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pk9j\" (UniqueName: \"kubernetes.io/projected/0109302a-8951-450d-a280-98f4ab4fd093-kube-api-access-4pk9j\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb\" (UID: \"0109302a-8951-450d-a280-98f4ab4fd093\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" Oct 31 13:14:12.016965 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.016903 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0109302a-8951-450d-a280-98f4ab4fd093-istio-token\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb\" (UID: \"0109302a-8951-450d-a280-98f4ab4fd093\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" Oct 31 13:14:12.122777 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.121762 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" Oct 31 13:14:12.575496 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:14:12.575461 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0109302a_8951_450d_a280_98f4ab4fd093.slice/crio-e3eeff3bd202759aa63e639fccc3f979423d5d7c031b22ad8d80f9ebe0db33ad WatchSource:0}: Error finding container e3eeff3bd202759aa63e639fccc3f979423d5d7c031b22ad8d80f9ebe0db33ad: Status 404 returned error can't find the container with id e3eeff3bd202759aa63e639fccc3f979423d5d7c031b22ad8d80f9ebe0db33ad Oct 31 13:14:12.576978 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.576950 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb"] Oct 31 13:14:12.579001 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.578968 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:14:12.579083 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.579043 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:14:12.579129 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.579089 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:14:12.704837 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.704805 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx"] Oct 31 13:14:12.710145 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.710120 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx" Oct 31 13:14:12.725886 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.725808 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-29e3a0\"/\"agc-cred-c4174febc8d5\"" Oct 31 13:14:12.730855 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.730792 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-29e3a0\"/\"payload-processing-external-model-plugins-e2e-extmod-b-29e3a0\"" Oct 31 13:14:12.731012 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.730963 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-29e3a0\"/\"kube-root-ca.crt\"" Oct 31 13:14:12.731082 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.731072 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-29e3a0\"/\"routing-overlay\"" Oct 31 13:14:12.742914 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.742883 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-29e3a0\"/\"openshift-service-ca.crt\"" Oct 31 13:14:12.744152 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.744124 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx"] Oct 31 13:14:12.846439 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.846360 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d77d1186-90c6-426b-aa02-758d811fb743-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx\" (UID: \"d77d1186-90c6-426b-aa02-758d811fb743\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx" Oct 31 13:14:12.846718 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.846691 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sttbb\" (UniqueName: \"kubernetes.io/projected/d77d1186-90c6-426b-aa02-758d811fb743-kube-api-access-sttbb\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx\" (UID: \"d77d1186-90c6-426b-aa02-758d811fb743\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx" Oct 31 13:14:12.846824 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.846804 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d77d1186-90c6-426b-aa02-758d811fb743-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx\" (UID: \"d77d1186-90c6-426b-aa02-758d811fb743\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx" Oct 31 13:14:12.846901 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.846889 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d77d1186-90c6-426b-aa02-758d811fb743-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx\" (UID: \"d77d1186-90c6-426b-aa02-758d811fb743\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx" Oct 31 13:14:12.954663 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.951732 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sttbb\" (UniqueName: \"kubernetes.io/projected/d77d1186-90c6-426b-aa02-758d811fb743-kube-api-access-sttbb\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx\" (UID: \"d77d1186-90c6-426b-aa02-758d811fb743\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx" Oct 31 13:14:12.954663 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.951835 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d77d1186-90c6-426b-aa02-758d811fb743-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx\" (UID: \"d77d1186-90c6-426b-aa02-758d811fb743\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx" Oct 31 13:14:12.954663 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.951943 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d77d1186-90c6-426b-aa02-758d811fb743-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx\" (UID: \"d77d1186-90c6-426b-aa02-758d811fb743\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx" Oct 31 13:14:12.954663 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.952000 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d77d1186-90c6-426b-aa02-758d811fb743-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx\" (UID: \"d77d1186-90c6-426b-aa02-758d811fb743\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx" Oct 31 13:14:12.954663 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.953090 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d77d1186-90c6-426b-aa02-758d811fb743-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx\" (UID: \"d77d1186-90c6-426b-aa02-758d811fb743\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx" Oct 31 13:14:12.963661 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.961501 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d77d1186-90c6-426b-aa02-758d811fb743-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx\" (UID: \"d77d1186-90c6-426b-aa02-758d811fb743\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx" Oct 31 13:14:12.989665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.985315 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d77d1186-90c6-426b-aa02-758d811fb743-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx\" (UID: \"d77d1186-90c6-426b-aa02-758d811fb743\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx" Oct 31 13:14:12.990910 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:12.990874 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sttbb\" (UniqueName: \"kubernetes.io/projected/d77d1186-90c6-426b-aa02-758d811fb743-kube-api-access-sttbb\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx\" (UID: \"d77d1186-90c6-426b-aa02-758d811fb743\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx" Oct 31 13:14:13.033226 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:13.032924 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx" Oct 31 13:14:13.366968 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:14:13.366913 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd77d1186_90c6_426b_aa02_758d811fb743.slice/crio-8f46f617713e147f0522ba19dc9eda343dd937f2435961fbb8694dbec5ad8f65 WatchSource:0}: Error finding container 8f46f617713e147f0522ba19dc9eda343dd937f2435961fbb8694dbec5ad8f65: Status 404 returned error can't find the container with id 8f46f617713e147f0522ba19dc9eda343dd937f2435961fbb8694dbec5ad8f65 Oct 31 13:14:13.372288 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:13.372256 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx"] Oct 31 13:14:13.394007 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:13.392915 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" event={"ID":"0109302a-8951-450d-a280-98f4ab4fd093","Type":"ContainerStarted","Data":"acf7dbe173f51d07eeea94b98dc33b9a2eb190503f877531ddab9c3c296a85e2"} Oct 31 13:14:13.394007 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:13.392994 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" event={"ID":"0109302a-8951-450d-a280-98f4ab4fd093","Type":"ContainerStarted","Data":"e3eeff3bd202759aa63e639fccc3f979423d5d7c031b22ad8d80f9ebe0db33ad"} Oct 31 13:14:13.426525 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:13.426459 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" podStartSLOduration=2.426438646 podStartE2EDuration="2.426438646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:14:13.423080754 +0000 UTC m=+2504.130667049" watchObservedRunningTime="2026-10-31 13:14:13.426438646 +0000 UTC m=+2504.134024944" Oct 31 13:14:14.123219 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:14.123190 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" Oct 31 13:14:14.124753 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:14.124727 5693 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 31 13:14:14.124883 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:14.124775 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" podUID="0109302a-8951-450d-a280-98f4ab4fd093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 31 13:14:14.407314 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:14.407277 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx" event={"ID":"d77d1186-90c6-426b-aa02-758d811fb743","Type":"ContainerStarted","Data":"4638ad65c377854e5671af3908a96c59e5efbc7c61dd47d3c474c2e91196f734"} Oct 31 13:14:14.407518 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:14.407330 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx" event={"ID":"d77d1186-90c6-426b-aa02-758d811fb743","Type":"ContainerStarted","Data":"8f46f617713e147f0522ba19dc9eda343dd937f2435961fbb8694dbec5ad8f65"} Oct 31 13:14:14.407518 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:14.407394 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx" Oct 31 13:14:14.436877 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:14.436824 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx" podStartSLOduration=2.436807961 podStartE2EDuration="2.436807961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:14:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:14:14.433192838 +0000 UTC m=+2505.140779135" watchObservedRunningTime="2026-10-31 13:14:14.436807961 +0000 UTC m=+2505.144394256" Oct 31 13:14:15.123236 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:15.123200 5693 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 31 13:14:15.123683 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:15.123254 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" podUID="0109302a-8951-450d-a280-98f4ab4fd093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 31 13:14:15.234821 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:15.234794 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0c1fbb-9c8cfbcf-rthxp" Oct 31 13:14:15.234966 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:15.234867 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-0c1fbb-5dcd8b5b65-qlkdw" Oct 31 13:14:16.122938 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:16.122898 5693 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 31 13:14:16.123114 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:16.122951 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" podUID="0109302a-8951-450d-a280-98f4ab4fd093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 31 13:14:17.122633 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:17.122593 5693 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 31 13:14:17.123055 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:17.122707 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" podUID="0109302a-8951-450d-a280-98f4ab4fd093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 31 13:14:18.123114 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:18.123076 5693 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 31 13:14:18.123531 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:18.123127 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" podUID="0109302a-8951-450d-a280-98f4ab4fd093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 31 13:14:18.711446 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:14:18.711406 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:14:19.130060 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:19.129973 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" Oct 31 13:14:19.130514 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:19.130494 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" Oct 31 13:14:19.131216 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:19.131198 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" Oct 31 13:14:22.368368 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:22.368338 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-c2c98b-755946465f-m2g26" Oct 31 13:14:24.451530 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:24.451462 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk"] Oct 31 13:14:24.456579 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:24.456553 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk" Oct 31 13:14:24.473324 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:24.473300 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-0c1fbb-serving-cert\"" Oct 31 13:14:24.493530 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:24.493503 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk"] Oct 31 13:14:24.531398 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:24.531350 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/942b3c70-0134-49ee-a30e-9631e318b001-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk\" (UID: \"942b3c70-0134-49ee-a30e-9631e318b001\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk" Oct 31 13:14:24.531760 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:24.531733 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/942b3c70-0134-49ee-a30e-9631e318b001-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk\" (UID: \"942b3c70-0134-49ee-a30e-9631e318b001\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk" Oct 31 13:14:24.531891 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:24.531820 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4txgn\" (UniqueName: \"kubernetes.io/projected/942b3c70-0134-49ee-a30e-9631e318b001-kube-api-access-4txgn\") pod \"maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk\" (UID: \"942b3c70-0134-49ee-a30e-9631e318b001\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk" Oct 31 13:14:24.531961 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:24.531913 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/942b3c70-0134-49ee-a30e-9631e318b001-metrics-certs\") pod \"maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk\" (UID: \"942b3c70-0134-49ee-a30e-9631e318b001\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk" Oct 31 13:14:24.632990 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:24.632958 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/942b3c70-0134-49ee-a30e-9631e318b001-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk\" (UID: \"942b3c70-0134-49ee-a30e-9631e318b001\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk" Oct 31 13:14:24.633171 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:24.633074 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/942b3c70-0134-49ee-a30e-9631e318b001-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk\" (UID: \"942b3c70-0134-49ee-a30e-9631e318b001\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk" Oct 31 13:14:24.633171 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:24.633119 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4txgn\" (UniqueName: \"kubernetes.io/projected/942b3c70-0134-49ee-a30e-9631e318b001-kube-api-access-4txgn\") pod \"maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk\" (UID: \"942b3c70-0134-49ee-a30e-9631e318b001\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk" Oct 31 13:14:24.633320 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:24.633176 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/942b3c70-0134-49ee-a30e-9631e318b001-metrics-certs\") pod \"maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk\" (UID: \"942b3c70-0134-49ee-a30e-9631e318b001\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk" Oct 31 13:14:24.633691 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:24.633663 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/942b3c70-0134-49ee-a30e-9631e318b001-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk\" (UID: \"942b3c70-0134-49ee-a30e-9631e318b001\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk" Oct 31 13:14:24.635848 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:24.635824 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/942b3c70-0134-49ee-a30e-9631e318b001-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk\" (UID: \"942b3c70-0134-49ee-a30e-9631e318b001\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk" Oct 31 13:14:24.636058 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:24.636034 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/942b3c70-0134-49ee-a30e-9631e318b001-metrics-certs\") pod \"maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk\" (UID: \"942b3c70-0134-49ee-a30e-9631e318b001\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk" Oct 31 13:14:24.685678 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:24.682950 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4txgn\" (UniqueName: \"kubernetes.io/projected/942b3c70-0134-49ee-a30e-9631e318b001-kube-api-access-4txgn\") pod \"maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk\" (UID: \"942b3c70-0134-49ee-a30e-9631e318b001\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk" Oct 31 13:14:24.772747 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:24.771176 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk" Oct 31 13:14:25.078131 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:25.078098 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk"] Oct 31 13:14:25.421006 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:25.420980 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx" Oct 31 13:14:25.509173 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:25.509135 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk" event={"ID":"942b3c70-0134-49ee-a30e-9631e318b001","Type":"ContainerStarted","Data":"56c725c5e8d6a654a80215c1ca6e9fed078215ae1babe8a44d51efaa70feab5a"} Oct 31 13:14:26.520453 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:26.519505 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk" event={"ID":"942b3c70-0134-49ee-a30e-9631e318b001","Type":"ContainerStarted","Data":"1e0a99efad4594b50aa60785891c4ad1619afd1e689bc391f51045476d718275"} Oct 31 13:14:26.520453 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:26.519579 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk" Oct 31 13:14:26.543453 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:26.543407 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk" podStartSLOduration=2.0266325 podStartE2EDuration="2.54339279s" totalImagesPullingTime="516.76029ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:14:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:14:26.540029703 +0000 UTC m=+2517.247615999" watchObservedRunningTime="2026-10-31 13:14:26.54339279 +0000 UTC m=+2517.250979086" Oct 31 13:14:32.089969 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:32.089938 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5"] Oct 31 13:14:32.095216 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:32.095193 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5" Oct 31 13:14:32.107269 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:32.107244 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-c2c98b-serving-cert\"" Oct 31 13:14:32.117116 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:32.117085 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5"] Oct 31 13:14:32.149382 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:32.149347 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfcw4\" (UniqueName: \"kubernetes.io/projected/32747ebc-c185-4a7f-877a-f4903862b567-kube-api-access-vfcw4\") pod \"maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5\" (UID: \"32747ebc-c185-4a7f-877a-f4903862b567\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5" Oct 31 13:14:32.149542 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:32.149528 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32747ebc-c185-4a7f-877a-f4903862b567-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5\" (UID: \"32747ebc-c185-4a7f-877a-f4903862b567\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5" Oct 31 13:14:32.149589 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:32.149567 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32747ebc-c185-4a7f-877a-f4903862b567-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5\" (UID: \"32747ebc-c185-4a7f-877a-f4903862b567\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5" Oct 31 13:14:32.149802 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:32.149780 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32747ebc-c185-4a7f-877a-f4903862b567-metrics-certs\") pod \"maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5\" (UID: \"32747ebc-c185-4a7f-877a-f4903862b567\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5" Oct 31 13:14:32.254674 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:32.252395 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32747ebc-c185-4a7f-877a-f4903862b567-metrics-certs\") pod \"maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5\" (UID: \"32747ebc-c185-4a7f-877a-f4903862b567\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5" Oct 31 13:14:32.254674 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:32.252589 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfcw4\" (UniqueName: \"kubernetes.io/projected/32747ebc-c185-4a7f-877a-f4903862b567-kube-api-access-vfcw4\") pod \"maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5\" (UID: \"32747ebc-c185-4a7f-877a-f4903862b567\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5" Oct 31 13:14:32.254674 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:32.252762 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32747ebc-c185-4a7f-877a-f4903862b567-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5\" (UID: \"32747ebc-c185-4a7f-877a-f4903862b567\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5" Oct 31 13:14:32.254674 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:32.252806 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32747ebc-c185-4a7f-877a-f4903862b567-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5\" (UID: \"32747ebc-c185-4a7f-877a-f4903862b567\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5" Oct 31 13:14:32.255158 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:32.254685 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32747ebc-c185-4a7f-877a-f4903862b567-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5\" (UID: \"32747ebc-c185-4a7f-877a-f4903862b567\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5" Oct 31 13:14:32.259754 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:32.258132 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32747ebc-c185-4a7f-877a-f4903862b567-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5\" (UID: \"32747ebc-c185-4a7f-877a-f4903862b567\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5" Oct 31 13:14:32.259754 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:32.259444 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32747ebc-c185-4a7f-877a-f4903862b567-metrics-certs\") pod \"maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5\" (UID: \"32747ebc-c185-4a7f-877a-f4903862b567\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5" Oct 31 13:14:32.270253 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:32.270219 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfcw4\" (UniqueName: \"kubernetes.io/projected/32747ebc-c185-4a7f-877a-f4903862b567-kube-api-access-vfcw4\") pod \"maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5\" (UID: \"32747ebc-c185-4a7f-877a-f4903862b567\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5" Oct 31 13:14:32.410753 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:32.410722 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5" Oct 31 13:14:32.537062 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:32.537030 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk" Oct 31 13:14:32.703468 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:32.703438 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5"] Oct 31 13:14:33.655901 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:33.655864 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5" event={"ID":"32747ebc-c185-4a7f-877a-f4903862b567","Type":"ContainerStarted","Data":"410c4bc509c278aea61f47a4cbe1f74f3eb6fb486ed2946dca12a281d108101c"} Oct 31 13:14:33.656371 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:33.655951 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5" event={"ID":"32747ebc-c185-4a7f-877a-f4903862b567","Type":"ContainerStarted","Data":"42ac5a60c5321a2f90a9bf8e3c4deff4a946e7310bc54159ecb20a0b2ab6be85"} Oct 31 13:14:33.656371 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:33.656140 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5" Oct 31 13:14:33.680396 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:33.680351 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5" podStartSLOduration=1.188988602 podStartE2EDuration="1.680337644s" totalImagesPullingTime="491.349042ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:14:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:14:33.676782382 +0000 UTC m=+2524.384368679" watchObservedRunningTime="2026-10-31 13:14:33.680337644 +0000 UTC m=+2524.387923938" Oct 31 13:14:39.670380 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:39.670310 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5" Oct 31 13:14:48.693694 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:14:48.693618 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:14:57.036408 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:57.036369 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv"] Oct 31 13:14:57.042958 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:57.042930 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" Oct 31 13:14:57.067396 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:57.067366 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv"] Oct 31 13:14:57.135678 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:57.135621 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92d4bf25-3aad-42dc-989f-88c4d68dd188-istio-podinfo\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv\" (UID: \"92d4bf25-3aad-42dc-989f-88c4d68dd188\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" Oct 31 13:14:57.135856 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:57.135739 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92d4bf25-3aad-42dc-989f-88c4d68dd188-workload-certs\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv\" (UID: \"92d4bf25-3aad-42dc-989f-88c4d68dd188\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" Oct 31 13:14:57.135856 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:57.135780 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7p9v\" (UniqueName: \"kubernetes.io/projected/92d4bf25-3aad-42dc-989f-88c4d68dd188-kube-api-access-m7p9v\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv\" (UID: \"92d4bf25-3aad-42dc-989f-88c4d68dd188\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" Oct 31 13:14:57.135935 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:57.135902 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d4bf25-3aad-42dc-989f-88c4d68dd188-credential-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv\" (UID: \"92d4bf25-3aad-42dc-989f-88c4d68dd188\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" Oct 31 13:14:57.135974 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:57.135930 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92d4bf25-3aad-42dc-989f-88c4d68dd188-istio-data\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv\" (UID: \"92d4bf25-3aad-42dc-989f-88c4d68dd188\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" Oct 31 13:14:57.135974 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:57.135961 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92d4bf25-3aad-42dc-989f-88c4d68dd188-istiod-ca-cert\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv\" (UID: \"92d4bf25-3aad-42dc-989f-88c4d68dd188\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" Oct 31 13:14:57.136121 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:57.136067 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d4bf25-3aad-42dc-989f-88c4d68dd188-workload-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv\" (UID: \"92d4bf25-3aad-42dc-989f-88c4d68dd188\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" Oct 31 13:14:57.136225 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:57.136204 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92d4bf25-3aad-42dc-989f-88c4d68dd188-istio-token\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv\" (UID: \"92d4bf25-3aad-42dc-989f-88c4d68dd188\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" Oct 31 13:14:57.136309 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:57.136294 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92d4bf25-3aad-42dc-989f-88c4d68dd188-istio-envoy\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv\" (UID: \"92d4bf25-3aad-42dc-989f-88c4d68dd188\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" Oct 31 13:14:57.237334 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:57.237301 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92d4bf25-3aad-42dc-989f-88c4d68dd188-istio-token\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv\" (UID: \"92d4bf25-3aad-42dc-989f-88c4d68dd188\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" Oct 31 13:14:57.237549 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:57.237357 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92d4bf25-3aad-42dc-989f-88c4d68dd188-istio-envoy\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv\" (UID: \"92d4bf25-3aad-42dc-989f-88c4d68dd188\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" Oct 31 13:14:57.237549 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:57.237436 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92d4bf25-3aad-42dc-989f-88c4d68dd188-istio-podinfo\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv\" (UID: \"92d4bf25-3aad-42dc-989f-88c4d68dd188\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" Oct 31 13:14:57.237549 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:57.237460 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92d4bf25-3aad-42dc-989f-88c4d68dd188-workload-certs\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv\" (UID: \"92d4bf25-3aad-42dc-989f-88c4d68dd188\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" Oct 31 13:14:57.237549 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:57.237488 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7p9v\" (UniqueName: \"kubernetes.io/projected/92d4bf25-3aad-42dc-989f-88c4d68dd188-kube-api-access-m7p9v\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv\" (UID: \"92d4bf25-3aad-42dc-989f-88c4d68dd188\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" Oct 31 13:14:57.237860 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:57.237563 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d4bf25-3aad-42dc-989f-88c4d68dd188-credential-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv\" (UID: \"92d4bf25-3aad-42dc-989f-88c4d68dd188\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" Oct 31 13:14:57.237860 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:57.237598 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92d4bf25-3aad-42dc-989f-88c4d68dd188-istio-data\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv\" (UID: \"92d4bf25-3aad-42dc-989f-88c4d68dd188\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" Oct 31 13:14:57.237860 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:57.237656 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92d4bf25-3aad-42dc-989f-88c4d68dd188-istiod-ca-cert\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv\" (UID: \"92d4bf25-3aad-42dc-989f-88c4d68dd188\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" Oct 31 13:14:57.237860 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:57.237720 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d4bf25-3aad-42dc-989f-88c4d68dd188-workload-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv\" (UID: \"92d4bf25-3aad-42dc-989f-88c4d68dd188\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" Oct 31 13:14:57.238146 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:57.237929 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92d4bf25-3aad-42dc-989f-88c4d68dd188-workload-certs\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv\" (UID: \"92d4bf25-3aad-42dc-989f-88c4d68dd188\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" Oct 31 13:14:57.238146 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:57.238049 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92d4bf25-3aad-42dc-989f-88c4d68dd188-istio-data\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv\" (UID: \"92d4bf25-3aad-42dc-989f-88c4d68dd188\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" Oct 31 13:14:57.238146 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:57.238093 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d4bf25-3aad-42dc-989f-88c4d68dd188-credential-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv\" (UID: \"92d4bf25-3aad-42dc-989f-88c4d68dd188\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" Oct 31 13:14:57.238257 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:57.238145 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d4bf25-3aad-42dc-989f-88c4d68dd188-workload-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv\" (UID: \"92d4bf25-3aad-42dc-989f-88c4d68dd188\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" Oct 31 13:14:57.238464 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:57.238444 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92d4bf25-3aad-42dc-989f-88c4d68dd188-istiod-ca-cert\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv\" (UID: \"92d4bf25-3aad-42dc-989f-88c4d68dd188\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" Oct 31 13:14:57.239948 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:57.239927 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92d4bf25-3aad-42dc-989f-88c4d68dd188-istio-envoy\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv\" (UID: \"92d4bf25-3aad-42dc-989f-88c4d68dd188\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" Oct 31 13:14:57.240184 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:57.240166 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92d4bf25-3aad-42dc-989f-88c4d68dd188-istio-podinfo\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv\" (UID: \"92d4bf25-3aad-42dc-989f-88c4d68dd188\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" Oct 31 13:14:57.260977 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:57.260950 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7p9v\" (UniqueName: \"kubernetes.io/projected/92d4bf25-3aad-42dc-989f-88c4d68dd188-kube-api-access-m7p9v\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv\" (UID: \"92d4bf25-3aad-42dc-989f-88c4d68dd188\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" Oct 31 13:14:57.262685 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:57.262627 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92d4bf25-3aad-42dc-989f-88c4d68dd188-istio-token\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv\" (UID: \"92d4bf25-3aad-42dc-989f-88c4d68dd188\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" Oct 31 13:14:57.372205 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:57.372119 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" Oct 31 13:14:57.671320 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:57.671290 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv"] Oct 31 13:14:57.671555 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:14:57.671534 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92d4bf25_3aad_42dc_989f_88c4d68dd188.slice/crio-297dde0307fee3ffa3638a18393bf5b65864c6e11c92635dbd466d530d5d3c3f WatchSource:0}: Error finding container 297dde0307fee3ffa3638a18393bf5b65864c6e11c92635dbd466d530d5d3c3f: Status 404 returned error can't find the container with id 297dde0307fee3ffa3638a18393bf5b65864c6e11c92635dbd466d530d5d3c3f Oct 31 13:14:57.674939 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:57.674903 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:14:57.675037 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:57.674971 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:14:57.675037 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:57.675020 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:14:57.901316 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:57.901282 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" event={"ID":"92d4bf25-3aad-42dc-989f-88c4d68dd188","Type":"ContainerStarted","Data":"e19fa931f2ee8d9f63c64b1975f3a3734222762613789ecd2a3f1c992042f8d6"} Oct 31 13:14:57.901703 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:57.901337 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" event={"ID":"92d4bf25-3aad-42dc-989f-88c4d68dd188","Type":"ContainerStarted","Data":"297dde0307fee3ffa3638a18393bf5b65864c6e11c92635dbd466d530d5d3c3f"} Oct 31 13:14:57.930608 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:57.930496 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" podStartSLOduration=1.930482986 podStartE2EDuration="1.930482986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:14:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:14:57.926226313 +0000 UTC m=+2548.633812609" watchObservedRunningTime="2026-10-31 13:14:57.930482986 +0000 UTC m=+2548.638069358" Oct 31 13:14:58.373324 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:58.373237 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" Oct 31 13:14:58.374596 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:58.374572 5693 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Oct 31 13:14:58.374744 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:58.374632 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" podUID="92d4bf25-3aad-42dc-989f-88c4d68dd188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Oct 31 13:14:59.374342 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:59.372542 5693 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Oct 31 13:14:59.374342 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:14:59.372605 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" podUID="92d4bf25-3aad-42dc-989f-88c4d68dd188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Oct 31 13:15:00.310754 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.310715 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890875-wdmvp"] Oct 31 13:15:00.315843 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.315819 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890875-wdmvp" Oct 31 13:15:00.329770 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.329741 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890875-wdmvp"] Oct 31 13:15:00.336499 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.336455 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74df98d2-73a9-43bb-b0c2-258f69413210-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890875-wdmvp\" (UID: \"74df98d2-73a9-43bb-b0c2-258f69413210\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890875-wdmvp" Oct 31 13:15:00.336669 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.336519 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnmcg\" (UniqueName: \"kubernetes.io/projected/74df98d2-73a9-43bb-b0c2-258f69413210-kube-api-access-lnmcg\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890875-wdmvp\" (UID: \"74df98d2-73a9-43bb-b0c2-258f69413210\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890875-wdmvp" Oct 31 13:15:00.337058 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.337035 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890875-cxsps"] Oct 31 13:15:00.344376 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.344354 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890875-hnnqk"] Oct 31 13:15:00.345779 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.344492 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-cxsps" Oct 31 13:15:00.351629 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.351610 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890875-hnnqk" Oct 31 13:15:00.354773 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.354747 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890875-cxsps"] Oct 31 13:15:00.413666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.400630 5693 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Oct 31 13:15:00.413666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.400706 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" podUID="92d4bf25-3aad-42dc-989f-88c4d68dd188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Oct 31 13:15:00.417403 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.417375 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890875-hnnqk"] Oct 31 13:15:00.421366 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.421335 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:15:00.424068 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.424044 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890875-6rsmb"] Oct 31 13:15:00.430356 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.430339 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890875-6rsmb" Oct 31 13:15:00.433235 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.430817 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890875-hvbgx"] Oct 31 13:15:00.435517 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.435378 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890875-hvbgx" Oct 31 13:15:00.441784 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.441450 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a135ae97-5e5e-4d5d-b329-90a8de080e16-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890875-cxsps\" (UID: \"a135ae97-5e5e-4d5d-b329-90a8de080e16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-cxsps" Oct 31 13:15:00.442027 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.441990 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q87k8\" (UniqueName: \"kubernetes.io/projected/cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31-kube-api-access-q87k8\") pod \"maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890875-6rsmb\" (UID: \"cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890875-6rsmb" Oct 31 13:15:00.442134 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.442100 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krcq6\" (UniqueName: \"kubernetes.io/projected/705199d4-0bb0-4d6f-b08e-7cb3d8582e24-kube-api-access-krcq6\") pod \"maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890875-hnnqk\" (UID: \"705199d4-0bb0-4d6f-b08e-7cb3d8582e24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890875-hnnqk" Oct 31 13:15:00.442194 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.442152 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/705199d4-0bb0-4d6f-b08e-7cb3d8582e24-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890875-hnnqk\" (UID: \"705199d4-0bb0-4d6f-b08e-7cb3d8582e24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890875-hnnqk" Oct 31 13:15:00.442246 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.442205 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74df98d2-73a9-43bb-b0c2-258f69413210-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890875-wdmvp\" (UID: \"74df98d2-73a9-43bb-b0c2-258f69413210\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890875-wdmvp" Oct 31 13:15:00.442246 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.442235 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnmcg\" (UniqueName: \"kubernetes.io/projected/74df98d2-73a9-43bb-b0c2-258f69413210-kube-api-access-lnmcg\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890875-wdmvp\" (UID: \"74df98d2-73a9-43bb-b0c2-258f69413210\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890875-wdmvp" Oct 31 13:15:00.442349 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.442282 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qmfs\" (UniqueName: \"kubernetes.io/projected/a135ae97-5e5e-4d5d-b329-90a8de080e16-kube-api-access-2qmfs\") pod \"maas-api-key-cleanup-29890875-cxsps\" (UID: \"a135ae97-5e5e-4d5d-b329-90a8de080e16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-cxsps" Oct 31 13:15:00.443038 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.443011 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890875-6rsmb\" (UID: \"cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890875-6rsmb" Oct 31 13:15:00.443223 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.443199 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74df98d2-73a9-43bb-b0c2-258f69413210-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890875-wdmvp\" (UID: \"74df98d2-73a9-43bb-b0c2-258f69413210\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890875-wdmvp" Oct 31 13:15:00.444219 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.444191 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890875-tc5zf"] Oct 31 13:15:00.452138 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.452110 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890875-hvbgx"] Oct 31 13:15:00.452261 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.452218 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890875-tc5zf" Oct 31 13:15:00.467131 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.467104 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnmcg\" (UniqueName: \"kubernetes.io/projected/74df98d2-73a9-43bb-b0c2-258f69413210-kube-api-access-lnmcg\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890875-wdmvp\" (UID: \"74df98d2-73a9-43bb-b0c2-258f69413210\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890875-wdmvp" Oct 31 13:15:00.472955 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.472884 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890875-6rsmb"] Oct 31 13:15:00.518879 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.518156 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890875-7jpzj"] Oct 31 13:15:00.542837 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.542791 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890875-r8b8z"] Oct 31 13:15:00.548662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.547572 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890875-6rsmb\" (UID: \"cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890875-6rsmb" Oct 31 13:15:00.548662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.547672 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8rm5\" (UniqueName: \"kubernetes.io/projected/cb0c915e-681f-4fb6-bff9-91ae28572928-kube-api-access-b8rm5\") pod \"maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890875-hvbgx\" (UID: \"cb0c915e-681f-4fb6-bff9-91ae28572928\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890875-hvbgx" Oct 31 13:15:00.548662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.547739 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb0c915e-681f-4fb6-bff9-91ae28572928-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890875-hvbgx\" (UID: \"cb0c915e-681f-4fb6-bff9-91ae28572928\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890875-hvbgx" Oct 31 13:15:00.548662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.547879 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a135ae97-5e5e-4d5d-b329-90a8de080e16-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890875-cxsps\" (UID: \"a135ae97-5e5e-4d5d-b329-90a8de080e16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-cxsps" Oct 31 13:15:00.548662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.547926 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q87k8\" (UniqueName: \"kubernetes.io/projected/cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31-kube-api-access-q87k8\") pod \"maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890875-6rsmb\" (UID: \"cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890875-6rsmb" Oct 31 13:15:00.548662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.547975 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krcq6\" (UniqueName: \"kubernetes.io/projected/705199d4-0bb0-4d6f-b08e-7cb3d8582e24-kube-api-access-krcq6\") pod \"maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890875-hnnqk\" (UID: \"705199d4-0bb0-4d6f-b08e-7cb3d8582e24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890875-hnnqk" Oct 31 13:15:00.548662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.548020 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/705199d4-0bb0-4d6f-b08e-7cb3d8582e24-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890875-hnnqk\" (UID: \"705199d4-0bb0-4d6f-b08e-7cb3d8582e24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890875-hnnqk" Oct 31 13:15:00.548662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.548095 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qmfs\" (UniqueName: \"kubernetes.io/projected/a135ae97-5e5e-4d5d-b329-90a8de080e16-kube-api-access-2qmfs\") pod \"maas-api-key-cleanup-29890875-cxsps\" (UID: \"a135ae97-5e5e-4d5d-b329-90a8de080e16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-cxsps" Oct 31 13:15:00.548662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.548204 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4whr6\" (UniqueName: \"kubernetes.io/projected/22dbab0d-c6ad-4083-86bc-c5e0dfad5db3-kube-api-access-4whr6\") pod \"maas-api-key-cleanup-e2e-worker-w0-107955-29890875-tc5zf\" (UID: \"22dbab0d-c6ad-4083-86bc-c5e0dfad5db3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890875-tc5zf" Oct 31 13:15:00.548662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.548252 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22dbab0d-c6ad-4083-86bc-c5e0dfad5db3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-107955-29890875-tc5zf\" (UID: \"22dbab0d-c6ad-4083-86bc-c5e0dfad5db3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890875-tc5zf" Oct 31 13:15:00.551876 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.550959 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a135ae97-5e5e-4d5d-b329-90a8de080e16-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890875-cxsps\" (UID: \"a135ae97-5e5e-4d5d-b329-90a8de080e16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-cxsps" Oct 31 13:15:00.553632 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.553227 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/705199d4-0bb0-4d6f-b08e-7cb3d8582e24-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890875-hnnqk\" (UID: \"705199d4-0bb0-4d6f-b08e-7cb3d8582e24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890875-hnnqk" Oct 31 13:15:00.554834 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.554152 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890875-r8b8z" Oct 31 13:15:00.558607 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.558580 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890875-6rsmb\" (UID: \"cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890875-6rsmb" Oct 31 13:15:00.558920 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.558902 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890875-7jpzj" Oct 31 13:15:00.569854 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.569768 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890875-tc5zf"] Oct 31 13:15:00.596902 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.592341 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q87k8\" (UniqueName: \"kubernetes.io/projected/cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31-kube-api-access-q87k8\") pod \"maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890875-6rsmb\" (UID: \"cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890875-6rsmb" Oct 31 13:15:00.612588 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.612167 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qmfs\" (UniqueName: \"kubernetes.io/projected/a135ae97-5e5e-4d5d-b329-90a8de080e16-kube-api-access-2qmfs\") pod \"maas-api-key-cleanup-29890875-cxsps\" (UID: \"a135ae97-5e5e-4d5d-b329-90a8de080e16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-cxsps" Oct 31 13:15:00.612588 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.612360 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krcq6\" (UniqueName: \"kubernetes.io/projected/705199d4-0bb0-4d6f-b08e-7cb3d8582e24-kube-api-access-krcq6\") pod \"maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890875-hnnqk\" (UID: \"705199d4-0bb0-4d6f-b08e-7cb3d8582e24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890875-hnnqk" Oct 31 13:15:00.612846 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.612797 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890875-r8b8z"] Oct 31 13:15:00.629228 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.629190 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890875-7jpzj"] Oct 31 13:15:00.639941 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.639913 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890875-wdmvp" Oct 31 13:15:00.653029 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.651282 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jc5g\" (UniqueName: \"kubernetes.io/projected/eb473765-2907-4d0a-85f9-5f9fa76b77a0-kube-api-access-4jc5g\") pod \"maas-api-key-cleanup-e2e-extmod-a-54a16b-29890875-7jpzj\" (UID: \"eb473765-2907-4d0a-85f9-5f9fa76b77a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890875-7jpzj" Oct 31 13:15:00.653029 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.651424 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blfpk\" (UniqueName: \"kubernetes.io/projected/195c9f96-e37a-46e1-b2c7-8f8a15fff884-kube-api-access-blfpk\") pod \"maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890875-r8b8z\" (UID: \"195c9f96-e37a-46e1-b2c7-8f8a15fff884\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890875-r8b8z" Oct 31 13:15:00.653029 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.651512 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/195c9f96-e37a-46e1-b2c7-8f8a15fff884-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890875-r8b8z\" (UID: \"195c9f96-e37a-46e1-b2c7-8f8a15fff884\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890875-r8b8z" Oct 31 13:15:00.653029 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.651609 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb473765-2907-4d0a-85f9-5f9fa76b77a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-54a16b-29890875-7jpzj\" (UID: \"eb473765-2907-4d0a-85f9-5f9fa76b77a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890875-7jpzj" Oct 31 13:15:00.653029 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.651713 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4whr6\" (UniqueName: \"kubernetes.io/projected/22dbab0d-c6ad-4083-86bc-c5e0dfad5db3-kube-api-access-4whr6\") pod \"maas-api-key-cleanup-e2e-worker-w0-107955-29890875-tc5zf\" (UID: \"22dbab0d-c6ad-4083-86bc-c5e0dfad5db3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890875-tc5zf" Oct 31 13:15:00.653029 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.651891 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22dbab0d-c6ad-4083-86bc-c5e0dfad5db3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-107955-29890875-tc5zf\" (UID: \"22dbab0d-c6ad-4083-86bc-c5e0dfad5db3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890875-tc5zf" Oct 31 13:15:00.653029 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.651980 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8rm5\" (UniqueName: \"kubernetes.io/projected/cb0c915e-681f-4fb6-bff9-91ae28572928-kube-api-access-b8rm5\") pod \"maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890875-hvbgx\" (UID: \"cb0c915e-681f-4fb6-bff9-91ae28572928\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890875-hvbgx" Oct 31 13:15:00.653029 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.652047 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb0c915e-681f-4fb6-bff9-91ae28572928-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890875-hvbgx\" (UID: \"cb0c915e-681f-4fb6-bff9-91ae28572928\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890875-hvbgx" Oct 31 13:15:00.653029 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.652891 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb0c915e-681f-4fb6-bff9-91ae28572928-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890875-hvbgx\" (UID: \"cb0c915e-681f-4fb6-bff9-91ae28572928\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890875-hvbgx" Oct 31 13:15:00.653029 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.652987 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22dbab0d-c6ad-4083-86bc-c5e0dfad5db3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-107955-29890875-tc5zf\" (UID: \"22dbab0d-c6ad-4083-86bc-c5e0dfad5db3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890875-tc5zf" Oct 31 13:15:00.670778 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.670747 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-cxsps" Oct 31 13:15:00.719226 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.717302 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890875-hnnqk" Oct 31 13:15:00.727878 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.721278 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8rm5\" (UniqueName: \"kubernetes.io/projected/cb0c915e-681f-4fb6-bff9-91ae28572928-kube-api-access-b8rm5\") pod \"maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890875-hvbgx\" (UID: \"cb0c915e-681f-4fb6-bff9-91ae28572928\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890875-hvbgx" Oct 31 13:15:00.727878 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.724286 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4whr6\" (UniqueName: \"kubernetes.io/projected/22dbab0d-c6ad-4083-86bc-c5e0dfad5db3-kube-api-access-4whr6\") pod \"maas-api-key-cleanup-e2e-worker-w0-107955-29890875-tc5zf\" (UID: \"22dbab0d-c6ad-4083-86bc-c5e0dfad5db3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890875-tc5zf" Oct 31 13:15:00.757299 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.753872 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jc5g\" (UniqueName: \"kubernetes.io/projected/eb473765-2907-4d0a-85f9-5f9fa76b77a0-kube-api-access-4jc5g\") pod \"maas-api-key-cleanup-e2e-extmod-a-54a16b-29890875-7jpzj\" (UID: \"eb473765-2907-4d0a-85f9-5f9fa76b77a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890875-7jpzj" Oct 31 13:15:00.757299 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.754007 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blfpk\" (UniqueName: \"kubernetes.io/projected/195c9f96-e37a-46e1-b2c7-8f8a15fff884-kube-api-access-blfpk\") pod \"maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890875-r8b8z\" (UID: \"195c9f96-e37a-46e1-b2c7-8f8a15fff884\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890875-r8b8z" Oct 31 13:15:00.757299 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.754121 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/195c9f96-e37a-46e1-b2c7-8f8a15fff884-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890875-r8b8z\" (UID: \"195c9f96-e37a-46e1-b2c7-8f8a15fff884\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890875-r8b8z" Oct 31 13:15:00.757299 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.754233 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb473765-2907-4d0a-85f9-5f9fa76b77a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-54a16b-29890875-7jpzj\" (UID: \"eb473765-2907-4d0a-85f9-5f9fa76b77a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890875-7jpzj" Oct 31 13:15:00.757299 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.755239 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb473765-2907-4d0a-85f9-5f9fa76b77a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-54a16b-29890875-7jpzj\" (UID: \"eb473765-2907-4d0a-85f9-5f9fa76b77a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890875-7jpzj" Oct 31 13:15:00.757299 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.757241 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/195c9f96-e37a-46e1-b2c7-8f8a15fff884-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890875-r8b8z\" (UID: \"195c9f96-e37a-46e1-b2c7-8f8a15fff884\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890875-r8b8z" Oct 31 13:15:00.802671 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.761949 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890875-6rsmb" Oct 31 13:15:00.802671 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.793552 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jc5g\" (UniqueName: \"kubernetes.io/projected/eb473765-2907-4d0a-85f9-5f9fa76b77a0-kube-api-access-4jc5g\") pod \"maas-api-key-cleanup-e2e-extmod-a-54a16b-29890875-7jpzj\" (UID: \"eb473765-2907-4d0a-85f9-5f9fa76b77a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890875-7jpzj" Oct 31 13:15:00.853455 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.853419 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blfpk\" (UniqueName: \"kubernetes.io/projected/195c9f96-e37a-46e1-b2c7-8f8a15fff884-kube-api-access-blfpk\") pod \"maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890875-r8b8z\" (UID: \"195c9f96-e37a-46e1-b2c7-8f8a15fff884\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890875-r8b8z" Oct 31 13:15:00.865894 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.865860 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890875-hvbgx" Oct 31 13:15:00.880031 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.872386 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890875-tc5zf" Oct 31 13:15:00.944562 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:00.944483 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890875-7jpzj" Oct 31 13:15:01.044201 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:01.026112 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890875-r8b8z" Oct 31 13:15:01.127932 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:01.127900 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890875-wdmvp"] Oct 31 13:15:01.133531 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:15:01.133495 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74df98d2_73a9_43bb_b0c2_258f69413210.slice/crio-072d24d284052cc9e078ef2d8d8d887b5c8ecbffb663ea4d5cb3d5ce27de4fe0 WatchSource:0}: Error finding container 072d24d284052cc9e078ef2d8d8d887b5c8ecbffb663ea4d5cb3d5ce27de4fe0: Status 404 returned error can't find the container with id 072d24d284052cc9e078ef2d8d8d887b5c8ecbffb663ea4d5cb3d5ce27de4fe0 Oct 31 13:15:01.373318 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:01.373230 5693 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Oct 31 13:15:01.373318 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:01.373278 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" podUID="92d4bf25-3aad-42dc-989f-88c4d68dd188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Oct 31 13:15:01.511544 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:01.511514 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890875-cxsps"] Oct 31 13:15:01.524314 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:15:01.524259 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda135ae97_5e5e_4d5d_b329_90a8de080e16.slice/crio-a49d8da0154627f0fc77dfc2a89e03c670ba76e738937da50aa26acaae7eea60 WatchSource:0}: Error finding container a49d8da0154627f0fc77dfc2a89e03c670ba76e738937da50aa26acaae7eea60: Status 404 returned error can't find the container with id a49d8da0154627f0fc77dfc2a89e03c670ba76e738937da50aa26acaae7eea60 Oct 31 13:15:01.540463 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:01.538461 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890875-hnnqk"] Oct 31 13:15:01.549331 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:15:01.549299 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod705199d4_0bb0_4d6f_b08e_7cb3d8582e24.slice/crio-122c8b418470d2fc51098dfcdb704806c24575bbb8dbdd197394bdb9fb0ebf41 WatchSource:0}: Error finding container 122c8b418470d2fc51098dfcdb704806c24575bbb8dbdd197394bdb9fb0ebf41: Status 404 returned error can't find the container with id 122c8b418470d2fc51098dfcdb704806c24575bbb8dbdd197394bdb9fb0ebf41 Oct 31 13:15:01.913164 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:15:01.913136 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc6d20d_2b6f_4ce4_8d5f_28eb518f8b31.slice/crio-31c1470cbf38e362eb29b7c1d222bafe4f64043ad1af2ccdbedac7da89644f90 WatchSource:0}: Error finding container 31c1470cbf38e362eb29b7c1d222bafe4f64043ad1af2ccdbedac7da89644f90: Status 404 returned error can't find the container with id 31c1470cbf38e362eb29b7c1d222bafe4f64043ad1af2ccdbedac7da89644f90 Oct 31 13:15:01.913354 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:01.913333 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890875-6rsmb"] Oct 31 13:15:01.923132 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:01.922921 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890875-tc5zf"] Oct 31 13:15:01.929964 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:15:01.929422 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb0c915e_681f_4fb6_bff9_91ae28572928.slice/crio-8b9d8103bb7d31c66f6c10a0122e855d817138ebf10b9da5ff67e84a172a9181 WatchSource:0}: Error finding container 8b9d8103bb7d31c66f6c10a0122e855d817138ebf10b9da5ff67e84a172a9181: Status 404 returned error can't find the container with id 8b9d8103bb7d31c66f6c10a0122e855d817138ebf10b9da5ff67e84a172a9181 Oct 31 13:15:01.937052 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:01.934924 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890875-7jpzj"] Oct 31 13:15:01.942531 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:01.942510 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890875-hvbgx"] Oct 31 13:15:01.949554 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:01.948757 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890875-r8b8z"] Oct 31 13:15:01.959750 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:01.959721 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890875-hvbgx" event={"ID":"cb0c915e-681f-4fb6-bff9-91ae28572928","Type":"ContainerStarted","Data":"8b9d8103bb7d31c66f6c10a0122e855d817138ebf10b9da5ff67e84a172a9181"} Oct 31 13:15:01.964761 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:01.964735 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890875-6rsmb" event={"ID":"cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31","Type":"ContainerStarted","Data":"31c1470cbf38e362eb29b7c1d222bafe4f64043ad1af2ccdbedac7da89644f90"} Oct 31 13:15:01.973743 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:01.973709 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890875-7jpzj" event={"ID":"eb473765-2907-4d0a-85f9-5f9fa76b77a0","Type":"ContainerStarted","Data":"b6e3b5f89394d8b7576e036b95fccedb89ec065511f15580b02af617f1cb9f58"} Oct 31 13:15:01.975662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:01.975614 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890875-hnnqk" event={"ID":"705199d4-0bb0-4d6f-b08e-7cb3d8582e24","Type":"ContainerStarted","Data":"122c8b418470d2fc51098dfcdb704806c24575bbb8dbdd197394bdb9fb0ebf41"} Oct 31 13:15:02.007920 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:02.000409 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-cxsps" event={"ID":"a135ae97-5e5e-4d5d-b329-90a8de080e16","Type":"ContainerStarted","Data":"a49d8da0154627f0fc77dfc2a89e03c670ba76e738937da50aa26acaae7eea60"} Oct 31 13:15:02.007920 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:02.002193 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890875-r8b8z" event={"ID":"195c9f96-e37a-46e1-b2c7-8f8a15fff884","Type":"ContainerStarted","Data":"00cc6f23dee7a454bc97f29968e566dea2198f3621a9c7f7e8afd29d00325796"} Oct 31 13:15:02.012280 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:02.012242 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890875-wdmvp" event={"ID":"74df98d2-73a9-43bb-b0c2-258f69413210","Type":"ContainerStarted","Data":"072d24d284052cc9e078ef2d8d8d887b5c8ecbffb663ea4d5cb3d5ce27de4fe0"} Oct 31 13:15:02.014976 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:02.014946 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890875-tc5zf" event={"ID":"22dbab0d-c6ad-4083-86bc-c5e0dfad5db3","Type":"ContainerStarted","Data":"208f448f84e2996bb6e634b882bca236908375ef1b39074909419e75ab36c797"} Oct 31 13:15:02.036609 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:02.036568 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890875-wdmvp" podStartSLOduration=2.036551976 podStartE2EDuration="2.036551976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:02.034355608 +0000 UTC m=+2552.741941922" watchObservedRunningTime="2026-10-31 13:15:02.036551976 +0000 UTC m=+2552.744138270" Oct 31 13:15:02.400664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:02.372473 5693 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Oct 31 13:15:02.400664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:02.372532 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" podUID="92d4bf25-3aad-42dc-989f-88c4d68dd188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Oct 31 13:15:03.031729 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:03.031686 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890875-6rsmb" event={"ID":"cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31","Type":"ContainerStarted","Data":"f4f876d410509d8e3c63d4e36e5e301dc3c987a8a3e327eafebee2f83de7ffdb"} Oct 31 13:15:03.034866 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:03.034834 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890875-hnnqk" event={"ID":"705199d4-0bb0-4d6f-b08e-7cb3d8582e24","Type":"ContainerStarted","Data":"f51e8556e3f240db6e54e24d0e78f6a43d95421520dc76764c78be06d34f166b"} Oct 31 13:15:03.037936 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:03.037905 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-cxsps" event={"ID":"a135ae97-5e5e-4d5d-b329-90a8de080e16","Type":"ContainerStarted","Data":"dc322be6814a3bb3d8de49e049857d8db919c33782a5b2b2f205bb4374dcaf00"} Oct 31 13:15:03.039783 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:03.039745 5693 generic.go:356] "Generic (PLEG): container finished" podID="74df98d2-73a9-43bb-b0c2-258f69413210" containerID="17d87072287cc45f65d12fc4cfdc48371adb55c9ac79530be3325ca474041201" exitCode=60 Oct 31 13:15:03.039982 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:03.039962 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890875-wdmvp" event={"ID":"74df98d2-73a9-43bb-b0c2-258f69413210","Type":"ContainerDied","Data":"17d87072287cc45f65d12fc4cfdc48371adb55c9ac79530be3325ca474041201"} Oct 31 13:15:03.040173 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:03.040154 5693 scope.go:122] "RemoveContainer" containerID="17d87072287cc45f65d12fc4cfdc48371adb55c9ac79530be3325ca474041201" Oct 31 13:15:03.042538 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:03.042521 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890875-tc5zf" event={"ID":"22dbab0d-c6ad-4083-86bc-c5e0dfad5db3","Type":"ContainerStarted","Data":"aafbe4b543d8c9bcdcb79451779299b653851e818da9ebadaa20c756a4364f94"} Oct 31 13:15:03.061026 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:03.060983 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890875-6rsmb" podStartSLOduration=3.060971172 podStartE2EDuration="3.060971172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:03.059813695 +0000 UTC m=+2553.767399998" watchObservedRunningTime="2026-10-31 13:15:03.060971172 +0000 UTC m=+2553.768557467" Oct 31 13:15:03.186874 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:03.181443 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890875-hnnqk" podStartSLOduration=3.181420036 podStartE2EDuration="3.181420036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:03.090112589 +0000 UTC m=+2553.797698885" watchObservedRunningTime="2026-10-31 13:15:03.181420036 +0000 UTC m=+2553.889006335" Oct 31 13:15:03.268704 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:03.266374 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890875-tc5zf" podStartSLOduration=3.2663566250000002 podStartE2EDuration="3.266356625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:03.143841889 +0000 UTC m=+2553.851428185" watchObservedRunningTime="2026-10-31 13:15:03.266356625 +0000 UTC m=+2553.973942921" Oct 31 13:15:03.286866 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:03.273722 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-cxsps" podStartSLOduration=3.273709007 podStartE2EDuration="3.273709007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:03.190884229 +0000 UTC m=+2553.898470527" watchObservedRunningTime="2026-10-31 13:15:03.273709007 +0000 UTC m=+2553.981295304" Oct 31 13:15:03.372626 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:03.372600 5693 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Oct 31 13:15:03.372791 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:03.372677 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" podUID="92d4bf25-3aad-42dc-989f-88c4d68dd188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Oct 31 13:15:04.039598 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:15:04.039568 5693 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod195c9f96_e37a_46e1_b2c7_8f8a15fff884.slice/crio-conmon-4b10bf72986931426e6eb5a0f41eec90d9033f7b2856a266779183daa159e64d.scope\": RecentStats: unable to find data in memory cache]" Oct 31 13:15:04.055041 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:04.053987 5693 generic.go:356] "Generic (PLEG): container finished" podID="195c9f96-e37a-46e1-b2c7-8f8a15fff884" containerID="4b10bf72986931426e6eb5a0f41eec90d9033f7b2856a266779183daa159e64d" exitCode=60 Oct 31 13:15:04.055041 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:04.054052 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890875-r8b8z" event={"ID":"195c9f96-e37a-46e1-b2c7-8f8a15fff884","Type":"ContainerDied","Data":"4b10bf72986931426e6eb5a0f41eec90d9033f7b2856a266779183daa159e64d"} Oct 31 13:15:04.055041 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:04.054423 5693 scope.go:122] "RemoveContainer" containerID="4b10bf72986931426e6eb5a0f41eec90d9033f7b2856a266779183daa159e64d" Oct 31 13:15:04.061513 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:04.058860 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890875-wdmvp" event={"ID":"74df98d2-73a9-43bb-b0c2-258f69413210","Type":"ContainerStarted","Data":"05cce919830f91a196113cd6d4de344fc671c38390af59459deaa177ee8d32f5"} Oct 31 13:15:04.061513 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:04.060859 5693 generic.go:356] "Generic (PLEG): container finished" podID="22dbab0d-c6ad-4083-86bc-c5e0dfad5db3" containerID="aafbe4b543d8c9bcdcb79451779299b653851e818da9ebadaa20c756a4364f94" exitCode=60 Oct 31 13:15:04.061513 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:04.060974 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890875-tc5zf" event={"ID":"22dbab0d-c6ad-4083-86bc-c5e0dfad5db3","Type":"ContainerDied","Data":"aafbe4b543d8c9bcdcb79451779299b653851e818da9ebadaa20c756a4364f94"} Oct 31 13:15:04.061513 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:04.061235 5693 scope.go:122] "RemoveContainer" containerID="aafbe4b543d8c9bcdcb79451779299b653851e818da9ebadaa20c756a4364f94" Oct 31 13:15:04.063681 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:04.063050 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890875-hvbgx" event={"ID":"cb0c915e-681f-4fb6-bff9-91ae28572928","Type":"ContainerStarted","Data":"6e1b5194649184630f7e68c951d2f9ee31b8b0ce97f8c75cc66f870773209e9a"} Oct 31 13:15:04.069051 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:04.067439 5693 generic.go:356] "Generic (PLEG): container finished" podID="cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31" containerID="f4f876d410509d8e3c63d4e36e5e301dc3c987a8a3e327eafebee2f83de7ffdb" exitCode=60 Oct 31 13:15:04.069051 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:04.067547 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890875-6rsmb" event={"ID":"cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31","Type":"ContainerDied","Data":"f4f876d410509d8e3c63d4e36e5e301dc3c987a8a3e327eafebee2f83de7ffdb"} Oct 31 13:15:04.069051 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:04.068006 5693 scope.go:122] "RemoveContainer" containerID="f4f876d410509d8e3c63d4e36e5e301dc3c987a8a3e327eafebee2f83de7ffdb" Oct 31 13:15:04.072932 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:04.072889 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890875-7jpzj" event={"ID":"eb473765-2907-4d0a-85f9-5f9fa76b77a0","Type":"ContainerStarted","Data":"19b3217f1ca633fcd66a780f1439f2fb864ffa419372b2e094e465a1c9098f00"} Oct 31 13:15:04.075524 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:04.075503 5693 generic.go:356] "Generic (PLEG): container finished" podID="705199d4-0bb0-4d6f-b08e-7cb3d8582e24" containerID="f51e8556e3f240db6e54e24d0e78f6a43d95421520dc76764c78be06d34f166b" exitCode=60 Oct 31 13:15:04.087672 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:04.075741 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890875-hnnqk" event={"ID":"705199d4-0bb0-4d6f-b08e-7cb3d8582e24","Type":"ContainerDied","Data":"f51e8556e3f240db6e54e24d0e78f6a43d95421520dc76764c78be06d34f166b"} Oct 31 13:15:04.087672 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:04.076055 5693 scope.go:122] "RemoveContainer" containerID="f51e8556e3f240db6e54e24d0e78f6a43d95421520dc76764c78be06d34f166b" Oct 31 13:15:04.115110 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:04.114710 5693 generic.go:356] "Generic (PLEG): container finished" podID="a135ae97-5e5e-4d5d-b329-90a8de080e16" containerID="dc322be6814a3bb3d8de49e049857d8db919c33782a5b2b2f205bb4374dcaf00" exitCode=60 Oct 31 13:15:04.115110 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:04.114761 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-cxsps" event={"ID":"a135ae97-5e5e-4d5d-b329-90a8de080e16","Type":"ContainerDied","Data":"dc322be6814a3bb3d8de49e049857d8db919c33782a5b2b2f205bb4374dcaf00"} Oct 31 13:15:04.115405 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:04.115181 5693 scope.go:122] "RemoveContainer" containerID="dc322be6814a3bb3d8de49e049857d8db919c33782a5b2b2f205bb4374dcaf00" Oct 31 13:15:04.166071 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:04.166023 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890875-hvbgx" podStartSLOduration=4.16600823 podStartE2EDuration="4.16600823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:04.143390097 +0000 UTC m=+2554.850976392" watchObservedRunningTime="2026-10-31 13:15:04.16600823 +0000 UTC m=+2554.873594525" Oct 31 13:15:04.215406 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:04.215350 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890875-7jpzj" podStartSLOduration=4.215337621 podStartE2EDuration="4.215337621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:04.210450025 +0000 UTC m=+2554.918036320" watchObservedRunningTime="2026-10-31 13:15:04.215337621 +0000 UTC m=+2554.922923916" Oct 31 13:15:04.372599 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:04.372566 5693 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Oct 31 13:15:04.372793 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:04.372616 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" podUID="92d4bf25-3aad-42dc-989f-88c4d68dd188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Oct 31 13:15:05.125672 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:05.125565 5693 generic.go:356] "Generic (PLEG): container finished" podID="eb473765-2907-4d0a-85f9-5f9fa76b77a0" containerID="19b3217f1ca633fcd66a780f1439f2fb864ffa419372b2e094e465a1c9098f00" exitCode=60 Oct 31 13:15:05.126091 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:05.125660 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890875-7jpzj" event={"ID":"eb473765-2907-4d0a-85f9-5f9fa76b77a0","Type":"ContainerDied","Data":"19b3217f1ca633fcd66a780f1439f2fb864ffa419372b2e094e465a1c9098f00"} Oct 31 13:15:05.126091 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:05.126083 5693 scope.go:122] "RemoveContainer" containerID="19b3217f1ca633fcd66a780f1439f2fb864ffa419372b2e094e465a1c9098f00" Oct 31 13:15:05.128082 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:05.128035 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890875-hnnqk" event={"ID":"705199d4-0bb0-4d6f-b08e-7cb3d8582e24","Type":"ContainerStarted","Data":"b0d18ccc153a9fe3d81748954e52c0678874860c2cf4104ce198d415b4de4a91"} Oct 31 13:15:05.130409 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:05.130360 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-cxsps" event={"ID":"a135ae97-5e5e-4d5d-b329-90a8de080e16","Type":"ContainerStarted","Data":"8d33f94c7f1ba7f8cbef41b770242eca51a50881299f98dbca5f2bb9829dc291"} Oct 31 13:15:05.136348 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:05.133782 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890875-r8b8z" event={"ID":"195c9f96-e37a-46e1-b2c7-8f8a15fff884","Type":"ContainerStarted","Data":"d26c65fbf0ace9c3989a67b5b37492ae075b663906f5eee9834a965027cc199c"} Oct 31 13:15:05.138855 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:05.138829 5693 generic.go:356] "Generic (PLEG): container finished" podID="74df98d2-73a9-43bb-b0c2-258f69413210" containerID="05cce919830f91a196113cd6d4de344fc671c38390af59459deaa177ee8d32f5" exitCode=60 Oct 31 13:15:05.138955 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:05.138866 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890875-wdmvp" event={"ID":"74df98d2-73a9-43bb-b0c2-258f69413210","Type":"ContainerDied","Data":"05cce919830f91a196113cd6d4de344fc671c38390af59459deaa177ee8d32f5"} Oct 31 13:15:05.138955 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:05.138925 5693 scope.go:122] "RemoveContainer" containerID="17d87072287cc45f65d12fc4cfdc48371adb55c9ac79530be3325ca474041201" Oct 31 13:15:05.139181 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:05.139165 5693 scope.go:122] "RemoveContainer" containerID="05cce919830f91a196113cd6d4de344fc671c38390af59459deaa177ee8d32f5" Oct 31 13:15:05.139479 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:15:05.139457 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890875-wdmvp_odh-ai-gateway-infra(74df98d2-73a9-43bb-b0c2-258f69413210)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890875-wdmvp" podUID="74df98d2-73a9-43bb-b0c2-258f69413210" Oct 31 13:15:05.141057 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:05.141027 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890875-tc5zf" event={"ID":"22dbab0d-c6ad-4083-86bc-c5e0dfad5db3","Type":"ContainerStarted","Data":"4b67a92a0fc65dcc29a32d45843edbb765a7fe59fc652778f1b515e8eaa300b9"} Oct 31 13:15:05.142576 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:05.142546 5693 generic.go:356] "Generic (PLEG): container finished" podID="cb0c915e-681f-4fb6-bff9-91ae28572928" containerID="6e1b5194649184630f7e68c951d2f9ee31b8b0ce97f8c75cc66f870773209e9a" exitCode=60 Oct 31 13:15:05.142714 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:05.142616 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890875-hvbgx" event={"ID":"cb0c915e-681f-4fb6-bff9-91ae28572928","Type":"ContainerDied","Data":"6e1b5194649184630f7e68c951d2f9ee31b8b0ce97f8c75cc66f870773209e9a"} Oct 31 13:15:05.142897 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:05.142877 5693 scope.go:122] "RemoveContainer" containerID="6e1b5194649184630f7e68c951d2f9ee31b8b0ce97f8c75cc66f870773209e9a" Oct 31 13:15:05.147594 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:05.147568 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890875-6rsmb" event={"ID":"cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31","Type":"ContainerStarted","Data":"5c43393184d45bea45b18dc763bfc46fbe98160b66a1baa4c43be83a88d5a903"} Oct 31 13:15:05.241474 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:05.241424 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890875-r8b8z" podStartSLOduration=5.2414113780000005 podStartE2EDuration="5.241411378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:05.236240533 +0000 UTC m=+2555.943826828" watchObservedRunningTime="2026-10-31 13:15:05.241411378 +0000 UTC m=+2555.948997673" Oct 31 13:15:05.381100 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:05.380969 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" Oct 31 13:15:05.381829 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:05.381704 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" Oct 31 13:15:05.383189 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:05.383126 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" Oct 31 13:15:06.161654 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:06.161615 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890875-7jpzj" event={"ID":"eb473765-2907-4d0a-85f9-5f9fa76b77a0","Type":"ContainerStarted","Data":"d8eb5108f46fb5b65eff27161b27ed822f830b5a1b84db0de8ac181e37d38355"} Oct 31 13:15:06.164602 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:06.164571 5693 generic.go:356] "Generic (PLEG): container finished" podID="705199d4-0bb0-4d6f-b08e-7cb3d8582e24" containerID="b0d18ccc153a9fe3d81748954e52c0678874860c2cf4104ce198d415b4de4a91" exitCode=60 Oct 31 13:15:06.164754 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:06.164672 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890875-hnnqk" event={"ID":"705199d4-0bb0-4d6f-b08e-7cb3d8582e24","Type":"ContainerDied","Data":"b0d18ccc153a9fe3d81748954e52c0678874860c2cf4104ce198d415b4de4a91"} Oct 31 13:15:06.164754 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:06.164728 5693 scope.go:122] "RemoveContainer" containerID="f51e8556e3f240db6e54e24d0e78f6a43d95421520dc76764c78be06d34f166b" Oct 31 13:15:06.164977 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:06.164941 5693 scope.go:122] "RemoveContainer" containerID="b0d18ccc153a9fe3d81748954e52c0678874860c2cf4104ce198d415b4de4a91" Oct 31 13:15:06.165362 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:15:06.165338 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890875-hnnqk_odh-ai-gateway-infra(705199d4-0bb0-4d6f-b08e-7cb3d8582e24)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890875-hnnqk" podUID="705199d4-0bb0-4d6f-b08e-7cb3d8582e24" Oct 31 13:15:06.167075 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:06.167054 5693 generic.go:356] "Generic (PLEG): container finished" podID="a135ae97-5e5e-4d5d-b329-90a8de080e16" containerID="8d33f94c7f1ba7f8cbef41b770242eca51a50881299f98dbca5f2bb9829dc291" exitCode=60 Oct 31 13:15:06.167186 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:06.167162 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-cxsps" event={"ID":"a135ae97-5e5e-4d5d-b329-90a8de080e16","Type":"ContainerDied","Data":"8d33f94c7f1ba7f8cbef41b770242eca51a50881299f98dbca5f2bb9829dc291"} Oct 31 13:15:06.167393 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:06.167377 5693 scope.go:122] "RemoveContainer" containerID="8d33f94c7f1ba7f8cbef41b770242eca51a50881299f98dbca5f2bb9829dc291" Oct 31 13:15:06.167713 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:15:06.167689 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890875-cxsps_odh-ai-gateway-infra(a135ae97-5e5e-4d5d-b329-90a8de080e16)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-cxsps" podUID="a135ae97-5e5e-4d5d-b329-90a8de080e16" Oct 31 13:15:06.169271 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:06.169248 5693 generic.go:356] "Generic (PLEG): container finished" podID="195c9f96-e37a-46e1-b2c7-8f8a15fff884" containerID="d26c65fbf0ace9c3989a67b5b37492ae075b663906f5eee9834a965027cc199c" exitCode=60 Oct 31 13:15:06.169364 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:06.169318 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890875-r8b8z" event={"ID":"195c9f96-e37a-46e1-b2c7-8f8a15fff884","Type":"ContainerDied","Data":"d26c65fbf0ace9c3989a67b5b37492ae075b663906f5eee9834a965027cc199c"} Oct 31 13:15:06.169564 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:06.169506 5693 scope.go:122] "RemoveContainer" containerID="d26c65fbf0ace9c3989a67b5b37492ae075b663906f5eee9834a965027cc199c" Oct 31 13:15:06.169792 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:15:06.169753 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890875-r8b8z_odh-ai-gateway-infra(195c9f96-e37a-46e1-b2c7-8f8a15fff884)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890875-r8b8z" podUID="195c9f96-e37a-46e1-b2c7-8f8a15fff884" Oct 31 13:15:06.171500 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:06.171479 5693 scope.go:122] "RemoveContainer" containerID="05cce919830f91a196113cd6d4de344fc671c38390af59459deaa177ee8d32f5" Oct 31 13:15:06.171839 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:15:06.171812 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890875-wdmvp_odh-ai-gateway-infra(74df98d2-73a9-43bb-b0c2-258f69413210)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890875-wdmvp" podUID="74df98d2-73a9-43bb-b0c2-258f69413210" Oct 31 13:15:06.173117 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:06.173098 5693 generic.go:356] "Generic (PLEG): container finished" podID="22dbab0d-c6ad-4083-86bc-c5e0dfad5db3" containerID="4b67a92a0fc65dcc29a32d45843edbb765a7fe59fc652778f1b515e8eaa300b9" exitCode=60 Oct 31 13:15:06.173214 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:06.173157 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890875-tc5zf" event={"ID":"22dbab0d-c6ad-4083-86bc-c5e0dfad5db3","Type":"ContainerDied","Data":"4b67a92a0fc65dcc29a32d45843edbb765a7fe59fc652778f1b515e8eaa300b9"} Oct 31 13:15:06.173361 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:06.173343 5693 scope.go:122] "RemoveContainer" containerID="4b67a92a0fc65dcc29a32d45843edbb765a7fe59fc652778f1b515e8eaa300b9" Oct 31 13:15:06.173672 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:15:06.173622 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-107955-29890875-tc5zf_odh-ai-gateway-infra(22dbab0d-c6ad-4083-86bc-c5e0dfad5db3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890875-tc5zf" podUID="22dbab0d-c6ad-4083-86bc-c5e0dfad5db3" Oct 31 13:15:06.175229 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:06.175207 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890875-hvbgx" event={"ID":"cb0c915e-681f-4fb6-bff9-91ae28572928","Type":"ContainerStarted","Data":"0b0f5425d505e17ebb309a3a62484b8608dbce9790c5a84155746ec31dd87763"} Oct 31 13:15:06.177151 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:06.177127 5693 generic.go:356] "Generic (PLEG): container finished" podID="cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31" containerID="5c43393184d45bea45b18dc763bfc46fbe98160b66a1baa4c43be83a88d5a903" exitCode=60 Oct 31 13:15:06.177820 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:06.177788 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890875-6rsmb" event={"ID":"cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31","Type":"ContainerDied","Data":"5c43393184d45bea45b18dc763bfc46fbe98160b66a1baa4c43be83a88d5a903"} Oct 31 13:15:06.177919 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:06.177811 5693 scope.go:122] "RemoveContainer" containerID="5c43393184d45bea45b18dc763bfc46fbe98160b66a1baa4c43be83a88d5a903" Oct 31 13:15:06.178067 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:15:06.178044 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890875-6rsmb_odh-ai-gateway-infra(cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890875-6rsmb" podUID="cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31" Oct 31 13:15:06.184586 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:06.184516 5693 scope.go:122] "RemoveContainer" containerID="dc322be6814a3bb3d8de49e049857d8db919c33782a5b2b2f205bb4374dcaf00" Oct 31 13:15:06.207985 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:06.207967 5693 scope.go:122] "RemoveContainer" containerID="4b10bf72986931426e6eb5a0f41eec90d9033f7b2856a266779183daa159e64d" Oct 31 13:15:06.223381 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:06.223360 5693 scope.go:122] "RemoveContainer" containerID="aafbe4b543d8c9bcdcb79451779299b653851e818da9ebadaa20c756a4364f94" Oct 31 13:15:06.237577 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:06.237545 5693 scope.go:122] "RemoveContainer" containerID="f4f876d410509d8e3c63d4e36e5e301dc3c987a8a3e327eafebee2f83de7ffdb" Oct 31 13:15:07.186994 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:07.186957 5693 generic.go:356] "Generic (PLEG): container finished" podID="eb473765-2907-4d0a-85f9-5f9fa76b77a0" containerID="d8eb5108f46fb5b65eff27161b27ed822f830b5a1b84db0de8ac181e37d38355" exitCode=60 Oct 31 13:15:07.187457 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:07.187041 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890875-7jpzj" event={"ID":"eb473765-2907-4d0a-85f9-5f9fa76b77a0","Type":"ContainerDied","Data":"d8eb5108f46fb5b65eff27161b27ed822f830b5a1b84db0de8ac181e37d38355"} Oct 31 13:15:07.187457 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:07.187141 5693 scope.go:122] "RemoveContainer" containerID="19b3217f1ca633fcd66a780f1439f2fb864ffa419372b2e094e465a1c9098f00" Oct 31 13:15:07.187457 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:07.187367 5693 scope.go:122] "RemoveContainer" containerID="d8eb5108f46fb5b65eff27161b27ed822f830b5a1b84db0de8ac181e37d38355" Oct 31 13:15:07.187888 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:15:07.187859 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-54a16b-29890875-7jpzj_odh-ai-gateway-infra(eb473765-2907-4d0a-85f9-5f9fa76b77a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890875-7jpzj" podUID="eb473765-2907-4d0a-85f9-5f9fa76b77a0" Oct 31 13:15:07.189807 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:07.189786 5693 scope.go:122] "RemoveContainer" containerID="b0d18ccc153a9fe3d81748954e52c0678874860c2cf4104ce198d415b4de4a91" Oct 31 13:15:07.190160 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:15:07.190126 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890875-hnnqk_odh-ai-gateway-infra(705199d4-0bb0-4d6f-b08e-7cb3d8582e24)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890875-hnnqk" podUID="705199d4-0bb0-4d6f-b08e-7cb3d8582e24" Oct 31 13:15:07.191609 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:07.191587 5693 scope.go:122] "RemoveContainer" containerID="8d33f94c7f1ba7f8cbef41b770242eca51a50881299f98dbca5f2bb9829dc291" Oct 31 13:15:07.192103 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:15:07.191957 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890875-cxsps_odh-ai-gateway-infra(a135ae97-5e5e-4d5d-b329-90a8de080e16)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-cxsps" podUID="a135ae97-5e5e-4d5d-b329-90a8de080e16" Oct 31 13:15:07.193794 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:07.193766 5693 scope.go:122] "RemoveContainer" containerID="d26c65fbf0ace9c3989a67b5b37492ae075b663906f5eee9834a965027cc199c" Oct 31 13:15:07.194130 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:15:07.194104 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890875-r8b8z_odh-ai-gateway-infra(195c9f96-e37a-46e1-b2c7-8f8a15fff884)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890875-r8b8z" podUID="195c9f96-e37a-46e1-b2c7-8f8a15fff884" Oct 31 13:15:07.228437 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:07.227855 5693 scope.go:122] "RemoveContainer" containerID="4b67a92a0fc65dcc29a32d45843edbb765a7fe59fc652778f1b515e8eaa300b9" Oct 31 13:15:07.228437 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:15:07.228335 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-107955-29890875-tc5zf_odh-ai-gateway-infra(22dbab0d-c6ad-4083-86bc-c5e0dfad5db3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890875-tc5zf" podUID="22dbab0d-c6ad-4083-86bc-c5e0dfad5db3" Oct 31 13:15:07.231555 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:07.231530 5693 generic.go:356] "Generic (PLEG): container finished" podID="cb0c915e-681f-4fb6-bff9-91ae28572928" containerID="0b0f5425d505e17ebb309a3a62484b8608dbce9790c5a84155746ec31dd87763" exitCode=60 Oct 31 13:15:07.231810 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:07.231781 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890875-hvbgx" event={"ID":"cb0c915e-681f-4fb6-bff9-91ae28572928","Type":"ContainerDied","Data":"0b0f5425d505e17ebb309a3a62484b8608dbce9790c5a84155746ec31dd87763"} Oct 31 13:15:07.231948 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:07.231846 5693 scope.go:122] "RemoveContainer" containerID="6e1b5194649184630f7e68c951d2f9ee31b8b0ce97f8c75cc66f870773209e9a" Oct 31 13:15:07.232291 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:07.232276 5693 scope.go:122] "RemoveContainer" containerID="0b0f5425d505e17ebb309a3a62484b8608dbce9790c5a84155746ec31dd87763" Oct 31 13:15:07.232685 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:15:07.232664 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890875-hvbgx_odh-ai-gateway-infra(cb0c915e-681f-4fb6-bff9-91ae28572928)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890875-hvbgx" podUID="cb0c915e-681f-4fb6-bff9-91ae28572928" Oct 31 13:15:07.236549 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:07.236529 5693 scope.go:122] "RemoveContainer" containerID="5c43393184d45bea45b18dc763bfc46fbe98160b66a1baa4c43be83a88d5a903" Oct 31 13:15:07.236966 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:15:07.236946 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890875-6rsmb_odh-ai-gateway-infra(cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890875-6rsmb" podUID="cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31" Oct 31 13:15:07.705852 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:07.705808 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-fa8c3f-6bc56f9c6c-r4ws4"] Oct 31 13:15:07.734538 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:07.734504 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-fa8c3f-78f95764bc-p8tx5"] Oct 31 13:15:08.264115 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:08.264088 5693 scope.go:122] "RemoveContainer" containerID="0b0f5425d505e17ebb309a3a62484b8608dbce9790c5a84155746ec31dd87763" Oct 31 13:15:08.266166 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:08.266125 5693 scope.go:122] "RemoveContainer" containerID="d8eb5108f46fb5b65eff27161b27ed822f830b5a1b84db0de8ac181e37d38355" Oct 31 13:15:08.266166 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:15:08.266149 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890875-hvbgx_odh-ai-gateway-infra(cb0c915e-681f-4fb6-bff9-91ae28572928)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890875-hvbgx" podUID="cb0c915e-681f-4fb6-bff9-91ae28572928" Oct 31 13:15:08.266579 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:15:08.266515 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-54a16b-29890875-7jpzj_odh-ai-gateway-infra(eb473765-2907-4d0a-85f9-5f9fa76b77a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890875-7jpzj" podUID="eb473765-2907-4d0a-85f9-5f9fa76b77a0" Oct 31 13:15:12.706948 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:12.706908 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-fa8c3f-6bc56f9c6c-r4ws4" podUID="46d567ad-1dde-4d69-9ed1-8a2cab56e2db" containerName="payload-processing" containerID="cri-o://69eddbaf46faf46106eeac3aa5e0c3850c8812f14a09cacdace290bfe2f8e9d6" gracePeriod=25 Oct 31 13:15:12.735411 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:12.735378 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-fa8c3f-78f95764bc-p8tx5" podUID="9264e5f3-1234-47de-97f9-020dfb0b03f1" containerName="payload-pre-processing" containerID="cri-o://75fb75e159216359bba5cd7733a4c01140f24da8c222dcbfcbc389d735eb12ef" gracePeriod=25 Oct 31 13:15:13.114185 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:13.114148 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-fa8c3f-6bc56f9c6c-r4ws4" Oct 31 13:15:13.143498 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:13.143473 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-fa8c3f-78f95764bc-p8tx5" Oct 31 13:15:13.227801 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:13.227761 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46d567ad-1dde-4d69-9ed1-8a2cab56e2db-kube-api-access-77b65\" (UniqueName: \"kubernetes.io/projected/46d567ad-1dde-4d69-9ed1-8a2cab56e2db-kube-api-access-77b65\") pod \"46d567ad-1dde-4d69-9ed1-8a2cab56e2db\" (UID: \"46d567ad-1dde-4d69-9ed1-8a2cab56e2db\") " Oct 31 13:15:13.227990 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:13.227969 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46d567ad-1dde-4d69-9ed1-8a2cab56e2db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46d567ad-1dde-4d69-9ed1-8a2cab56e2db-plugins-config-volume\") pod \"46d567ad-1dde-4d69-9ed1-8a2cab56e2db\" (UID: \"46d567ad-1dde-4d69-9ed1-8a2cab56e2db\") " Oct 31 13:15:13.228064 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:13.228051 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9264e5f3-1234-47de-97f9-020dfb0b03f1-kube-api-access-mzrlt\" (UniqueName: \"kubernetes.io/projected/9264e5f3-1234-47de-97f9-020dfb0b03f1-kube-api-access-mzrlt\") pod \"9264e5f3-1234-47de-97f9-020dfb0b03f1\" (UID: \"9264e5f3-1234-47de-97f9-020dfb0b03f1\") " Oct 31 13:15:13.228164 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:13.228149 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9264e5f3-1234-47de-97f9-020dfb0b03f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9264e5f3-1234-47de-97f9-020dfb0b03f1-plugins-config-volume\") pod \"9264e5f3-1234-47de-97f9-020dfb0b03f1\" (UID: \"9264e5f3-1234-47de-97f9-020dfb0b03f1\") " Oct 31 13:15:13.228489 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:13.228464 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46d567ad-1dde-4d69-9ed1-8a2cab56e2db-plugins-config-volume" pod "46d567ad-1dde-4d69-9ed1-8a2cab56e2db" (UID: "46d567ad-1dde-4d69-9ed1-8a2cab56e2db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:13.228574 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:13.228525 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9264e5f3-1234-47de-97f9-020dfb0b03f1-plugins-config-volume" pod "9264e5f3-1234-47de-97f9-020dfb0b03f1" (UID: "9264e5f3-1234-47de-97f9-020dfb0b03f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:13.228957 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:13.228935 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46d567ad-1dde-4d69-9ed1-8a2cab56e2db-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:13.228957 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:13.228955 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9264e5f3-1234-47de-97f9-020dfb0b03f1-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:13.230133 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:13.230111 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d567ad-1dde-4d69-9ed1-8a2cab56e2db-kube-api-access-77b65" pod "46d567ad-1dde-4d69-9ed1-8a2cab56e2db" (UID: "46d567ad-1dde-4d69-9ed1-8a2cab56e2db"). InnerVolumeSpecName "kube-api-access-77b65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:13.230329 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:13.230310 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9264e5f3-1234-47de-97f9-020dfb0b03f1-kube-api-access-mzrlt" pod "9264e5f3-1234-47de-97f9-020dfb0b03f1" (UID: "9264e5f3-1234-47de-97f9-020dfb0b03f1"). InnerVolumeSpecName "kube-api-access-mzrlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:13.322735 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:13.322312 5693 generic.go:356] "Generic (PLEG): container finished" podID="9264e5f3-1234-47de-97f9-020dfb0b03f1" containerID="75fb75e159216359bba5cd7733a4c01140f24da8c222dcbfcbc389d735eb12ef" exitCode=0 Oct 31 13:15:13.322735 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:13.322357 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-fa8c3f-78f95764bc-p8tx5" event={"ID":"9264e5f3-1234-47de-97f9-020dfb0b03f1","Type":"ContainerDied","Data":"75fb75e159216359bba5cd7733a4c01140f24da8c222dcbfcbc389d735eb12ef"} Oct 31 13:15:13.322735 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:13.322431 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-fa8c3f-78f95764bc-p8tx5" event={"ID":"9264e5f3-1234-47de-97f9-020dfb0b03f1","Type":"ContainerDied","Data":"43e64baecd174305d98340cd15f65dce59765cab4d7c740d710b04aed22055ec"} Oct 31 13:15:13.322735 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:13.322368 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-fa8c3f-78f95764bc-p8tx5" Oct 31 13:15:13.322735 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:13.322463 5693 scope.go:122] "RemoveContainer" containerID="75fb75e159216359bba5cd7733a4c01140f24da8c222dcbfcbc389d735eb12ef" Oct 31 13:15:13.324229 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:13.324205 5693 generic.go:356] "Generic (PLEG): container finished" podID="46d567ad-1dde-4d69-9ed1-8a2cab56e2db" containerID="69eddbaf46faf46106eeac3aa5e0c3850c8812f14a09cacdace290bfe2f8e9d6" exitCode=0 Oct 31 13:15:13.324354 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:13.324259 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-fa8c3f-6bc56f9c6c-r4ws4" Oct 31 13:15:13.324354 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:13.324311 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-fa8c3f-6bc56f9c6c-r4ws4" event={"ID":"46d567ad-1dde-4d69-9ed1-8a2cab56e2db","Type":"ContainerDied","Data":"69eddbaf46faf46106eeac3aa5e0c3850c8812f14a09cacdace290bfe2f8e9d6"} Oct 31 13:15:13.324452 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:13.324356 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-fa8c3f-6bc56f9c6c-r4ws4" event={"ID":"46d567ad-1dde-4d69-9ed1-8a2cab56e2db","Type":"ContainerDied","Data":"4c424d822d3faab13c825a33a8d50d955e87963ff90022e686e86efe68f39730"} Oct 31 13:15:13.335156 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:13.333905 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzrlt\" (UniqueName: \"kubernetes.io/projected/9264e5f3-1234-47de-97f9-020dfb0b03f1-kube-api-access-mzrlt\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:13.335156 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:13.333931 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77b65\" (UniqueName: \"kubernetes.io/projected/46d567ad-1dde-4d69-9ed1-8a2cab56e2db-kube-api-access-77b65\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:13.335499 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:13.335422 5693 scope.go:122] "RemoveContainer" containerID="75fb75e159216359bba5cd7733a4c01140f24da8c222dcbfcbc389d735eb12ef" Oct 31 13:15:13.335809 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:15:13.335786 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75fb75e159216359bba5cd7733a4c01140f24da8c222dcbfcbc389d735eb12ef\": container with ID starting with 75fb75e159216359bba5cd7733a4c01140f24da8c222dcbfcbc389d735eb12ef not found: ID does not exist" containerID="75fb75e159216359bba5cd7733a4c01140f24da8c222dcbfcbc389d735eb12ef" Oct 31 13:15:13.335915 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:13.335813 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75fb75e159216359bba5cd7733a4c01140f24da8c222dcbfcbc389d735eb12ef"} err="failed to get container status \"75fb75e159216359bba5cd7733a4c01140f24da8c222dcbfcbc389d735eb12ef\": rpc error: code = NotFound desc = could not find container \"75fb75e159216359bba5cd7733a4c01140f24da8c222dcbfcbc389d735eb12ef\": container with ID starting with 75fb75e159216359bba5cd7733a4c01140f24da8c222dcbfcbc389d735eb12ef not found: ID does not exist" Oct 31 13:15:13.335915 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:13.335831 5693 scope.go:122] "RemoveContainer" containerID="69eddbaf46faf46106eeac3aa5e0c3850c8812f14a09cacdace290bfe2f8e9d6" Oct 31 13:15:13.346791 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:13.346773 5693 scope.go:122] "RemoveContainer" containerID="69eddbaf46faf46106eeac3aa5e0c3850c8812f14a09cacdace290bfe2f8e9d6" Oct 31 13:15:13.347049 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:15:13.347032 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69eddbaf46faf46106eeac3aa5e0c3850c8812f14a09cacdace290bfe2f8e9d6\": container with ID starting with 69eddbaf46faf46106eeac3aa5e0c3850c8812f14a09cacdace290bfe2f8e9d6 not found: ID does not exist" containerID="69eddbaf46faf46106eeac3aa5e0c3850c8812f14a09cacdace290bfe2f8e9d6" Oct 31 13:15:13.347129 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:13.347056 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69eddbaf46faf46106eeac3aa5e0c3850c8812f14a09cacdace290bfe2f8e9d6"} err="failed to get container status \"69eddbaf46faf46106eeac3aa5e0c3850c8812f14a09cacdace290bfe2f8e9d6\": rpc error: code = NotFound desc = could not find container \"69eddbaf46faf46106eeac3aa5e0c3850c8812f14a09cacdace290bfe2f8e9d6\": container with ID starting with 69eddbaf46faf46106eeac3aa5e0c3850c8812f14a09cacdace290bfe2f8e9d6 not found: ID does not exist" Oct 31 13:15:13.354620 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:13.354595 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-fa8c3f-78f95764bc-p8tx5"] Oct 31 13:15:13.358941 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:13.358921 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-fa8c3f-78f95764bc-p8tx5"] Oct 31 13:15:13.369167 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:13.369144 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-fa8c3f-6bc56f9c6c-r4ws4"] Oct 31 13:15:13.372778 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:13.372758 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-fa8c3f-6bc56f9c6c-r4ws4"] Oct 31 13:15:13.410345 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:13.410312 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46d567ad-1dde-4d69-9ed1-8a2cab56e2db" path="/var/lib/kubelet/pods/46d567ad-1dde-4d69-9ed1-8a2cab56e2db/volumes" Oct 31 13:15:13.422266 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:13.411593 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9264e5f3-1234-47de-97f9-020dfb0b03f1" path="/var/lib/kubelet/pods/9264e5f3-1234-47de-97f9-020dfb0b03f1/volumes" Oct 31 13:15:16.401687 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:16.401621 5693 scope.go:122] "RemoveContainer" containerID="b0d18ccc153a9fe3d81748954e52c0678874860c2cf4104ce198d415b4de4a91" Oct 31 13:15:17.364191 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:17.364099 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890875-hnnqk" event={"ID":"705199d4-0bb0-4d6f-b08e-7cb3d8582e24","Type":"ContainerStarted","Data":"02a4156519ec6a679ee62bba65eadbe80bd9e6ee271ee81187ebff073ba65a6c"} Oct 31 13:15:17.402055 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:17.402029 5693 scope.go:122] "RemoveContainer" containerID="5c43393184d45bea45b18dc763bfc46fbe98160b66a1baa4c43be83a88d5a903" Oct 31 13:15:17.402484 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:17.402179 5693 scope.go:122] "RemoveContainer" containerID="05cce919830f91a196113cd6d4de344fc671c38390af59459deaa177ee8d32f5" Oct 31 13:15:17.402484 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:17.402317 5693 scope.go:122] "RemoveContainer" containerID="8d33f94c7f1ba7f8cbef41b770242eca51a50881299f98dbca5f2bb9829dc291" Oct 31 13:15:18.375073 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:18.375035 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890875-6rsmb" event={"ID":"cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31","Type":"ContainerStarted","Data":"793158814fbbd12647fd2f83ad853268944e995034201552ca134afaa4596b17"} Oct 31 13:15:18.377435 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:18.377408 5693 generic.go:356] "Generic (PLEG): container finished" podID="705199d4-0bb0-4d6f-b08e-7cb3d8582e24" containerID="02a4156519ec6a679ee62bba65eadbe80bd9e6ee271ee81187ebff073ba65a6c" exitCode=60 Oct 31 13:15:18.377551 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:18.377479 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890875-hnnqk" event={"ID":"705199d4-0bb0-4d6f-b08e-7cb3d8582e24","Type":"ContainerDied","Data":"02a4156519ec6a679ee62bba65eadbe80bd9e6ee271ee81187ebff073ba65a6c"} Oct 31 13:15:18.377551 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:18.377536 5693 scope.go:122] "RemoveContainer" containerID="b0d18ccc153a9fe3d81748954e52c0678874860c2cf4104ce198d415b4de4a91" Oct 31 13:15:18.378376 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:18.377914 5693 scope.go:122] "RemoveContainer" containerID="02a4156519ec6a679ee62bba65eadbe80bd9e6ee271ee81187ebff073ba65a6c" Oct 31 13:15:18.378376 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:15:18.378291 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890875-hnnqk_odh-ai-gateway-infra(705199d4-0bb0-4d6f-b08e-7cb3d8582e24)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890875-hnnqk" podUID="705199d4-0bb0-4d6f-b08e-7cb3d8582e24" Oct 31 13:15:18.387200 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:18.387174 5693 generic.go:356] "Generic (PLEG): container finished" podID="a135ae97-5e5e-4d5d-b329-90a8de080e16" containerID="64c8cb4d0d0722e7c895dd2c7f1103b839b740925fa5948847ebf8f4ed232389" exitCode=60 Oct 31 13:15:18.387313 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:18.387250 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-cxsps" event={"ID":"a135ae97-5e5e-4d5d-b329-90a8de080e16","Type":"ContainerDied","Data":"64c8cb4d0d0722e7c895dd2c7f1103b839b740925fa5948847ebf8f4ed232389"} Oct 31 13:15:18.387689 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:18.387672 5693 scope.go:122] "RemoveContainer" containerID="64c8cb4d0d0722e7c895dd2c7f1103b839b740925fa5948847ebf8f4ed232389" Oct 31 13:15:18.388058 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:15:18.388037 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890875-cxsps_odh-ai-gateway-infra(a135ae97-5e5e-4d5d-b329-90a8de080e16)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-cxsps" podUID="a135ae97-5e5e-4d5d-b329-90a8de080e16" Oct 31 13:15:18.389613 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:18.389594 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890875-wdmvp" event={"ID":"74df98d2-73a9-43bb-b0c2-258f69413210","Type":"ContainerStarted","Data":"26cf450fb36796ee8741a1827b417bb67e30ea97b8f9c61c21f4b2d2bdab2540"} Oct 31 13:15:18.401830 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:18.401810 5693 scope.go:122] "RemoveContainer" containerID="4b67a92a0fc65dcc29a32d45843edbb765a7fe59fc652778f1b515e8eaa300b9" Oct 31 13:15:18.418401 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:18.418372 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890875-hnnqk"] Oct 31 13:15:18.421140 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:18.421120 5693 scope.go:122] "RemoveContainer" containerID="8d33f94c7f1ba7f8cbef41b770242eca51a50881299f98dbca5f2bb9829dc291" Oct 31 13:15:18.690776 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:15:18.690742 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:15:19.400997 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:19.400964 5693 generic.go:356] "Generic (PLEG): container finished" podID="22dbab0d-c6ad-4083-86bc-c5e0dfad5db3" containerID="49773b1eb803c748466b7965025d9d026635aabb7293892cc12962016b59c40c" exitCode=60 Oct 31 13:15:19.403496 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:19.403472 5693 generic.go:356] "Generic (PLEG): container finished" podID="cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31" containerID="793158814fbbd12647fd2f83ad853268944e995034201552ca134afaa4596b17" exitCode=60 Oct 31 13:15:19.428679 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:19.420060 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890875-tc5zf" event={"ID":"22dbab0d-c6ad-4083-86bc-c5e0dfad5db3","Type":"ContainerDied","Data":"49773b1eb803c748466b7965025d9d026635aabb7293892cc12962016b59c40c"} Oct 31 13:15:19.428679 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:19.420112 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890875-6rsmb" event={"ID":"cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31","Type":"ContainerDied","Data":"793158814fbbd12647fd2f83ad853268944e995034201552ca134afaa4596b17"} Oct 31 13:15:19.428679 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:19.420661 5693 scope.go:122] "RemoveContainer" containerID="793158814fbbd12647fd2f83ad853268944e995034201552ca134afaa4596b17" Oct 31 13:15:19.428679 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:15:19.421108 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890875-6rsmb_odh-ai-gateway-infra(cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890875-6rsmb" podUID="cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31" Oct 31 13:15:19.428679 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:19.421963 5693 scope.go:122] "RemoveContainer" containerID="d26c65fbf0ace9c3989a67b5b37492ae075b663906f5eee9834a965027cc199c" Oct 31 13:15:19.428679 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:19.422534 5693 scope.go:122] "RemoveContainer" containerID="4b67a92a0fc65dcc29a32d45843edbb765a7fe59fc652778f1b515e8eaa300b9" Oct 31 13:15:19.451177 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:19.451145 5693 scope.go:122] "RemoveContainer" containerID="49773b1eb803c748466b7965025d9d026635aabb7293892cc12962016b59c40c" Oct 31 13:15:19.472594 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:19.472551 5693 generic.go:356] "Generic (PLEG): container finished" podID="74df98d2-73a9-43bb-b0c2-258f69413210" containerID="26cf450fb36796ee8741a1827b417bb67e30ea97b8f9c61c21f4b2d2bdab2540" exitCode=60 Oct 31 13:15:19.472842 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:19.472657 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890875-wdmvp" event={"ID":"74df98d2-73a9-43bb-b0c2-258f69413210","Type":"ContainerDied","Data":"26cf450fb36796ee8741a1827b417bb67e30ea97b8f9c61c21f4b2d2bdab2540"} Oct 31 13:15:19.475840 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:19.475025 5693 scope.go:122] "RemoveContainer" containerID="26cf450fb36796ee8741a1827b417bb67e30ea97b8f9c61c21f4b2d2bdab2540" Oct 31 13:15:19.475840 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:15:19.475389 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-107955-29890875-tc5zf_odh-ai-gateway-infra(22dbab0d-c6ad-4083-86bc-c5e0dfad5db3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890875-tc5zf" podUID="22dbab0d-c6ad-4083-86bc-c5e0dfad5db3" Oct 31 13:15:19.475840 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:15:19.475395 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890875-wdmvp_odh-ai-gateway-infra(74df98d2-73a9-43bb-b0c2-258f69413210)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890875-wdmvp" podUID="74df98d2-73a9-43bb-b0c2-258f69413210" Oct 31 13:15:19.490674 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:19.490616 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890875-6rsmb"] Oct 31 13:15:19.509084 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:19.508658 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890875-cxsps"] Oct 31 13:15:19.568104 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:19.568074 5693 scope.go:122] "RemoveContainer" containerID="5c43393184d45bea45b18dc763bfc46fbe98160b66a1baa4c43be83a88d5a903" Oct 31 13:15:19.568553 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:19.568324 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890875-wdmvp"] Oct 31 13:15:19.596031 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:19.595920 5693 scope.go:122] "RemoveContainer" containerID="05cce919830f91a196113cd6d4de344fc671c38390af59459deaa177ee8d32f5" Oct 31 13:15:19.807105 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:19.806796 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890875-hnnqk" Oct 31 13:15:19.813069 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:19.813046 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-cxsps" Oct 31 13:15:19.891729 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:19.891626 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a135ae97-5e5e-4d5d-b329-90a8de080e16-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a135ae97-5e5e-4d5d-b329-90a8de080e16-maas-api-service-ca\") pod \"a135ae97-5e5e-4d5d-b329-90a8de080e16\" (UID: \"a135ae97-5e5e-4d5d-b329-90a8de080e16\") " Oct 31 13:15:19.891900 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:19.891734 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a135ae97-5e5e-4d5d-b329-90a8de080e16-kube-api-access-2qmfs\" (UniqueName: \"kubernetes.io/projected/a135ae97-5e5e-4d5d-b329-90a8de080e16-kube-api-access-2qmfs\") pod \"a135ae97-5e5e-4d5d-b329-90a8de080e16\" (UID: \"a135ae97-5e5e-4d5d-b329-90a8de080e16\") " Oct 31 13:15:19.892005 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:19.891984 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/705199d4-0bb0-4d6f-b08e-7cb3d8582e24-kube-api-access-krcq6\" (UniqueName: \"kubernetes.io/projected/705199d4-0bb0-4d6f-b08e-7cb3d8582e24-kube-api-access-krcq6\") pod \"705199d4-0bb0-4d6f-b08e-7cb3d8582e24\" (UID: \"705199d4-0bb0-4d6f-b08e-7cb3d8582e24\") " Oct 31 13:15:19.892076 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:19.892055 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a135ae97-5e5e-4d5d-b329-90a8de080e16-maas-api-service-ca" pod "a135ae97-5e5e-4d5d-b329-90a8de080e16" (UID: "a135ae97-5e5e-4d5d-b329-90a8de080e16"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:19.892139 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:19.892075 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/705199d4-0bb0-4d6f-b08e-7cb3d8582e24-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/705199d4-0bb0-4d6f-b08e-7cb3d8582e24-maas-api-service-ca\") pod \"705199d4-0bb0-4d6f-b08e-7cb3d8582e24\" (UID: \"705199d4-0bb0-4d6f-b08e-7cb3d8582e24\") " Oct 31 13:15:19.892785 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:19.892652 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/705199d4-0bb0-4d6f-b08e-7cb3d8582e24-maas-api-service-ca" pod "705199d4-0bb0-4d6f-b08e-7cb3d8582e24" (UID: "705199d4-0bb0-4d6f-b08e-7cb3d8582e24"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:19.892928 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:19.892824 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/705199d4-0bb0-4d6f-b08e-7cb3d8582e24-maas-api-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:19.892928 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:19.892846 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a135ae97-5e5e-4d5d-b329-90a8de080e16-maas-api-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:19.894853 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:19.894825 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/705199d4-0bb0-4d6f-b08e-7cb3d8582e24-kube-api-access-krcq6" pod "705199d4-0bb0-4d6f-b08e-7cb3d8582e24" (UID: "705199d4-0bb0-4d6f-b08e-7cb3d8582e24"). InnerVolumeSpecName "kube-api-access-krcq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:19.895161 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:19.895140 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a135ae97-5e5e-4d5d-b329-90a8de080e16-kube-api-access-2qmfs" pod "a135ae97-5e5e-4d5d-b329-90a8de080e16" (UID: "a135ae97-5e5e-4d5d-b329-90a8de080e16"). InnerVolumeSpecName "kube-api-access-2qmfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:20.014183 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:20.012190 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qmfs\" (UniqueName: \"kubernetes.io/projected/a135ae97-5e5e-4d5d-b329-90a8de080e16-kube-api-access-2qmfs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:20.014183 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:20.012230 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krcq6\" (UniqueName: \"kubernetes.io/projected/705199d4-0bb0-4d6f-b08e-7cb3d8582e24-kube-api-access-krcq6\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:20.401980 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:20.401939 5693 scope.go:122] "RemoveContainer" containerID="0b0f5425d505e17ebb309a3a62484b8608dbce9790c5a84155746ec31dd87763" Oct 31 13:15:20.402280 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:20.402257 5693 scope.go:122] "RemoveContainer" containerID="d8eb5108f46fb5b65eff27161b27ed822f830b5a1b84db0de8ac181e37d38355" Oct 31 13:15:20.554251 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:20.554220 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-cxsps" Oct 31 13:15:20.554251 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:20.554233 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890875-cxsps" event={"ID":"a135ae97-5e5e-4d5d-b329-90a8de080e16","Type":"ContainerDied","Data":"a49d8da0154627f0fc77dfc2a89e03c670ba76e738937da50aa26acaae7eea60"} Oct 31 13:15:20.554923 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:20.554299 5693 scope.go:122] "RemoveContainer" containerID="64c8cb4d0d0722e7c895dd2c7f1103b839b740925fa5948847ebf8f4ed232389" Oct 31 13:15:20.556571 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:20.556545 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890875-r8b8z" event={"ID":"195c9f96-e37a-46e1-b2c7-8f8a15fff884","Type":"ContainerStarted","Data":"5e07b7dc1a74bbbb01508d527b57f6d59f0c598eb455a6919a7b136cfa0a6c8c"} Oct 31 13:15:20.563819 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:20.563799 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890875-hnnqk" event={"ID":"705199d4-0bb0-4d6f-b08e-7cb3d8582e24","Type":"ContainerDied","Data":"122c8b418470d2fc51098dfcdb704806c24575bbb8dbdd197394bdb9fb0ebf41"} Oct 31 13:15:20.563891 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:20.563845 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890875-hnnqk" Oct 31 13:15:20.599407 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:20.599371 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890875-tc5zf"] Oct 31 13:15:20.614949 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:20.614920 5693 scope.go:122] "RemoveContainer" containerID="02a4156519ec6a679ee62bba65eadbe80bd9e6ee271ee81187ebff073ba65a6c" Oct 31 13:15:20.699375 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:20.697220 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890875-cxsps"] Oct 31 13:15:20.704259 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:20.704230 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890875-cxsps"] Oct 31 13:15:20.722267 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:20.721850 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890875-hnnqk"] Oct 31 13:15:20.730679 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:20.730599 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890875-hnnqk"] Oct 31 13:15:20.905257 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:20.905227 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890875-wdmvp" Oct 31 13:15:20.911657 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:20.911586 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890875-6rsmb" Oct 31 13:15:20.925434 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:20.925412 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890875-tc5zf" Oct 31 13:15:20.938833 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:20.938803 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22dbab0d-c6ad-4083-86bc-c5e0dfad5db3-kube-api-access-4whr6\" (UniqueName: \"kubernetes.io/projected/22dbab0d-c6ad-4083-86bc-c5e0dfad5db3-kube-api-access-4whr6\") pod \"22dbab0d-c6ad-4083-86bc-c5e0dfad5db3\" (UID: \"22dbab0d-c6ad-4083-86bc-c5e0dfad5db3\") " Oct 31 13:15:20.939266 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:20.938942 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74df98d2-73a9-43bb-b0c2-258f69413210-kube-api-access-lnmcg\" (UniqueName: \"kubernetes.io/projected/74df98d2-73a9-43bb-b0c2-258f69413210-kube-api-access-lnmcg\") pod \"74df98d2-73a9-43bb-b0c2-258f69413210\" (UID: \"74df98d2-73a9-43bb-b0c2-258f69413210\") " Oct 31 13:15:20.939266 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:20.938989 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31-maas-api-service-ca\") pod \"cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31\" (UID: \"cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31\") " Oct 31 13:15:20.939266 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:20.939039 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74df98d2-73a9-43bb-b0c2-258f69413210-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74df98d2-73a9-43bb-b0c2-258f69413210-maas-api-service-ca\") pod \"74df98d2-73a9-43bb-b0c2-258f69413210\" (UID: \"74df98d2-73a9-43bb-b0c2-258f69413210\") " Oct 31 13:15:20.940117 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:20.939520 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31-kube-api-access-q87k8\" (UniqueName: \"kubernetes.io/projected/cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31-kube-api-access-q87k8\") pod \"cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31\" (UID: \"cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31\") " Oct 31 13:15:20.940117 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:20.939745 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22dbab0d-c6ad-4083-86bc-c5e0dfad5db3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22dbab0d-c6ad-4083-86bc-c5e0dfad5db3-maas-api-service-ca\") pod \"22dbab0d-c6ad-4083-86bc-c5e0dfad5db3\" (UID: \"22dbab0d-c6ad-4083-86bc-c5e0dfad5db3\") " Oct 31 13:15:20.940117 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:20.939887 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74df98d2-73a9-43bb-b0c2-258f69413210-maas-api-service-ca" pod "74df98d2-73a9-43bb-b0c2-258f69413210" (UID: "74df98d2-73a9-43bb-b0c2-258f69413210"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:20.940117 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:20.939964 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31-maas-api-service-ca" pod "cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31" (UID: "cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:20.940352 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:20.940208 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22dbab0d-c6ad-4083-86bc-c5e0dfad5db3-maas-api-service-ca" pod "22dbab0d-c6ad-4083-86bc-c5e0dfad5db3" (UID: "22dbab0d-c6ad-4083-86bc-c5e0dfad5db3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:20.940462 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:20.940439 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31-maas-api-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:20.940462 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:20.940464 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74df98d2-73a9-43bb-b0c2-258f69413210-maas-api-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:20.940695 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:20.940477 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22dbab0d-c6ad-4083-86bc-c5e0dfad5db3-maas-api-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:20.942522 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:20.942502 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74df98d2-73a9-43bb-b0c2-258f69413210-kube-api-access-lnmcg" pod "74df98d2-73a9-43bb-b0c2-258f69413210" (UID: "74df98d2-73a9-43bb-b0c2-258f69413210"). InnerVolumeSpecName "kube-api-access-lnmcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:20.942712 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:20.942688 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22dbab0d-c6ad-4083-86bc-c5e0dfad5db3-kube-api-access-4whr6" pod "22dbab0d-c6ad-4083-86bc-c5e0dfad5db3" (UID: "22dbab0d-c6ad-4083-86bc-c5e0dfad5db3"). InnerVolumeSpecName "kube-api-access-4whr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:20.942792 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:20.942725 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31-kube-api-access-q87k8" pod "cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31" (UID: "cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31"). InnerVolumeSpecName "kube-api-access-q87k8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:21.042301 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:21.042268 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4whr6\" (UniqueName: \"kubernetes.io/projected/22dbab0d-c6ad-4083-86bc-c5e0dfad5db3-kube-api-access-4whr6\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:21.042301 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:21.042299 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnmcg\" (UniqueName: \"kubernetes.io/projected/74df98d2-73a9-43bb-b0c2-258f69413210-kube-api-access-lnmcg\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:21.042301 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:21.042309 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q87k8\" (UniqueName: \"kubernetes.io/projected/cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31-kube-api-access-q87k8\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:21.411318 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:21.411284 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="705199d4-0bb0-4d6f-b08e-7cb3d8582e24" path="/var/lib/kubelet/pods/705199d4-0bb0-4d6f-b08e-7cb3d8582e24/volumes" Oct 31 13:15:21.411831 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:21.411815 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a135ae97-5e5e-4d5d-b329-90a8de080e16" path="/var/lib/kubelet/pods/a135ae97-5e5e-4d5d-b329-90a8de080e16/volumes" Oct 31 13:15:21.573224 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:21.573192 5693 generic.go:356] "Generic (PLEG): container finished" podID="195c9f96-e37a-46e1-b2c7-8f8a15fff884" containerID="5e07b7dc1a74bbbb01508d527b57f6d59f0c598eb455a6919a7b136cfa0a6c8c" exitCode=60 Oct 31 13:15:21.573635 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:21.573262 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890875-r8b8z" event={"ID":"195c9f96-e37a-46e1-b2c7-8f8a15fff884","Type":"ContainerDied","Data":"5e07b7dc1a74bbbb01508d527b57f6d59f0c598eb455a6919a7b136cfa0a6c8c"} Oct 31 13:15:21.573635 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:21.573322 5693 scope.go:122] "RemoveContainer" containerID="d26c65fbf0ace9c3989a67b5b37492ae075b663906f5eee9834a965027cc199c" Oct 31 13:15:21.573797 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:21.573780 5693 scope.go:122] "RemoveContainer" containerID="5e07b7dc1a74bbbb01508d527b57f6d59f0c598eb455a6919a7b136cfa0a6c8c" Oct 31 13:15:21.574165 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:15:21.574140 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890875-r8b8z_odh-ai-gateway-infra(195c9f96-e37a-46e1-b2c7-8f8a15fff884)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890875-r8b8z" podUID="195c9f96-e37a-46e1-b2c7-8f8a15fff884" Oct 31 13:15:21.574824 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:21.574806 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890875-wdmvp" Oct 31 13:15:21.574931 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:21.574839 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890875-wdmvp" event={"ID":"74df98d2-73a9-43bb-b0c2-258f69413210","Type":"ContainerDied","Data":"072d24d284052cc9e078ef2d8d8d887b5c8ecbffb663ea4d5cb3d5ce27de4fe0"} Oct 31 13:15:21.576338 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:21.576316 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890875-tc5zf" Oct 31 13:15:21.576440 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:21.576348 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890875-tc5zf" event={"ID":"22dbab0d-c6ad-4083-86bc-c5e0dfad5db3","Type":"ContainerDied","Data":"208f448f84e2996bb6e634b882bca236908375ef1b39074909419e75ab36c797"} Oct 31 13:15:21.578504 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:21.578479 5693 generic.go:356] "Generic (PLEG): container finished" podID="cb0c915e-681f-4fb6-bff9-91ae28572928" containerID="32e28883869d668e6f3ca876d3b2154796dc895ad8e9649d612e542152695949" exitCode=60 Oct 31 13:15:21.578593 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:21.578548 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890875-hvbgx" event={"ID":"cb0c915e-681f-4fb6-bff9-91ae28572928","Type":"ContainerDied","Data":"32e28883869d668e6f3ca876d3b2154796dc895ad8e9649d612e542152695949"} Oct 31 13:15:21.579001 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:21.578985 5693 scope.go:122] "RemoveContainer" containerID="32e28883869d668e6f3ca876d3b2154796dc895ad8e9649d612e542152695949" Oct 31 13:15:21.579281 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:15:21.579261 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890875-hvbgx_odh-ai-gateway-infra(cb0c915e-681f-4fb6-bff9-91ae28572928)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890875-hvbgx" podUID="cb0c915e-681f-4fb6-bff9-91ae28572928" Oct 31 13:15:21.580074 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:21.580060 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890875-6rsmb" Oct 31 13:15:21.580162 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:21.580083 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890875-6rsmb" event={"ID":"cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31","Type":"ContainerDied","Data":"31c1470cbf38e362eb29b7c1d222bafe4f64043ad1af2ccdbedac7da89644f90"} Oct 31 13:15:21.582002 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:21.581983 5693 generic.go:356] "Generic (PLEG): container finished" podID="eb473765-2907-4d0a-85f9-5f9fa76b77a0" containerID="d37b895467422ac33c93619db11018ba8c16ed05731868f0b5e6ad58b6aba746" exitCode=60 Oct 31 13:15:21.582078 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:21.582002 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890875-7jpzj" event={"ID":"eb473765-2907-4d0a-85f9-5f9fa76b77a0","Type":"ContainerDied","Data":"d37b895467422ac33c93619db11018ba8c16ed05731868f0b5e6ad58b6aba746"} Oct 31 13:15:21.582374 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:21.582360 5693 scope.go:122] "RemoveContainer" containerID="d37b895467422ac33c93619db11018ba8c16ed05731868f0b5e6ad58b6aba746" Oct 31 13:15:21.582632 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:15:21.582611 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-54a16b-29890875-7jpzj_odh-ai-gateway-infra(eb473765-2907-4d0a-85f9-5f9fa76b77a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890875-7jpzj" podUID="eb473765-2907-4d0a-85f9-5f9fa76b77a0" Oct 31 13:15:21.641553 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:21.641489 5693 scope.go:122] "RemoveContainer" containerID="26cf450fb36796ee8741a1827b417bb67e30ea97b8f9c61c21f4b2d2bdab2540" Oct 31 13:15:21.648112 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:21.648070 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890875-r8b8z"] Oct 31 13:15:21.654330 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:21.654310 5693 scope.go:122] "RemoveContainer" containerID="49773b1eb803c748466b7965025d9d026635aabb7293892cc12962016b59c40c" Oct 31 13:15:21.665225 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:21.664979 5693 scope.go:122] "RemoveContainer" containerID="0b0f5425d505e17ebb309a3a62484b8608dbce9790c5a84155746ec31dd87763" Oct 31 13:15:21.681367 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:21.681346 5693 scope.go:122] "RemoveContainer" containerID="793158814fbbd12647fd2f83ad853268944e995034201552ca134afaa4596b17" Oct 31 13:15:21.693590 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:21.693566 5693 scope.go:122] "RemoveContainer" containerID="d8eb5108f46fb5b65eff27161b27ed822f830b5a1b84db0de8ac181e37d38355" Oct 31 13:15:21.709782 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:21.709729 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890875-tc5zf"] Oct 31 13:15:21.715061 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:21.715035 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890875-tc5zf"] Oct 31 13:15:21.735313 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:21.735277 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890875-6rsmb"] Oct 31 13:15:21.741868 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:21.741841 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890875-6rsmb"] Oct 31 13:15:21.758831 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:21.758803 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890875-wdmvp"] Oct 31 13:15:21.765329 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:21.765305 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890875-wdmvp"] Oct 31 13:15:22.675710 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:22.675674 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890875-hvbgx"] Oct 31 13:15:22.737959 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:22.737928 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890875-7jpzj"] Oct 31 13:15:22.916286 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:22.916250 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890875-r8b8z" Oct 31 13:15:23.007386 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:23.007347 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/195c9f96-e37a-46e1-b2c7-8f8a15fff884-kube-api-access-blfpk\" (UniqueName: \"kubernetes.io/projected/195c9f96-e37a-46e1-b2c7-8f8a15fff884-kube-api-access-blfpk\") pod \"195c9f96-e37a-46e1-b2c7-8f8a15fff884\" (UID: \"195c9f96-e37a-46e1-b2c7-8f8a15fff884\") " Oct 31 13:15:23.007536 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:23.007496 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/195c9f96-e37a-46e1-b2c7-8f8a15fff884-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/195c9f96-e37a-46e1-b2c7-8f8a15fff884-maas-api-service-ca\") pod \"195c9f96-e37a-46e1-b2c7-8f8a15fff884\" (UID: \"195c9f96-e37a-46e1-b2c7-8f8a15fff884\") " Oct 31 13:15:23.008100 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:23.008070 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/195c9f96-e37a-46e1-b2c7-8f8a15fff884-maas-api-service-ca" pod "195c9f96-e37a-46e1-b2c7-8f8a15fff884" (UID: "195c9f96-e37a-46e1-b2c7-8f8a15fff884"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:23.008541 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:23.008518 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/195c9f96-e37a-46e1-b2c7-8f8a15fff884-maas-api-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:23.015068 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:23.015040 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/195c9f96-e37a-46e1-b2c7-8f8a15fff884-kube-api-access-blfpk" pod "195c9f96-e37a-46e1-b2c7-8f8a15fff884" (UID: "195c9f96-e37a-46e1-b2c7-8f8a15fff884"). InnerVolumeSpecName "kube-api-access-blfpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:23.025421 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:23.025396 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890875-hvbgx" Oct 31 13:15:23.041611 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:23.041593 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890875-7jpzj" Oct 31 13:15:23.109785 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:23.109758 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb473765-2907-4d0a-85f9-5f9fa76b77a0-kube-api-access-4jc5g\" (UniqueName: \"kubernetes.io/projected/eb473765-2907-4d0a-85f9-5f9fa76b77a0-kube-api-access-4jc5g\") pod \"eb473765-2907-4d0a-85f9-5f9fa76b77a0\" (UID: \"eb473765-2907-4d0a-85f9-5f9fa76b77a0\") " Oct 31 13:15:23.109992 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:23.109917 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb473765-2907-4d0a-85f9-5f9fa76b77a0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb473765-2907-4d0a-85f9-5f9fa76b77a0-maas-api-service-ca\") pod \"eb473765-2907-4d0a-85f9-5f9fa76b77a0\" (UID: \"eb473765-2907-4d0a-85f9-5f9fa76b77a0\") " Oct 31 13:15:23.110086 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:23.110064 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb0c915e-681f-4fb6-bff9-91ae28572928-kube-api-access-b8rm5\" (UniqueName: \"kubernetes.io/projected/cb0c915e-681f-4fb6-bff9-91ae28572928-kube-api-access-b8rm5\") pod \"cb0c915e-681f-4fb6-bff9-91ae28572928\" (UID: \"cb0c915e-681f-4fb6-bff9-91ae28572928\") " Oct 31 13:15:23.110222 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:23.110204 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb0c915e-681f-4fb6-bff9-91ae28572928-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb0c915e-681f-4fb6-bff9-91ae28572928-maas-api-service-ca\") pod \"cb0c915e-681f-4fb6-bff9-91ae28572928\" (UID: \"cb0c915e-681f-4fb6-bff9-91ae28572928\") " Oct 31 13:15:23.110430 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:23.110399 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb473765-2907-4d0a-85f9-5f9fa76b77a0-maas-api-service-ca" pod "eb473765-2907-4d0a-85f9-5f9fa76b77a0" (UID: "eb473765-2907-4d0a-85f9-5f9fa76b77a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:23.110624 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:23.110601 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb0c915e-681f-4fb6-bff9-91ae28572928-maas-api-service-ca" pod "cb0c915e-681f-4fb6-bff9-91ae28572928" (UID: "cb0c915e-681f-4fb6-bff9-91ae28572928"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:23.110941 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:23.110921 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blfpk\" (UniqueName: \"kubernetes.io/projected/195c9f96-e37a-46e1-b2c7-8f8a15fff884-kube-api-access-blfpk\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:23.110941 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:23.110940 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb0c915e-681f-4fb6-bff9-91ae28572928-maas-api-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:23.111114 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:23.110949 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb473765-2907-4d0a-85f9-5f9fa76b77a0-maas-api-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:23.112981 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:23.112948 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb0c915e-681f-4fb6-bff9-91ae28572928-kube-api-access-b8rm5" pod "cb0c915e-681f-4fb6-bff9-91ae28572928" (UID: "cb0c915e-681f-4fb6-bff9-91ae28572928"). InnerVolumeSpecName "kube-api-access-b8rm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:23.113093 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:23.112983 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb473765-2907-4d0a-85f9-5f9fa76b77a0-kube-api-access-4jc5g" pod "eb473765-2907-4d0a-85f9-5f9fa76b77a0" (UID: "eb473765-2907-4d0a-85f9-5f9fa76b77a0"). InnerVolumeSpecName "kube-api-access-4jc5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:23.212858 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:23.212823 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jc5g\" (UniqueName: \"kubernetes.io/projected/eb473765-2907-4d0a-85f9-5f9fa76b77a0-kube-api-access-4jc5g\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:23.212858 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:23.212852 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8rm5\" (UniqueName: \"kubernetes.io/projected/cb0c915e-681f-4fb6-bff9-91ae28572928-kube-api-access-b8rm5\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:23.412632 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:23.412507 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22dbab0d-c6ad-4083-86bc-c5e0dfad5db3" path="/var/lib/kubelet/pods/22dbab0d-c6ad-4083-86bc-c5e0dfad5db3/volumes" Oct 31 13:15:23.413023 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:23.413004 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74df98d2-73a9-43bb-b0c2-258f69413210" path="/var/lib/kubelet/pods/74df98d2-73a9-43bb-b0c2-258f69413210/volumes" Oct 31 13:15:23.413406 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:23.413394 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31" path="/var/lib/kubelet/pods/cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31/volumes" Oct 31 13:15:23.626687 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:23.626585 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890875-hvbgx" Oct 31 13:15:23.626687 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:23.626598 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890875-hvbgx" event={"ID":"cb0c915e-681f-4fb6-bff9-91ae28572928","Type":"ContainerDied","Data":"8b9d8103bb7d31c66f6c10a0122e855d817138ebf10b9da5ff67e84a172a9181"} Oct 31 13:15:23.626946 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:23.626715 5693 scope.go:122] "RemoveContainer" containerID="32e28883869d668e6f3ca876d3b2154796dc895ad8e9649d612e542152695949" Oct 31 13:15:23.628183 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:23.628164 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890875-7jpzj" Oct 31 13:15:23.628453 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:23.628214 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890875-7jpzj" event={"ID":"eb473765-2907-4d0a-85f9-5f9fa76b77a0","Type":"ContainerDied","Data":"b6e3b5f89394d8b7576e036b95fccedb89ec065511f15580b02af617f1cb9f58"} Oct 31 13:15:23.629989 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:23.629968 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890875-r8b8z" event={"ID":"195c9f96-e37a-46e1-b2c7-8f8a15fff884","Type":"ContainerDied","Data":"00cc6f23dee7a454bc97f29968e566dea2198f3621a9c7f7e8afd29d00325796"} Oct 31 13:15:23.630109 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:23.630003 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890875-r8b8z" Oct 31 13:15:23.639654 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:23.639620 5693 scope.go:122] "RemoveContainer" containerID="d37b895467422ac33c93619db11018ba8c16ed05731868f0b5e6ad58b6aba746" Oct 31 13:15:23.649947 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:23.649926 5693 scope.go:122] "RemoveContainer" containerID="5e07b7dc1a74bbbb01508d527b57f6d59f0c598eb455a6919a7b136cfa0a6c8c" Oct 31 13:15:23.669312 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:23.669287 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890875-r8b8z"] Oct 31 13:15:23.678025 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:23.678000 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890875-r8b8z"] Oct 31 13:15:23.707374 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:23.707209 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890875-hvbgx"] Oct 31 13:15:23.711221 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:23.711194 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890875-hvbgx"] Oct 31 13:15:23.750145 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:23.750117 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890875-7jpzj"] Oct 31 13:15:23.754788 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:23.754761 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890875-7jpzj"] Oct 31 13:15:25.411450 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:25.411415 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="195c9f96-e37a-46e1-b2c7-8f8a15fff884" path="/var/lib/kubelet/pods/195c9f96-e37a-46e1-b2c7-8f8a15fff884/volumes" Oct 31 13:15:25.411942 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:25.411927 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb0c915e-681f-4fb6-bff9-91ae28572928" path="/var/lib/kubelet/pods/cb0c915e-681f-4fb6-bff9-91ae28572928/volumes" Oct 31 13:15:25.412383 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:25.412370 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb473765-2907-4d0a-85f9-5f9fa76b77a0" path="/var/lib/kubelet/pods/eb473765-2907-4d0a-85f9-5f9fa76b77a0/volumes" Oct 31 13:15:35.498678 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.498627 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j"] Oct 31 13:15:35.499522 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499503 5693 cpu_manager.go:414] "containerMap: removing container" podUID="eb473765-2907-4d0a-85f9-5f9fa76b77a0" containerName="cleanup" Oct 31 13:15:35.499567 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499523 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb473765-2907-4d0a-85f9-5f9fa76b77a0" containerName="cleanup" Oct 31 13:15:35.499567 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499542 5693 cpu_manager.go:414] "containerMap: removing container" podUID="cb0c915e-681f-4fb6-bff9-91ae28572928" containerName="cleanup" Oct 31 13:15:35.499567 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499547 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb0c915e-681f-4fb6-bff9-91ae28572928" containerName="cleanup" Oct 31 13:15:35.499567 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499556 5693 cpu_manager.go:414] "containerMap: removing container" podUID="cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31" containerName="cleanup" Oct 31 13:15:35.499567 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499562 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31" containerName="cleanup" Oct 31 13:15:35.499804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499569 5693 cpu_manager.go:414] "containerMap: removing container" podUID="eb473765-2907-4d0a-85f9-5f9fa76b77a0" containerName="cleanup" Oct 31 13:15:35.499804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499574 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb473765-2907-4d0a-85f9-5f9fa76b77a0" containerName="cleanup" Oct 31 13:15:35.499804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499580 5693 cpu_manager.go:414] "containerMap: removing container" podUID="a135ae97-5e5e-4d5d-b329-90a8de080e16" containerName="cleanup" Oct 31 13:15:35.499804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499584 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a135ae97-5e5e-4d5d-b329-90a8de080e16" containerName="cleanup" Oct 31 13:15:35.499804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499596 5693 cpu_manager.go:414] "containerMap: removing container" podUID="46d567ad-1dde-4d69-9ed1-8a2cab56e2db" containerName="payload-processing" Oct 31 13:15:35.499804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499600 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46d567ad-1dde-4d69-9ed1-8a2cab56e2db" containerName="payload-processing" Oct 31 13:15:35.499804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499610 5693 cpu_manager.go:414] "containerMap: removing container" podUID="74df98d2-73a9-43bb-b0c2-258f69413210" containerName="cleanup" Oct 31 13:15:35.499804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499614 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74df98d2-73a9-43bb-b0c2-258f69413210" containerName="cleanup" Oct 31 13:15:35.499804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499620 5693 cpu_manager.go:414] "containerMap: removing container" podUID="cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31" containerName="cleanup" Oct 31 13:15:35.499804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499624 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31" containerName="cleanup" Oct 31 13:15:35.499804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499630 5693 cpu_manager.go:414] "containerMap: removing container" podUID="22dbab0d-c6ad-4083-86bc-c5e0dfad5db3" containerName="cleanup" Oct 31 13:15:35.499804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499634 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22dbab0d-c6ad-4083-86bc-c5e0dfad5db3" containerName="cleanup" Oct 31 13:15:35.499804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499655 5693 cpu_manager.go:414] "containerMap: removing container" podUID="a135ae97-5e5e-4d5d-b329-90a8de080e16" containerName="cleanup" Oct 31 13:15:35.499804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499659 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a135ae97-5e5e-4d5d-b329-90a8de080e16" containerName="cleanup" Oct 31 13:15:35.499804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499669 5693 cpu_manager.go:414] "containerMap: removing container" podUID="74df98d2-73a9-43bb-b0c2-258f69413210" containerName="cleanup" Oct 31 13:15:35.499804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499673 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74df98d2-73a9-43bb-b0c2-258f69413210" containerName="cleanup" Oct 31 13:15:35.499804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499689 5693 cpu_manager.go:414] "containerMap: removing container" podUID="22dbab0d-c6ad-4083-86bc-c5e0dfad5db3" containerName="cleanup" Oct 31 13:15:35.499804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499693 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22dbab0d-c6ad-4083-86bc-c5e0dfad5db3" containerName="cleanup" Oct 31 13:15:35.499804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499699 5693 cpu_manager.go:414] "containerMap: removing container" podUID="705199d4-0bb0-4d6f-b08e-7cb3d8582e24" containerName="cleanup" Oct 31 13:15:35.499804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499703 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="705199d4-0bb0-4d6f-b08e-7cb3d8582e24" containerName="cleanup" Oct 31 13:15:35.499804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499713 5693 cpu_manager.go:414] "containerMap: removing container" podUID="22dbab0d-c6ad-4083-86bc-c5e0dfad5db3" containerName="cleanup" Oct 31 13:15:35.499804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499716 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22dbab0d-c6ad-4083-86bc-c5e0dfad5db3" containerName="cleanup" Oct 31 13:15:35.499804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499725 5693 cpu_manager.go:414] "containerMap: removing container" podUID="705199d4-0bb0-4d6f-b08e-7cb3d8582e24" containerName="cleanup" Oct 31 13:15:35.499804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499729 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="705199d4-0bb0-4d6f-b08e-7cb3d8582e24" containerName="cleanup" Oct 31 13:15:35.499804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499738 5693 cpu_manager.go:414] "containerMap: removing container" podUID="195c9f96-e37a-46e1-b2c7-8f8a15fff884" containerName="cleanup" Oct 31 13:15:35.499804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499742 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="195c9f96-e37a-46e1-b2c7-8f8a15fff884" containerName="cleanup" Oct 31 13:15:35.499804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499757 5693 cpu_manager.go:414] "containerMap: removing container" podUID="cb0c915e-681f-4fb6-bff9-91ae28572928" containerName="cleanup" Oct 31 13:15:35.499804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499761 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb0c915e-681f-4fb6-bff9-91ae28572928" containerName="cleanup" Oct 31 13:15:35.499804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499780 5693 cpu_manager.go:414] "containerMap: removing container" podUID="74df98d2-73a9-43bb-b0c2-258f69413210" containerName="cleanup" Oct 31 13:15:35.499804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499786 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74df98d2-73a9-43bb-b0c2-258f69413210" containerName="cleanup" Oct 31 13:15:35.499804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499795 5693 cpu_manager.go:414] "containerMap: removing container" podUID="9264e5f3-1234-47de-97f9-020dfb0b03f1" containerName="payload-pre-processing" Oct 31 13:15:35.499804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499801 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9264e5f3-1234-47de-97f9-020dfb0b03f1" containerName="payload-pre-processing" Oct 31 13:15:35.499804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499808 5693 cpu_manager.go:414] "containerMap: removing container" podUID="195c9f96-e37a-46e1-b2c7-8f8a15fff884" containerName="cleanup" Oct 31 13:15:35.499804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499813 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="195c9f96-e37a-46e1-b2c7-8f8a15fff884" containerName="cleanup" Oct 31 13:15:35.500864 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499929 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="46d567ad-1dde-4d69-9ed1-8a2cab56e2db" containerName="payload-processing" Oct 31 13:15:35.500864 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499939 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="74df98d2-73a9-43bb-b0c2-258f69413210" containerName="cleanup" Oct 31 13:15:35.500864 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499944 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31" containerName="cleanup" Oct 31 13:15:35.500864 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499950 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="22dbab0d-c6ad-4083-86bc-c5e0dfad5db3" containerName="cleanup" Oct 31 13:15:35.500864 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499956 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="a135ae97-5e5e-4d5d-b329-90a8de080e16" containerName="cleanup" Oct 31 13:15:35.500864 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499961 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="74df98d2-73a9-43bb-b0c2-258f69413210" containerName="cleanup" Oct 31 13:15:35.500864 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499968 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb0c915e-681f-4fb6-bff9-91ae28572928" containerName="cleanup" Oct 31 13:15:35.500864 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499972 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="22dbab0d-c6ad-4083-86bc-c5e0dfad5db3" containerName="cleanup" Oct 31 13:15:35.500864 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499977 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="705199d4-0bb0-4d6f-b08e-7cb3d8582e24" containerName="cleanup" Oct 31 13:15:35.500864 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499984 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="22dbab0d-c6ad-4083-86bc-c5e0dfad5db3" containerName="cleanup" Oct 31 13:15:35.500864 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499990 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="705199d4-0bb0-4d6f-b08e-7cb3d8582e24" containerName="cleanup" Oct 31 13:15:35.500864 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.499996 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="195c9f96-e37a-46e1-b2c7-8f8a15fff884" containerName="cleanup" Oct 31 13:15:35.500864 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.500002 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="705199d4-0bb0-4d6f-b08e-7cb3d8582e24" containerName="cleanup" Oct 31 13:15:35.500864 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.500007 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb473765-2907-4d0a-85f9-5f9fa76b77a0" containerName="cleanup" Oct 31 13:15:35.500864 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.500012 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb0c915e-681f-4fb6-bff9-91ae28572928" containerName="cleanup" Oct 31 13:15:35.500864 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.500017 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="a135ae97-5e5e-4d5d-b329-90a8de080e16" containerName="cleanup" Oct 31 13:15:35.500864 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.500023 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="9264e5f3-1234-47de-97f9-020dfb0b03f1" containerName="payload-pre-processing" Oct 31 13:15:35.500864 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.500029 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="195c9f96-e37a-46e1-b2c7-8f8a15fff884" containerName="cleanup" Oct 31 13:15:35.500864 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.500034 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb473765-2907-4d0a-85f9-5f9fa76b77a0" containerName="cleanup" Oct 31 13:15:35.500864 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.500041 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb0c915e-681f-4fb6-bff9-91ae28572928" containerName="cleanup" Oct 31 13:15:35.500864 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.500047 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31" containerName="cleanup" Oct 31 13:15:35.506269 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.506250 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j" Oct 31 13:15:35.522623 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.522596 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-29e3a0\"/\"payload-processing-external-model-e2e--e205175e-dockercfg-ssrqv\"" Oct 31 13:15:35.525086 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.525061 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j"] Oct 31 13:15:35.606379 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.606344 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d0e4939-0623-427a-9592-d438b52a2a5d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j\" (UID: \"8d0e4939-0623-427a-9592-d438b52a2a5d\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j" Oct 31 13:15:35.606593 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.606578 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8d0e4939-0623-427a-9592-d438b52a2a5d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j\" (UID: \"8d0e4939-0623-427a-9592-d438b52a2a5d\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j" Oct 31 13:15:35.606675 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.606609 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8d0e4939-0623-427a-9592-d438b52a2a5d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j\" (UID: \"8d0e4939-0623-427a-9592-d438b52a2a5d\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j" Oct 31 13:15:35.606675 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.606656 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvvh4\" (UniqueName: \"kubernetes.io/projected/8d0e4939-0623-427a-9592-d438b52a2a5d-kube-api-access-mvvh4\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j\" (UID: \"8d0e4939-0623-427a-9592-d438b52a2a5d\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j" Oct 31 13:15:35.630248 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.630224 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-29e3a0\"/\"agc-cred-26173d7f0173\"" Oct 31 13:15:35.708532 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.708497 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8d0e4939-0623-427a-9592-d438b52a2a5d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j\" (UID: \"8d0e4939-0623-427a-9592-d438b52a2a5d\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j" Oct 31 13:15:35.708532 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.708539 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8d0e4939-0623-427a-9592-d438b52a2a5d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j\" (UID: \"8d0e4939-0623-427a-9592-d438b52a2a5d\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j" Oct 31 13:15:35.708807 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.708571 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvvh4\" (UniqueName: \"kubernetes.io/projected/8d0e4939-0623-427a-9592-d438b52a2a5d-kube-api-access-mvvh4\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j\" (UID: \"8d0e4939-0623-427a-9592-d438b52a2a5d\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j" Oct 31 13:15:35.708807 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.708734 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d0e4939-0623-427a-9592-d438b52a2a5d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j\" (UID: \"8d0e4939-0623-427a-9592-d438b52a2a5d\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j" Oct 31 13:15:35.709267 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.709242 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8d0e4939-0623-427a-9592-d438b52a2a5d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j\" (UID: \"8d0e4939-0623-427a-9592-d438b52a2a5d\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j" Oct 31 13:15:35.709440 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.709421 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d0e4939-0623-427a-9592-d438b52a2a5d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j\" (UID: \"8d0e4939-0623-427a-9592-d438b52a2a5d\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j" Oct 31 13:15:35.711422 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.711399 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8d0e4939-0623-427a-9592-d438b52a2a5d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j\" (UID: \"8d0e4939-0623-427a-9592-d438b52a2a5d\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j" Oct 31 13:15:35.726359 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.726334 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvvh4\" (UniqueName: \"kubernetes.io/projected/8d0e4939-0623-427a-9592-d438b52a2a5d-kube-api-access-mvvh4\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j\" (UID: \"8d0e4939-0623-427a-9592-d438b52a2a5d\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j" Oct 31 13:15:35.821810 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:35.821732 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j" Oct 31 13:15:36.150088 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:36.150063 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j"] Oct 31 13:15:36.152771 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:15:36.152743 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d0e4939_0623_427a_9592_d438b52a2a5d.slice/crio-b19f7576080a25abbd3f614bce854cec069b8483367206e959fb74c6c0a91e21 WatchSource:0}: Error finding container b19f7576080a25abbd3f614bce854cec069b8483367206e959fb74c6c0a91e21: Status 404 returned error can't find the container with id b19f7576080a25abbd3f614bce854cec069b8483367206e959fb74c6c0a91e21 Oct 31 13:15:36.749855 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:36.749819 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j" event={"ID":"8d0e4939-0623-427a-9592-d438b52a2a5d","Type":"ContainerStarted","Data":"c1c92b854d08e54860182362065613d7379e9dd31a5d8a58244497c6060d7bcd"} Oct 31 13:15:36.750275 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:36.749877 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j" event={"ID":"8d0e4939-0623-427a-9592-d438b52a2a5d","Type":"ContainerStarted","Data":"b19f7576080a25abbd3f614bce854cec069b8483367206e959fb74c6c0a91e21"} Oct 31 13:15:36.750275 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:36.750022 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j" Oct 31 13:15:36.778782 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:36.778734 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j" podStartSLOduration=1.7787199519999999 podStartE2EDuration="1.778719952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:36.77617672 +0000 UTC m=+2587.483763016" watchObservedRunningTime="2026-10-31 13:15:36.778719952 +0000 UTC m=+2587.486306276" Oct 31 13:15:45.878119 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:45.878068 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg"] Oct 31 13:15:47.761788 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:47.761752 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j" Oct 31 13:15:47.827209 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:47.827163 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx"] Oct 31 13:15:48.689212 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:15:48.689175 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:15:50.879139 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:50.879087 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg" podUID="80106c11-89ff-40fb-9222-d2e431273541" containerName="payload-processing" containerID="cri-o://c777877c9c91fa6bf12e9f54cd8eb2bf958136aec81b0a3023bf9af0d2afb727" gracePeriod=25 Oct 31 13:15:51.370203 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:51.370175 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg" Oct 31 13:15:51.379897 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:51.379865 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80106c11-89ff-40fb-9222-d2e431273541-provider-credentials\" (UniqueName: \"kubernetes.io/projected/80106c11-89ff-40fb-9222-d2e431273541-provider-credentials\") pod \"80106c11-89ff-40fb-9222-d2e431273541\" (UID: \"80106c11-89ff-40fb-9222-d2e431273541\") " Oct 31 13:15:51.380039 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:51.379924 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80106c11-89ff-40fb-9222-d2e431273541-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80106c11-89ff-40fb-9222-d2e431273541-plugins-config-volume\") pod \"80106c11-89ff-40fb-9222-d2e431273541\" (UID: \"80106c11-89ff-40fb-9222-d2e431273541\") " Oct 31 13:15:51.380039 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:51.380011 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80106c11-89ff-40fb-9222-d2e431273541-kube-api-access-ln8x8\" (UniqueName: \"kubernetes.io/projected/80106c11-89ff-40fb-9222-d2e431273541-kube-api-access-ln8x8\") pod \"80106c11-89ff-40fb-9222-d2e431273541\" (UID: \"80106c11-89ff-40fb-9222-d2e431273541\") " Oct 31 13:15:51.380169 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:51.380090 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80106c11-89ff-40fb-9222-d2e431273541-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80106c11-89ff-40fb-9222-d2e431273541-routing-overlay-volume\") pod \"80106c11-89ff-40fb-9222-d2e431273541\" (UID: \"80106c11-89ff-40fb-9222-d2e431273541\") " Oct 31 13:15:51.380403 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:51.380316 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80106c11-89ff-40fb-9222-d2e431273541-plugins-config-volume" pod "80106c11-89ff-40fb-9222-d2e431273541" (UID: "80106c11-89ff-40fb-9222-d2e431273541"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:51.380502 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:51.380479 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80106c11-89ff-40fb-9222-d2e431273541-routing-overlay-volume" pod "80106c11-89ff-40fb-9222-d2e431273541" (UID: "80106c11-89ff-40fb-9222-d2e431273541"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:51.381124 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:51.381101 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80106c11-89ff-40fb-9222-d2e431273541-routing-overlay-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:51.381172 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:51.381134 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80106c11-89ff-40fb-9222-d2e431273541-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:51.382447 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:51.382418 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80106c11-89ff-40fb-9222-d2e431273541-provider-credentials" pod "80106c11-89ff-40fb-9222-d2e431273541" (UID: "80106c11-89ff-40fb-9222-d2e431273541"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:51.382544 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:51.382473 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80106c11-89ff-40fb-9222-d2e431273541-kube-api-access-ln8x8" pod "80106c11-89ff-40fb-9222-d2e431273541" (UID: "80106c11-89ff-40fb-9222-d2e431273541"). InnerVolumeSpecName "kube-api-access-ln8x8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:51.483380 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:51.483340 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/80106c11-89ff-40fb-9222-d2e431273541-provider-credentials\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:51.483380 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:51.483380 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln8x8\" (UniqueName: \"kubernetes.io/projected/80106c11-89ff-40fb-9222-d2e431273541-kube-api-access-ln8x8\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:51.925915 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:51.925879 5693 generic.go:356] "Generic (PLEG): container finished" podID="80106c11-89ff-40fb-9222-d2e431273541" containerID="c777877c9c91fa6bf12e9f54cd8eb2bf958136aec81b0a3023bf9af0d2afb727" exitCode=0 Oct 31 13:15:51.926448 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:51.925931 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg" Oct 31 13:15:51.926448 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:51.925968 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg" event={"ID":"80106c11-89ff-40fb-9222-d2e431273541","Type":"ContainerDied","Data":"c777877c9c91fa6bf12e9f54cd8eb2bf958136aec81b0a3023bf9af0d2afb727"} Oct 31 13:15:51.926448 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:51.926036 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg" event={"ID":"80106c11-89ff-40fb-9222-d2e431273541","Type":"ContainerDied","Data":"aee01a9143d3237c8bf8940c8afe478574419901c95aa9e2c939c81d8b52eb26"} Oct 31 13:15:51.926448 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:51.926064 5693 scope.go:122] "RemoveContainer" containerID="c777877c9c91fa6bf12e9f54cd8eb2bf958136aec81b0a3023bf9af0d2afb727" Oct 31 13:15:51.938752 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:51.938733 5693 scope.go:122] "RemoveContainer" containerID="c777877c9c91fa6bf12e9f54cd8eb2bf958136aec81b0a3023bf9af0d2afb727" Oct 31 13:15:51.939078 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:15:51.939048 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c777877c9c91fa6bf12e9f54cd8eb2bf958136aec81b0a3023bf9af0d2afb727\": container with ID starting with c777877c9c91fa6bf12e9f54cd8eb2bf958136aec81b0a3023bf9af0d2afb727 not found: ID does not exist" containerID="c777877c9c91fa6bf12e9f54cd8eb2bf958136aec81b0a3023bf9af0d2afb727" Oct 31 13:15:51.939133 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:51.939086 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c777877c9c91fa6bf12e9f54cd8eb2bf958136aec81b0a3023bf9af0d2afb727"} err="failed to get container status \"c777877c9c91fa6bf12e9f54cd8eb2bf958136aec81b0a3023bf9af0d2afb727\": rpc error: code = NotFound desc = could not find container \"c777877c9c91fa6bf12e9f54cd8eb2bf958136aec81b0a3023bf9af0d2afb727\": container with ID starting with c777877c9c91fa6bf12e9f54cd8eb2bf958136aec81b0a3023bf9af0d2afb727 not found: ID does not exist" Oct 31 13:15:51.952003 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:51.951976 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg"] Oct 31 13:15:51.955871 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:51.955849 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79sbtfg"] Oct 31 13:15:52.828473 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:52.828420 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx" podUID="d77d1186-90c6-426b-aa02-758d811fb743" containerName="payload-processing" containerID="cri-o://4638ad65c377854e5671af3908a96c59e5efbc7c61dd47d3c474c2e91196f734" gracePeriod=25 Oct 31 13:15:53.173504 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:53.173470 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx" Oct 31 13:15:53.208816 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:53.208776 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d77d1186-90c6-426b-aa02-758d811fb743-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d77d1186-90c6-426b-aa02-758d811fb743-plugins-config-volume\") pod \"d77d1186-90c6-426b-aa02-758d811fb743\" (UID: \"d77d1186-90c6-426b-aa02-758d811fb743\") " Oct 31 13:15:53.208982 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:53.208830 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d77d1186-90c6-426b-aa02-758d811fb743-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d77d1186-90c6-426b-aa02-758d811fb743-provider-credentials\") pod \"d77d1186-90c6-426b-aa02-758d811fb743\" (UID: \"d77d1186-90c6-426b-aa02-758d811fb743\") " Oct 31 13:15:53.208982 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:53.208962 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d77d1186-90c6-426b-aa02-758d811fb743-kube-api-access-sttbb\" (UniqueName: \"kubernetes.io/projected/d77d1186-90c6-426b-aa02-758d811fb743-kube-api-access-sttbb\") pod \"d77d1186-90c6-426b-aa02-758d811fb743\" (UID: \"d77d1186-90c6-426b-aa02-758d811fb743\") " Oct 31 13:15:53.209128 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:53.209107 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d77d1186-90c6-426b-aa02-758d811fb743-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d77d1186-90c6-426b-aa02-758d811fb743-routing-overlay-volume\") pod \"d77d1186-90c6-426b-aa02-758d811fb743\" (UID: \"d77d1186-90c6-426b-aa02-758d811fb743\") " Oct 31 13:15:53.209281 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:53.209247 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d77d1186-90c6-426b-aa02-758d811fb743-plugins-config-volume" pod "d77d1186-90c6-426b-aa02-758d811fb743" (UID: "d77d1186-90c6-426b-aa02-758d811fb743"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:53.209535 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:53.209509 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d77d1186-90c6-426b-aa02-758d811fb743-routing-overlay-volume" pod "d77d1186-90c6-426b-aa02-758d811fb743" (UID: "d77d1186-90c6-426b-aa02-758d811fb743"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:53.210516 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:53.210489 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d77d1186-90c6-426b-aa02-758d811fb743-routing-overlay-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:53.210622 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:53.210520 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d77d1186-90c6-426b-aa02-758d811fb743-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:53.211364 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:53.211333 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d77d1186-90c6-426b-aa02-758d811fb743-provider-credentials" pod "d77d1186-90c6-426b-aa02-758d811fb743" (UID: "d77d1186-90c6-426b-aa02-758d811fb743"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:53.211769 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:53.211749 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d77d1186-90c6-426b-aa02-758d811fb743-kube-api-access-sttbb" pod "d77d1186-90c6-426b-aa02-758d811fb743" (UID: "d77d1186-90c6-426b-aa02-758d811fb743"). InnerVolumeSpecName "kube-api-access-sttbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:53.312615 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:53.312558 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sttbb\" (UniqueName: \"kubernetes.io/projected/d77d1186-90c6-426b-aa02-758d811fb743-kube-api-access-sttbb\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:53.312615 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:53.312600 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d77d1186-90c6-426b-aa02-758d811fb743-provider-credentials\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:53.411228 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:53.411198 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80106c11-89ff-40fb-9222-d2e431273541" path="/var/lib/kubelet/pods/80106c11-89ff-40fb-9222-d2e431273541/volumes" Oct 31 13:15:53.969153 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:53.967419 5693 generic.go:356] "Generic (PLEG): container finished" podID="d77d1186-90c6-426b-aa02-758d811fb743" containerID="4638ad65c377854e5671af3908a96c59e5efbc7c61dd47d3c474c2e91196f734" exitCode=0 Oct 31 13:15:53.969153 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:53.967723 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx" event={"ID":"d77d1186-90c6-426b-aa02-758d811fb743","Type":"ContainerDied","Data":"4638ad65c377854e5671af3908a96c59e5efbc7c61dd47d3c474c2e91196f734"} Oct 31 13:15:53.969153 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:53.967795 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx" event={"ID":"d77d1186-90c6-426b-aa02-758d811fb743","Type":"ContainerDied","Data":"8f46f617713e147f0522ba19dc9eda343dd937f2435961fbb8694dbec5ad8f65"} Oct 31 13:15:53.969153 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:53.967833 5693 scope.go:122] "RemoveContainer" containerID="4638ad65c377854e5671af3908a96c59e5efbc7c61dd47d3c474c2e91196f734" Oct 31 13:15:53.969153 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:53.967967 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx" Oct 31 13:15:53.984860 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:53.984840 5693 scope.go:122] "RemoveContainer" containerID="4638ad65c377854e5671af3908a96c59e5efbc7c61dd47d3c474c2e91196f734" Oct 31 13:15:53.985193 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:15:53.985173 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4638ad65c377854e5671af3908a96c59e5efbc7c61dd47d3c474c2e91196f734\": container with ID starting with 4638ad65c377854e5671af3908a96c59e5efbc7c61dd47d3c474c2e91196f734 not found: ID does not exist" containerID="4638ad65c377854e5671af3908a96c59e5efbc7c61dd47d3c474c2e91196f734" Oct 31 13:15:53.985249 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:53.985203 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4638ad65c377854e5671af3908a96c59e5efbc7c61dd47d3c474c2e91196f734"} err="failed to get container status \"4638ad65c377854e5671af3908a96c59e5efbc7c61dd47d3c474c2e91196f734\": rpc error: code = NotFound desc = could not find container \"4638ad65c377854e5671af3908a96c59e5efbc7c61dd47d3c474c2e91196f734\": container with ID starting with 4638ad65c377854e5671af3908a96c59e5efbc7c61dd47d3c474c2e91196f734 not found: ID does not exist" Oct 31 13:15:53.993791 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:53.993765 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx"] Oct 31 13:15:54.004480 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:54.004444 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958m7dxx"] Oct 31 13:15:55.410835 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:15:55.410803 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d77d1186-90c6-426b-aa02-758d811fb743" path="/var/lib/kubelet/pods/d77d1186-90c6-426b-aa02-758d811fb743/volumes" Oct 31 13:16:18.694429 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:16:18.694348 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:16:19.031271 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.031191 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5a9e45-698ff58799-ztzw8"] Oct 31 13:16:19.032196 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.032174 5693 cpu_manager.go:414] "containerMap: removing container" podUID="705199d4-0bb0-4d6f-b08e-7cb3d8582e24" containerName="cleanup" Oct 31 13:16:19.032196 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.032196 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="705199d4-0bb0-4d6f-b08e-7cb3d8582e24" containerName="cleanup" Oct 31 13:16:19.032428 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.032213 5693 cpu_manager.go:414] "containerMap: removing container" podUID="80106c11-89ff-40fb-9222-d2e431273541" containerName="payload-processing" Oct 31 13:16:19.032428 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.032220 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80106c11-89ff-40fb-9222-d2e431273541" containerName="payload-processing" Oct 31 13:16:19.032428 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.032238 5693 cpu_manager.go:414] "containerMap: removing container" podUID="d77d1186-90c6-426b-aa02-758d811fb743" containerName="payload-processing" Oct 31 13:16:19.032428 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.032243 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d77d1186-90c6-426b-aa02-758d811fb743" containerName="payload-processing" Oct 31 13:16:19.032428 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.032250 5693 cpu_manager.go:414] "containerMap: removing container" podUID="cb0c915e-681f-4fb6-bff9-91ae28572928" containerName="cleanup" Oct 31 13:16:19.032428 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.032255 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb0c915e-681f-4fb6-bff9-91ae28572928" containerName="cleanup" Oct 31 13:16:19.032428 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.032271 5693 cpu_manager.go:414] "containerMap: removing container" podUID="195c9f96-e37a-46e1-b2c7-8f8a15fff884" containerName="cleanup" Oct 31 13:16:19.032428 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.032275 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="195c9f96-e37a-46e1-b2c7-8f8a15fff884" containerName="cleanup" Oct 31 13:16:19.032428 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.032283 5693 cpu_manager.go:414] "containerMap: removing container" podUID="eb473765-2907-4d0a-85f9-5f9fa76b77a0" containerName="cleanup" Oct 31 13:16:19.032428 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.032287 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb473765-2907-4d0a-85f9-5f9fa76b77a0" containerName="cleanup" Oct 31 13:16:19.032428 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.032308 5693 cpu_manager.go:414] "containerMap: removing container" podUID="cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31" containerName="cleanup" Oct 31 13:16:19.032428 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.032312 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31" containerName="cleanup" Oct 31 13:16:19.032428 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.032320 5693 cpu_manager.go:414] "containerMap: removing container" podUID="a135ae97-5e5e-4d5d-b329-90a8de080e16" containerName="cleanup" Oct 31 13:16:19.032428 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.032326 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a135ae97-5e5e-4d5d-b329-90a8de080e16" containerName="cleanup" Oct 31 13:16:19.033025 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.032503 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb473765-2907-4d0a-85f9-5f9fa76b77a0" containerName="cleanup" Oct 31 13:16:19.033025 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.032522 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="d77d1186-90c6-426b-aa02-758d811fb743" containerName="payload-processing" Oct 31 13:16:19.033025 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.032534 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="195c9f96-e37a-46e1-b2c7-8f8a15fff884" containerName="cleanup" Oct 31 13:16:19.033025 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.032546 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbc6d20d-2b6f-4ce4-8d5f-28eb518f8b31" containerName="cleanup" Oct 31 13:16:19.033025 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.032558 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="74df98d2-73a9-43bb-b0c2-258f69413210" containerName="cleanup" Oct 31 13:16:19.033025 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.032566 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="a135ae97-5e5e-4d5d-b329-90a8de080e16" containerName="cleanup" Oct 31 13:16:19.033025 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.032579 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="80106c11-89ff-40fb-9222-d2e431273541" containerName="payload-processing" Oct 31 13:16:19.038206 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.038179 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5a9e45-698ff58799-ztzw8" Oct 31 13:16:19.046898 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.046718 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5a9e45-698ff58799-ztzw8"] Oct 31 13:16:19.047307 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.047263 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-5a9e45\"" Oct 31 13:16:19.048555 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.048531 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-5a9e45-dockercfg-fbsrk\"" Oct 31 13:16:19.101470 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.101434 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5a9e45-7b647fbff9-6xcpd"] Oct 31 13:16:19.107557 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.107528 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-5a9e45-7b647fbff9-6xcpd" Oct 31 13:16:19.145672 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.144894 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5a9e45-7b647fbff9-6xcpd"] Oct 31 13:16:19.192123 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.184907 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b21df5f3-b4bb-4e47-9287-b42befdf81f1-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-5a9e45-7b647fbff9-6xcpd\" (UID: \"b21df5f3-b4bb-4e47-9287-b42befdf81f1\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5a9e45-7b647fbff9-6xcpd" Oct 31 13:16:19.192123 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.185141 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v97r8\" (UniqueName: \"kubernetes.io/projected/b21df5f3-b4bb-4e47-9287-b42befdf81f1-kube-api-access-v97r8\") pod \"payload-processing-e2e-extmod-b-5a9e45-7b647fbff9-6xcpd\" (UID: \"b21df5f3-b4bb-4e47-9287-b42befdf81f1\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5a9e45-7b647fbff9-6xcpd" Oct 31 13:16:19.192123 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.185238 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b31a91e-a876-450d-b35c-d5134eb09c1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-5a9e45-698ff58799-ztzw8\" (UID: \"2b31a91e-a876-450d-b35c-d5134eb09c1c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5a9e45-698ff58799-ztzw8" Oct 31 13:16:19.192123 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.185321 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf74f\" (UniqueName: \"kubernetes.io/projected/2b31a91e-a876-450d-b35c-d5134eb09c1c-kube-api-access-gf74f\") pod \"payload-pre-processing-e2e-extmod-b-5a9e45-698ff58799-ztzw8\" (UID: \"2b31a91e-a876-450d-b35c-d5134eb09c1c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5a9e45-698ff58799-ztzw8" Oct 31 13:16:19.298715 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.296920 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf74f\" (UniqueName: \"kubernetes.io/projected/2b31a91e-a876-450d-b35c-d5134eb09c1c-kube-api-access-gf74f\") pod \"payload-pre-processing-e2e-extmod-b-5a9e45-698ff58799-ztzw8\" (UID: \"2b31a91e-a876-450d-b35c-d5134eb09c1c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5a9e45-698ff58799-ztzw8" Oct 31 13:16:19.298715 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.297077 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b21df5f3-b4bb-4e47-9287-b42befdf81f1-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-5a9e45-7b647fbff9-6xcpd\" (UID: \"b21df5f3-b4bb-4e47-9287-b42befdf81f1\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5a9e45-7b647fbff9-6xcpd" Oct 31 13:16:19.298715 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.297298 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v97r8\" (UniqueName: \"kubernetes.io/projected/b21df5f3-b4bb-4e47-9287-b42befdf81f1-kube-api-access-v97r8\") pod \"payload-processing-e2e-extmod-b-5a9e45-7b647fbff9-6xcpd\" (UID: \"b21df5f3-b4bb-4e47-9287-b42befdf81f1\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5a9e45-7b647fbff9-6xcpd" Oct 31 13:16:19.298715 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.297397 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b31a91e-a876-450d-b35c-d5134eb09c1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-5a9e45-698ff58799-ztzw8\" (UID: \"2b31a91e-a876-450d-b35c-d5134eb09c1c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5a9e45-698ff58799-ztzw8" Oct 31 13:16:19.298715 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.298471 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b31a91e-a876-450d-b35c-d5134eb09c1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-5a9e45-698ff58799-ztzw8\" (UID: \"2b31a91e-a876-450d-b35c-d5134eb09c1c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5a9e45-698ff58799-ztzw8" Oct 31 13:16:19.299634 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.299600 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b21df5f3-b4bb-4e47-9287-b42befdf81f1-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-5a9e45-7b647fbff9-6xcpd\" (UID: \"b21df5f3-b4bb-4e47-9287-b42befdf81f1\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5a9e45-7b647fbff9-6xcpd" Oct 31 13:16:19.387908 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.387832 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v97r8\" (UniqueName: \"kubernetes.io/projected/b21df5f3-b4bb-4e47-9287-b42befdf81f1-kube-api-access-v97r8\") pod \"payload-processing-e2e-extmod-b-5a9e45-7b647fbff9-6xcpd\" (UID: \"b21df5f3-b4bb-4e47-9287-b42befdf81f1\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5a9e45-7b647fbff9-6xcpd" Oct 31 13:16:19.409913 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.409858 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf74f\" (UniqueName: \"kubernetes.io/projected/2b31a91e-a876-450d-b35c-d5134eb09c1c-kube-api-access-gf74f\") pod \"payload-pre-processing-e2e-extmod-b-5a9e45-698ff58799-ztzw8\" (UID: \"2b31a91e-a876-450d-b35c-d5134eb09c1c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5a9e45-698ff58799-ztzw8" Oct 31 13:16:19.541882 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.541845 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-5a9e45-7b647fbff9-6xcpd" Oct 31 13:16:19.669173 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.668900 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5a9e45-698ff58799-ztzw8" Oct 31 13:16:19.833858 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.833824 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5a9e45-7b647fbff9-6xcpd"] Oct 31 13:16:19.963304 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:16:19.963272 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b31a91e_a876_450d_b35c_d5134eb09c1c.slice/crio-81e731b4d35173ec0f746e51121de8e747cc1b0b25c67f7dfac4fba4d789fe3a WatchSource:0}: Error finding container 81e731b4d35173ec0f746e51121de8e747cc1b0b25c67f7dfac4fba4d789fe3a: Status 404 returned error can't find the container with id 81e731b4d35173ec0f746e51121de8e747cc1b0b25c67f7dfac4fba4d789fe3a Oct 31 13:16:19.963452 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:19.963380 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5a9e45-698ff58799-ztzw8"] Oct 31 13:16:20.389089 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:20.388987 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5a9e45-698ff58799-ztzw8" event={"ID":"2b31a91e-a876-450d-b35c-d5134eb09c1c","Type":"ContainerStarted","Data":"f9dce66d13c776d6a60bd7e3902907f8778805af66be47d432db359df8ec81ac"} Oct 31 13:16:20.389089 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:20.389046 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5a9e45-698ff58799-ztzw8" event={"ID":"2b31a91e-a876-450d-b35c-d5134eb09c1c","Type":"ContainerStarted","Data":"81e731b4d35173ec0f746e51121de8e747cc1b0b25c67f7dfac4fba4d789fe3a"} Oct 31 13:16:20.389089 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:20.389078 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5a9e45-698ff58799-ztzw8" Oct 31 13:16:20.390950 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:20.390924 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-5a9e45-7b647fbff9-6xcpd" event={"ID":"b21df5f3-b4bb-4e47-9287-b42befdf81f1","Type":"ContainerStarted","Data":"543a985c382264c3ed555b9e0ebd58c69e1182de2dfc79493a48f4ce118692dd"} Oct 31 13:16:20.391083 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:20.390970 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-5a9e45-7b647fbff9-6xcpd" event={"ID":"b21df5f3-b4bb-4e47-9287-b42befdf81f1","Type":"ContainerStarted","Data":"c7f1637d0887ceccbc665c43aeab0308806943ee75953d3edc8894aeb44668c1"} Oct 31 13:16:20.391083 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:20.391001 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-5a9e45-7b647fbff9-6xcpd" Oct 31 13:16:20.411191 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:20.411142 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5a9e45-698ff58799-ztzw8" podStartSLOduration=1.4111304869999999 podStartE2EDuration="1.411130487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:16:20.408592585 +0000 UTC m=+2631.116178871" watchObservedRunningTime="2026-10-31 13:16:20.411130487 +0000 UTC m=+2631.118716782" Oct 31 13:16:20.428547 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:20.428486 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-5a9e45-7b647fbff9-6xcpd" podStartSLOduration=1.4284704320000001 podStartE2EDuration="1.428470432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:16:20.426753132 +0000 UTC m=+2631.134339433" watchObservedRunningTime="2026-10-31 13:16:20.428470432 +0000 UTC m=+2631.136056729" Oct 31 13:16:25.535396 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:25.535363 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6"] Oct 31 13:16:25.541922 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:25.541900 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6" Oct 31 13:16:25.553754 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:25.553611 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-54a16b\"/\"routing-overlay\"" Oct 31 13:16:25.554470 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:25.554450 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-54a16b\"/\"openshift-service-ca.crt\"" Oct 31 13:16:25.554801 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:25.554779 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-54a16b\"/\"agc-cred-b26c7c3f28af\"" Oct 31 13:16:25.555122 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:25.554943 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-54a16b\"/\"payload-processing-external-model-e2e--427d574a-dockercfg-d5cg8\"" Oct 31 13:16:25.555122 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:25.555067 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-54a16b\"/\"payload-processing-external-model-plugins-e2e-extmod-a-54a16b\"" Oct 31 13:16:25.555816 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:25.555798 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-54a16b\"/\"kube-root-ca.crt\"" Oct 31 13:16:25.560331 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:25.560309 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6"] Oct 31 13:16:25.718226 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:25.718169 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/91a9e9c0-9683-4456-803a-ce6e233e2c31-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6\" (UID: \"91a9e9c0-9683-4456-803a-ce6e233e2c31\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6" Oct 31 13:16:25.718453 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:25.718356 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91a9e9c0-9683-4456-803a-ce6e233e2c31-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6\" (UID: \"91a9e9c0-9683-4456-803a-ce6e233e2c31\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6" Oct 31 13:16:25.718527 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:25.718485 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91a9e9c0-9683-4456-803a-ce6e233e2c31-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6\" (UID: \"91a9e9c0-9683-4456-803a-ce6e233e2c31\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6" Oct 31 13:16:25.718527 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:25.718513 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k64gt\" (UniqueName: \"kubernetes.io/projected/91a9e9c0-9683-4456-803a-ce6e233e2c31-kube-api-access-k64gt\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6\" (UID: \"91a9e9c0-9683-4456-803a-ce6e233e2c31\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6" Oct 31 13:16:25.836021 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:25.821941 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/91a9e9c0-9683-4456-803a-ce6e233e2c31-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6\" (UID: \"91a9e9c0-9683-4456-803a-ce6e233e2c31\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6" Oct 31 13:16:25.836021 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:25.822100 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91a9e9c0-9683-4456-803a-ce6e233e2c31-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6\" (UID: \"91a9e9c0-9683-4456-803a-ce6e233e2c31\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6" Oct 31 13:16:25.836021 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:25.822239 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91a9e9c0-9683-4456-803a-ce6e233e2c31-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6\" (UID: \"91a9e9c0-9683-4456-803a-ce6e233e2c31\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6" Oct 31 13:16:25.836021 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:25.822261 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k64gt\" (UniqueName: \"kubernetes.io/projected/91a9e9c0-9683-4456-803a-ce6e233e2c31-kube-api-access-k64gt\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6\" (UID: \"91a9e9c0-9683-4456-803a-ce6e233e2c31\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6" Oct 31 13:16:25.836021 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:25.829666 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91a9e9c0-9683-4456-803a-ce6e233e2c31-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6\" (UID: \"91a9e9c0-9683-4456-803a-ce6e233e2c31\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6" Oct 31 13:16:25.836021 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:25.830242 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91a9e9c0-9683-4456-803a-ce6e233e2c31-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6\" (UID: \"91a9e9c0-9683-4456-803a-ce6e233e2c31\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6" Oct 31 13:16:25.853941 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:25.853904 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k64gt\" (UniqueName: \"kubernetes.io/projected/91a9e9c0-9683-4456-803a-ce6e233e2c31-kube-api-access-k64gt\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6\" (UID: \"91a9e9c0-9683-4456-803a-ce6e233e2c31\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6" Oct 31 13:16:25.881684 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:25.881633 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/91a9e9c0-9683-4456-803a-ce6e233e2c31-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6\" (UID: \"91a9e9c0-9683-4456-803a-ce6e233e2c31\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6" Oct 31 13:16:26.100859 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:26.100778 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48"] Oct 31 13:16:26.105945 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:26.105925 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48" Oct 31 13:16:26.111458 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:26.111436 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0c1fbb\"/\"agc-cred-5db32c229bfc\"" Oct 31 13:16:26.115154 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:26.114951 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0c1fbb\"/\"routing-overlay\"" Oct 31 13:16:26.115664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:26.114956 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0c1fbb\"/\"payload-processing-external-model-plugins-e2e-extmod-b-0c1fbb\"" Oct 31 13:16:26.116596 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:26.115951 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0c1fbb\"/\"kube-root-ca.crt\"" Oct 31 13:16:26.116596 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:26.116286 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0c1fbb\"/\"openshift-service-ca.crt\"" Oct 31 13:16:26.121193 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:26.121172 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48"] Oct 31 13:16:26.128291 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:26.128261 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b8e9166c-4c50-4ae0-ba02-30396f4468c3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48\" (UID: \"b8e9166c-4c50-4ae0-ba02-30396f4468c3\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48" Oct 31 13:16:26.128516 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:26.128496 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2thv7\" (UniqueName: \"kubernetes.io/projected/b8e9166c-4c50-4ae0-ba02-30396f4468c3-kube-api-access-2thv7\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48\" (UID: \"b8e9166c-4c50-4ae0-ba02-30396f4468c3\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48" Oct 31 13:16:26.128688 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:26.128658 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8e9166c-4c50-4ae0-ba02-30396f4468c3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48\" (UID: \"b8e9166c-4c50-4ae0-ba02-30396f4468c3\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48" Oct 31 13:16:26.128809 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:26.128730 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b8e9166c-4c50-4ae0-ba02-30396f4468c3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48\" (UID: \"b8e9166c-4c50-4ae0-ba02-30396f4468c3\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48" Oct 31 13:16:26.183665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:26.176266 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6" Oct 31 13:16:26.189463 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:26.189431 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0c1fbb\"/\"payload-processing-external-model-e2e--9527f114-dockercfg-5ljgf\"" Oct 31 13:16:26.191704 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:26.191665 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0c1fbb\"/\"agc-cred-e4da4b834193\"" Oct 31 13:16:26.230434 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:26.230394 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b8e9166c-4c50-4ae0-ba02-30396f4468c3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48\" (UID: \"b8e9166c-4c50-4ae0-ba02-30396f4468c3\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48" Oct 31 13:16:26.230709 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:26.230529 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b8e9166c-4c50-4ae0-ba02-30396f4468c3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48\" (UID: \"b8e9166c-4c50-4ae0-ba02-30396f4468c3\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48" Oct 31 13:16:26.230816 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:26.230788 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2thv7\" (UniqueName: \"kubernetes.io/projected/b8e9166c-4c50-4ae0-ba02-30396f4468c3-kube-api-access-2thv7\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48\" (UID: \"b8e9166c-4c50-4ae0-ba02-30396f4468c3\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48" Oct 31 13:16:26.230923 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:26.230908 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8e9166c-4c50-4ae0-ba02-30396f4468c3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48\" (UID: \"b8e9166c-4c50-4ae0-ba02-30396f4468c3\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48" Oct 31 13:16:26.232567 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:26.232003 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8e9166c-4c50-4ae0-ba02-30396f4468c3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48\" (UID: \"b8e9166c-4c50-4ae0-ba02-30396f4468c3\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48" Oct 31 13:16:26.232792 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:26.232697 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b8e9166c-4c50-4ae0-ba02-30396f4468c3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48\" (UID: \"b8e9166c-4c50-4ae0-ba02-30396f4468c3\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48" Oct 31 13:16:26.245718 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:26.237349 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b8e9166c-4c50-4ae0-ba02-30396f4468c3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48\" (UID: \"b8e9166c-4c50-4ae0-ba02-30396f4468c3\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48" Oct 31 13:16:26.260089 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:26.260065 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2thv7\" (UniqueName: \"kubernetes.io/projected/b8e9166c-4c50-4ae0-ba02-30396f4468c3-kube-api-access-2thv7\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48\" (UID: \"b8e9166c-4c50-4ae0-ba02-30396f4468c3\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48" Oct 31 13:16:26.424345 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:26.423986 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48" Oct 31 13:16:27.012378 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.012346 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6"] Oct 31 13:16:27.012977 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.012956 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48"] Oct 31 13:16:27.246741 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.246711 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg"] Oct 31 13:16:27.265766 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.265614 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg"] Oct 31 13:16:27.265766 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.265764 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg" Oct 31 13:16:27.277204 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.277165 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-29e3a0\"/\"agc-cred-5442391e26cb\"" Oct 31 13:16:27.346781 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.346749 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg\" (UID: \"19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg" Oct 31 13:16:27.346978 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.346807 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvzts\" (UniqueName: \"kubernetes.io/projected/19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d-kube-api-access-tvzts\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg\" (UID: \"19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg" Oct 31 13:16:27.347097 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.346987 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg\" (UID: \"19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg" Oct 31 13:16:27.347097 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.347039 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg\" (UID: \"19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg" Oct 31 13:16:27.449454 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.449359 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg\" (UID: \"19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg" Oct 31 13:16:27.449756 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.449729 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvzts\" (UniqueName: \"kubernetes.io/projected/19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d-kube-api-access-tvzts\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg\" (UID: \"19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg" Oct 31 13:16:27.450022 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.450001 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg\" (UID: \"19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg" Oct 31 13:16:27.450160 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.450144 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg\" (UID: \"19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg" Oct 31 13:16:27.450327 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.450298 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg\" (UID: \"19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg" Oct 31 13:16:27.451007 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.450977 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg\" (UID: \"19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg" Oct 31 13:16:27.454188 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.454164 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg\" (UID: \"19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg" Oct 31 13:16:27.463009 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.462973 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvzts\" (UniqueName: \"kubernetes.io/projected/19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d-kube-api-access-tvzts\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg\" (UID: \"19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg" Oct 31 13:16:27.585967 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.585923 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48" event={"ID":"b8e9166c-4c50-4ae0-ba02-30396f4468c3","Type":"ContainerStarted","Data":"22c2ae36537df5d5bb0e9d8412beb6dfa498d1b55a69b6d7741ad7fc46a23f89"} Oct 31 13:16:27.586165 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.585985 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48" event={"ID":"b8e9166c-4c50-4ae0-ba02-30396f4468c3","Type":"ContainerStarted","Data":"6b0bd7f939f2d671ba365b2ebd70b064199b1cb77ca4c2c9328b673187717d2f"} Oct 31 13:16:27.586165 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.586013 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48" Oct 31 13:16:27.587809 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.587784 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6" event={"ID":"91a9e9c0-9683-4456-803a-ce6e233e2c31","Type":"ContainerStarted","Data":"4f53ea43aa8f691c12978582690aa85f47fd801967fd21ea9502d109f75ef6c9"} Oct 31 13:16:27.587926 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.587830 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6" event={"ID":"91a9e9c0-9683-4456-803a-ce6e233e2c31","Type":"ContainerStarted","Data":"56dbaf6d7c6caea74ca1d7e74eb97372053d0496e5819fba0c7d8b06610ca43b"} Oct 31 13:16:27.587926 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.587850 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6" Oct 31 13:16:27.609274 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.608503 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg" Oct 31 13:16:27.609444 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.609040 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48" podStartSLOduration=1.609027748 podStartE2EDuration="1.609027748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:16:27.606666611 +0000 UTC m=+2638.314252907" watchObservedRunningTime="2026-10-31 13:16:27.609027748 +0000 UTC m=+2638.316614035" Oct 31 13:16:27.632373 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.632307 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6" podStartSLOduration=2.632288181 podStartE2EDuration="2.632288181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:16:27.630939931 +0000 UTC m=+2638.338526228" watchObservedRunningTime="2026-10-31 13:16:27.632288181 +0000 UTC m=+2638.339874477" Oct 31 13:16:27.666880 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:16:27.658569 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-54a16b/agc-cred-b26c7c3f28af: secret "agc-cred-b26c7c3f28af" not found Oct 31 13:16:27.666880 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:16:27.658598 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6: secret "agc-cred-b26c7c3f28af" not found Oct 31 13:16:27.666880 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:16:27.658756 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91a9e9c0-9683-4456-803a-ce6e233e2c31-provider-credentials podName:91a9e9c0-9683-4456-803a-ce6e233e2c31 nodeName:}" failed. No retries permitted until 2026-10-31 13:16:28.158674575 +0000 UTC m=+2638.866260870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/91a9e9c0-9683-4456-803a-ce6e233e2c31-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6" (UID: "91a9e9c0-9683-4456-803a-ce6e233e2c31") : secret "agc-cred-b26c7c3f28af" not found Oct 31 13:16:27.696811 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.696769 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt"] Oct 31 13:16:27.704979 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.704250 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt"] Oct 31 13:16:27.704979 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.704387 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt" Oct 31 13:16:27.753350 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.753317 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-15a6a2\"/\"agc-cred-733a4fb02d51\"" Oct 31 13:16:27.753350 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.753402 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-15a6a2\"/\"payload-processing-external-model-plugins-e2e-extmod-b-15a6a2\"" Oct 31 13:16:27.753695 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.753525 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-15a6a2\"/\"openshift-service-ca.crt\"" Oct 31 13:16:27.753695 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.753632 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-15a6a2\"/\"agc-cred-49713eea7cf3\"" Oct 31 13:16:27.753806 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.753716 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-15a6a2\"/\"payload-processing-external-model-e2e--7a720dc0-dockercfg-bsn8h\"" Oct 31 13:16:27.753806 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.753748 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-15a6a2\"/\"kube-root-ca.crt\"" Oct 31 13:16:27.753977 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.753812 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-15a6a2\"/\"agc-cred-67eefb55f3c6\"" Oct 31 13:16:27.753977 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.753914 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-15a6a2\"/\"routing-overlay\"" Oct 31 13:16:27.863424 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.863192 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t27pt\" (UniqueName: \"kubernetes.io/projected/4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e-kube-api-access-t27pt\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt\" (UID: \"4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt" Oct 31 13:16:27.863424 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.863295 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt\" (UID: \"4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt" Oct 31 13:16:27.863717 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.863681 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt\" (UID: \"4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt" Oct 31 13:16:27.863798 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.863755 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt\" (UID: \"4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt" Oct 31 13:16:27.925300 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.923964 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6"] Oct 31 13:16:27.959500 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.959469 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg"] Oct 31 13:16:27.972257 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.972222 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt\" (UID: \"4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt" Oct 31 13:16:27.972424 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.972296 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt\" (UID: \"4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt" Oct 31 13:16:27.972564 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.972436 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t27pt\" (UniqueName: \"kubernetes.io/projected/4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e-kube-api-access-t27pt\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt\" (UID: \"4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt" Oct 31 13:16:27.972564 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.972463 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt\" (UID: \"4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt" Oct 31 13:16:27.973323 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.973298 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt\" (UID: \"4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt" Oct 31 13:16:27.974133 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.974109 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt\" (UID: \"4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt" Oct 31 13:16:27.980808 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.979341 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt\" (UID: \"4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt" Oct 31 13:16:27.989931 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:27.989909 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t27pt\" (UniqueName: \"kubernetes.io/projected/4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e-kube-api-access-t27pt\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt\" (UID: \"4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt" Oct 31 13:16:28.188665 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:16:28.184255 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-54a16b/agc-cred-b26c7c3f28af: secret "agc-cred-b26c7c3f28af" not found Oct 31 13:16:28.188665 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:16:28.184289 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6: secret "agc-cred-b26c7c3f28af" not found Oct 31 13:16:28.188665 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:16:28.184352 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91a9e9c0-9683-4456-803a-ce6e233e2c31-provider-credentials podName:91a9e9c0-9683-4456-803a-ce6e233e2c31 nodeName:}" failed. No retries permitted until 2026-10-31 13:16:29.184334135 +0000 UTC m=+2639.891920428 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/91a9e9c0-9683-4456-803a-ce6e233e2c31-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6" (UID: "91a9e9c0-9683-4456-803a-ce6e233e2c31") : secret "agc-cred-b26c7c3f28af" not found Oct 31 13:16:28.189288 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:28.188911 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt" Oct 31 13:16:28.598401 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:28.598304 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg" event={"ID":"19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d","Type":"ContainerStarted","Data":"37d7880daae9d87360fff3481cf9e7581bbb9ed703315dd481d4fe15b2b333ba"} Oct 31 13:16:28.598401 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:28.598359 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg" event={"ID":"19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d","Type":"ContainerStarted","Data":"863ed64fa5085ee59a8a29140a42866306578a0a91056d2d88ade944209aec41"} Oct 31 13:16:28.598625 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:28.598440 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg" Oct 31 13:16:28.598710 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:28.598661 5693 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6" secret="" err="secret \"payload-processing-external-model-e2e--427d574a-dockercfg-d5cg8\" not found" Oct 31 13:16:28.611957 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:28.611930 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt"] Oct 31 13:16:28.612712 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:16:28.612690 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d6f3ba8_fc0b_41ba_9e8e_9c67724dc17e.slice/crio-db5ec724ec3e7a6f4ee97c2eb611ecd2529b4ec1ca2f534907bc27da45958305 WatchSource:0}: Error finding container db5ec724ec3e7a6f4ee97c2eb611ecd2529b4ec1ca2f534907bc27da45958305: Status 404 returned error can't find the container with id db5ec724ec3e7a6f4ee97c2eb611ecd2529b4ec1ca2f534907bc27da45958305 Oct 31 13:16:28.625346 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:28.625302 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg" podStartSLOduration=1.625289273 podStartE2EDuration="1.625289273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:16:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:16:28.622540766 +0000 UTC m=+2639.330127063" watchObservedRunningTime="2026-10-31 13:16:28.625289273 +0000 UTC m=+2639.332875568" Oct 31 13:16:28.716553 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:16:28.716518 5693 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-plugins-e2e-extmod-a-54a16b: configmap "payload-processing-external-model-plugins-e2e-extmod-a-54a16b" not found Oct 31 13:16:28.716846 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:16:28.716606 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/91a9e9c0-9683-4456-803a-ce6e233e2c31-plugins-config-volume podName:91a9e9c0-9683-4456-803a-ce6e233e2c31 nodeName:}" failed. No retries permitted until 2026-10-31 13:16:29.216585866 +0000 UTC m=+2639.924172176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/91a9e9c0-9683-4456-803a-ce6e233e2c31-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6" (UID: "91a9e9c0-9683-4456-803a-ce6e233e2c31") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-54a16b" not found Oct 31 13:16:29.225084 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:16:29.225046 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-54a16b/agc-cred-b26c7c3f28af: secret "agc-cred-b26c7c3f28af" not found Oct 31 13:16:29.225084 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:16:29.225077 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6: secret "agc-cred-b26c7c3f28af" not found Oct 31 13:16:29.225629 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:16:29.225137 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91a9e9c0-9683-4456-803a-ce6e233e2c31-provider-credentials podName:91a9e9c0-9683-4456-803a-ce6e233e2c31 nodeName:}" failed. No retries permitted until 2026-10-31 13:16:31.225122215 +0000 UTC m=+2641.932708504 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/91a9e9c0-9683-4456-803a-ce6e233e2c31-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6" (UID: "91a9e9c0-9683-4456-803a-ce6e233e2c31") : secret "agc-cred-b26c7c3f28af" not found Oct 31 13:16:29.225629 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:16:29.225175 5693 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-plugins-e2e-extmod-a-54a16b: configmap "payload-processing-external-model-plugins-e2e-extmod-a-54a16b" not found Oct 31 13:16:29.225629 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:16:29.225242 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/91a9e9c0-9683-4456-803a-ce6e233e2c31-plugins-config-volume podName:91a9e9c0-9683-4456-803a-ce6e233e2c31 nodeName:}" failed. No retries permitted until 2026-10-31 13:16:30.225224008 +0000 UTC m=+2640.932810298 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/91a9e9c0-9683-4456-803a-ce6e233e2c31-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6" (UID: "91a9e9c0-9683-4456-803a-ce6e233e2c31") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-54a16b" not found Oct 31 13:16:29.608738 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:29.608609 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt" event={"ID":"4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e","Type":"ContainerStarted","Data":"3d0aa9576f987ba3bbc65eb4e5f06208f8c1da49b96b58245b6c2205fde28d40"} Oct 31 13:16:29.608738 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:29.608703 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt" event={"ID":"4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e","Type":"ContainerStarted","Data":"db5ec724ec3e7a6f4ee97c2eb611ecd2529b4ec1ca2f534907bc27da45958305"} Oct 31 13:16:29.608738 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:29.608738 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt" Oct 31 13:16:29.634854 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:29.634798 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt" podStartSLOduration=2.634784482 podStartE2EDuration="2.634784482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:16:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:16:29.629828863 +0000 UTC m=+2640.337415159" watchObservedRunningTime="2026-10-31 13:16:29.634784482 +0000 UTC m=+2640.342370777" Oct 31 13:16:30.244633 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:16:30.244588 5693 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-plugins-e2e-extmod-a-54a16b: configmap "payload-processing-external-model-plugins-e2e-extmod-a-54a16b" not found Oct 31 13:16:30.245060 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:16:30.244697 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/91a9e9c0-9683-4456-803a-ce6e233e2c31-plugins-config-volume podName:91a9e9c0-9683-4456-803a-ce6e233e2c31 nodeName:}" failed. No retries permitted until 2026-10-31 13:16:32.244678526 +0000 UTC m=+2642.952264822 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/91a9e9c0-9683-4456-803a-ce6e233e2c31-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6" (UID: "91a9e9c0-9683-4456-803a-ce6e233e2c31") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-54a16b" not found Oct 31 13:16:31.273068 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:16:31.273031 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-54a16b/agc-cred-b26c7c3f28af: secret "agc-cred-b26c7c3f28af" not found Oct 31 13:16:31.273068 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:16:31.273066 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6: secret "agc-cred-b26c7c3f28af" not found Oct 31 13:16:31.273658 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:16:31.273123 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91a9e9c0-9683-4456-803a-ce6e233e2c31-provider-credentials podName:91a9e9c0-9683-4456-803a-ce6e233e2c31 nodeName:}" failed. No retries permitted until 2026-10-31 13:16:35.273107438 +0000 UTC m=+2645.980693715 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/91a9e9c0-9683-4456-803a-ce6e233e2c31-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6" (UID: "91a9e9c0-9683-4456-803a-ce6e233e2c31") : secret "agc-cred-b26c7c3f28af" not found Oct 31 13:16:31.446180 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:31.446150 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5a9e45-698ff58799-ztzw8" Oct 31 13:16:31.446384 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:31.446229 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-5a9e45-7b647fbff9-6xcpd" Oct 31 13:16:32.292628 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:16:32.292596 5693 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-plugins-e2e-extmod-a-54a16b: configmap "payload-processing-external-model-plugins-e2e-extmod-a-54a16b" not found Oct 31 13:16:32.293096 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:16:32.292688 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/91a9e9c0-9683-4456-803a-ce6e233e2c31-plugins-config-volume podName:91a9e9c0-9683-4456-803a-ce6e233e2c31 nodeName:}" failed. No retries permitted until 2026-10-31 13:16:36.292667979 +0000 UTC m=+2647.000254252 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/91a9e9c0-9683-4456-803a-ce6e233e2c31-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6" (UID: "91a9e9c0-9683-4456-803a-ce6e233e2c31") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-54a16b" not found Oct 31 13:16:34.609244 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:34.609200 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6" podUID="91a9e9c0-9683-4456-803a-ce6e233e2c31" containerName="payload-processing" containerID="cri-o://4f53ea43aa8f691c12978582690aa85f47fd801967fd21ea9502d109f75ef6c9" gracePeriod=25 Oct 31 13:16:34.940430 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:34.940401 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6" Oct 31 13:16:35.041528 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:35.041494 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91a9e9c0-9683-4456-803a-ce6e233e2c31-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91a9e9c0-9683-4456-803a-ce6e233e2c31-plugins-config-volume\") pod \"91a9e9c0-9683-4456-803a-ce6e233e2c31\" (UID: \"91a9e9c0-9683-4456-803a-ce6e233e2c31\") " Oct 31 13:16:35.041749 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:35.041553 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91a9e9c0-9683-4456-803a-ce6e233e2c31-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91a9e9c0-9683-4456-803a-ce6e233e2c31-routing-overlay-volume\") pod \"91a9e9c0-9683-4456-803a-ce6e233e2c31\" (UID: \"91a9e9c0-9683-4456-803a-ce6e233e2c31\") " Oct 31 13:16:35.041749 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:35.041742 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91a9e9c0-9683-4456-803a-ce6e233e2c31-provider-credentials\" (UniqueName: \"kubernetes.io/projected/91a9e9c0-9683-4456-803a-ce6e233e2c31-provider-credentials\") pod \"91a9e9c0-9683-4456-803a-ce6e233e2c31\" (UID: \"91a9e9c0-9683-4456-803a-ce6e233e2c31\") " Oct 31 13:16:35.041870 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:35.041771 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91a9e9c0-9683-4456-803a-ce6e233e2c31-kube-api-access-k64gt\" (UniqueName: \"kubernetes.io/projected/91a9e9c0-9683-4456-803a-ce6e233e2c31-kube-api-access-k64gt\") pod \"91a9e9c0-9683-4456-803a-ce6e233e2c31\" (UID: \"91a9e9c0-9683-4456-803a-ce6e233e2c31\") " Oct 31 13:16:35.041937 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:35.041904 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91a9e9c0-9683-4456-803a-ce6e233e2c31-plugins-config-volume" pod "91a9e9c0-9683-4456-803a-ce6e233e2c31" (UID: "91a9e9c0-9683-4456-803a-ce6e233e2c31"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:16:35.041937 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:35.041910 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91a9e9c0-9683-4456-803a-ce6e233e2c31-routing-overlay-volume" pod "91a9e9c0-9683-4456-803a-ce6e233e2c31" (UID: "91a9e9c0-9683-4456-803a-ce6e233e2c31"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:16:35.042704 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:35.042506 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91a9e9c0-9683-4456-803a-ce6e233e2c31-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:35.042704 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:35.042534 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91a9e9c0-9683-4456-803a-ce6e233e2c31-routing-overlay-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:35.044341 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:35.044313 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91a9e9c0-9683-4456-803a-ce6e233e2c31-kube-api-access-k64gt" pod "91a9e9c0-9683-4456-803a-ce6e233e2c31" (UID: "91a9e9c0-9683-4456-803a-ce6e233e2c31"). InnerVolumeSpecName "kube-api-access-k64gt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:35.044460 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:35.044353 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91a9e9c0-9683-4456-803a-ce6e233e2c31-provider-credentials" pod "91a9e9c0-9683-4456-803a-ce6e233e2c31" (UID: "91a9e9c0-9683-4456-803a-ce6e233e2c31"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:35.144263 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:35.144227 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/91a9e9c0-9683-4456-803a-ce6e233e2c31-provider-credentials\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:35.144263 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:35.144256 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k64gt\" (UniqueName: \"kubernetes.io/projected/91a9e9c0-9683-4456-803a-ce6e233e2c31-kube-api-access-k64gt\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:35.694822 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:35.694738 5693 generic.go:356] "Generic (PLEG): container finished" podID="91a9e9c0-9683-4456-803a-ce6e233e2c31" containerID="4f53ea43aa8f691c12978582690aa85f47fd801967fd21ea9502d109f75ef6c9" exitCode=0 Oct 31 13:16:35.694822 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:35.694794 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6" Oct 31 13:16:35.695263 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:35.694825 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6" event={"ID":"91a9e9c0-9683-4456-803a-ce6e233e2c31","Type":"ContainerDied","Data":"4f53ea43aa8f691c12978582690aa85f47fd801967fd21ea9502d109f75ef6c9"} Oct 31 13:16:35.695263 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:35.694886 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6" event={"ID":"91a9e9c0-9683-4456-803a-ce6e233e2c31","Type":"ContainerDied","Data":"56dbaf6d7c6caea74ca1d7e74eb97372053d0496e5819fba0c7d8b06610ca43b"} Oct 31 13:16:35.695263 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:35.694909 5693 scope.go:122] "RemoveContainer" containerID="4f53ea43aa8f691c12978582690aa85f47fd801967fd21ea9502d109f75ef6c9" Oct 31 13:16:35.706343 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:35.706321 5693 scope.go:122] "RemoveContainer" containerID="4f53ea43aa8f691c12978582690aa85f47fd801967fd21ea9502d109f75ef6c9" Oct 31 13:16:35.706621 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:16:35.706600 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f53ea43aa8f691c12978582690aa85f47fd801967fd21ea9502d109f75ef6c9\": container with ID starting with 4f53ea43aa8f691c12978582690aa85f47fd801967fd21ea9502d109f75ef6c9 not found: ID does not exist" containerID="4f53ea43aa8f691c12978582690aa85f47fd801967fd21ea9502d109f75ef6c9" Oct 31 13:16:35.706715 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:35.706632 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f53ea43aa8f691c12978582690aa85f47fd801967fd21ea9502d109f75ef6c9"} err="failed to get container status \"4f53ea43aa8f691c12978582690aa85f47fd801967fd21ea9502d109f75ef6c9\": rpc error: code = NotFound desc = could not find container \"4f53ea43aa8f691c12978582690aa85f47fd801967fd21ea9502d109f75ef6c9\": container with ID starting with 4f53ea43aa8f691c12978582690aa85f47fd801967fd21ea9502d109f75ef6c9 not found: ID does not exist" Oct 31 13:16:35.716584 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:35.716557 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6"] Oct 31 13:16:35.721059 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:35.721035 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-cb45k26h6"] Oct 31 13:16:37.411787 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:37.411749 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91a9e9c0-9683-4456-803a-ce6e233e2c31" path="/var/lib/kubelet/pods/91a9e9c0-9683-4456-803a-ce6e233e2c31/volumes" Oct 31 13:16:38.602701 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:38.602674 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48" Oct 31 13:16:39.612436 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:39.612411 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg" Oct 31 13:16:39.672669 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:39.670616 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j"] Oct 31 13:16:40.620403 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:40.620377 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt" Oct 31 13:16:44.671955 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:44.671911 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j" podUID="8d0e4939-0623-427a-9592-d438b52a2a5d" containerName="payload-processing" containerID="cri-o://c1c92b854d08e54860182362065613d7379e9dd31a5d8a58244497c6060d7bcd" gracePeriod=25 Oct 31 13:16:45.042076 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:45.042050 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j" Oct 31 13:16:45.137267 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:45.137216 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d0e4939-0623-427a-9592-d438b52a2a5d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d0e4939-0623-427a-9592-d438b52a2a5d-plugins-config-volume\") pod \"8d0e4939-0623-427a-9592-d438b52a2a5d\" (UID: \"8d0e4939-0623-427a-9592-d438b52a2a5d\") " Oct 31 13:16:45.137480 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:45.137392 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d0e4939-0623-427a-9592-d438b52a2a5d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8d0e4939-0623-427a-9592-d438b52a2a5d-provider-credentials\") pod \"8d0e4939-0623-427a-9592-d438b52a2a5d\" (UID: \"8d0e4939-0623-427a-9592-d438b52a2a5d\") " Oct 31 13:16:45.137558 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:45.137536 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d0e4939-0623-427a-9592-d438b52a2a5d-kube-api-access-mvvh4\" (UniqueName: \"kubernetes.io/projected/8d0e4939-0623-427a-9592-d438b52a2a5d-kube-api-access-mvvh4\") pod \"8d0e4939-0623-427a-9592-d438b52a2a5d\" (UID: \"8d0e4939-0623-427a-9592-d438b52a2a5d\") " Oct 31 13:16:45.137618 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:45.137590 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d0e4939-0623-427a-9592-d438b52a2a5d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8d0e4939-0623-427a-9592-d438b52a2a5d-routing-overlay-volume\") pod \"8d0e4939-0623-427a-9592-d438b52a2a5d\" (UID: \"8d0e4939-0623-427a-9592-d438b52a2a5d\") " Oct 31 13:16:45.139019 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:45.138745 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d0e4939-0623-427a-9592-d438b52a2a5d-routing-overlay-volume" pod "8d0e4939-0623-427a-9592-d438b52a2a5d" (UID: "8d0e4939-0623-427a-9592-d438b52a2a5d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:16:45.139630 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:45.139330 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d0e4939-0623-427a-9592-d438b52a2a5d-plugins-config-volume" pod "8d0e4939-0623-427a-9592-d438b52a2a5d" (UID: "8d0e4939-0623-427a-9592-d438b52a2a5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:16:45.162898 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:45.162859 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d0e4939-0623-427a-9592-d438b52a2a5d-provider-credentials" pod "8d0e4939-0623-427a-9592-d438b52a2a5d" (UID: "8d0e4939-0623-427a-9592-d438b52a2a5d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:45.163069 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:45.162913 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d0e4939-0623-427a-9592-d438b52a2a5d-kube-api-access-mvvh4" pod "8d0e4939-0623-427a-9592-d438b52a2a5d" (UID: "8d0e4939-0623-427a-9592-d438b52a2a5d"). InnerVolumeSpecName "kube-api-access-mvvh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:45.240014 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:45.239981 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvvh4\" (UniqueName: \"kubernetes.io/projected/8d0e4939-0623-427a-9592-d438b52a2a5d-kube-api-access-mvvh4\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:45.240014 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:45.240011 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8d0e4939-0623-427a-9592-d438b52a2a5d-routing-overlay-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:45.240202 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:45.240029 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d0e4939-0623-427a-9592-d438b52a2a5d-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:45.240202 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:45.240041 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8d0e4939-0623-427a-9592-d438b52a2a5d-provider-credentials\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:45.852398 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:45.852282 5693 generic.go:356] "Generic (PLEG): container finished" podID="8d0e4939-0623-427a-9592-d438b52a2a5d" containerID="c1c92b854d08e54860182362065613d7379e9dd31a5d8a58244497c6060d7bcd" exitCode=0 Oct 31 13:16:45.853049 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:45.852431 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j" event={"ID":"8d0e4939-0623-427a-9592-d438b52a2a5d","Type":"ContainerDied","Data":"c1c92b854d08e54860182362065613d7379e9dd31a5d8a58244497c6060d7bcd"} Oct 31 13:16:45.853049 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:45.852520 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j" event={"ID":"8d0e4939-0623-427a-9592-d438b52a2a5d","Type":"ContainerDied","Data":"b19f7576080a25abbd3f614bce854cec069b8483367206e959fb74c6c0a91e21"} Oct 31 13:16:45.853049 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:45.852556 5693 scope.go:122] "RemoveContainer" containerID="c1c92b854d08e54860182362065613d7379e9dd31a5d8a58244497c6060d7bcd" Oct 31 13:16:45.853049 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:45.852771 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j" Oct 31 13:16:45.904764 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:45.904731 5693 scope.go:122] "RemoveContainer" containerID="c1c92b854d08e54860182362065613d7379e9dd31a5d8a58244497c6060d7bcd" Oct 31 13:16:45.905268 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:16:45.905238 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1c92b854d08e54860182362065613d7379e9dd31a5d8a58244497c6060d7bcd\": container with ID starting with c1c92b854d08e54860182362065613d7379e9dd31a5d8a58244497c6060d7bcd not found: ID does not exist" containerID="c1c92b854d08e54860182362065613d7379e9dd31a5d8a58244497c6060d7bcd" Oct 31 13:16:45.905386 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:45.905276 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1c92b854d08e54860182362065613d7379e9dd31a5d8a58244497c6060d7bcd"} err="failed to get container status \"c1c92b854d08e54860182362065613d7379e9dd31a5d8a58244497c6060d7bcd\": rpc error: code = NotFound desc = could not find container \"c1c92b854d08e54860182362065613d7379e9dd31a5d8a58244497c6060d7bcd\": container with ID starting with c1c92b854d08e54860182362065613d7379e9dd31a5d8a58244497c6060d7bcd not found: ID does not exist" Oct 31 13:16:45.905466 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:45.905444 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j"] Oct 31 13:16:45.911920 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:45.911890 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68jf87j"] Oct 31 13:16:47.410983 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:16:47.410953 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d0e4939-0623-427a-9592-d438b52a2a5d" path="/var/lib/kubelet/pods/8d0e4939-0623-427a-9592-d438b52a2a5d/volumes" Oct 31 13:16:48.734624 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:16:48.734572 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:17:00.605229 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:00.605162 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl"] Oct 31 13:17:00.606588 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:00.606553 5693 cpu_manager.go:414] "containerMap: removing container" podUID="8d0e4939-0623-427a-9592-d438b52a2a5d" containerName="payload-processing" Oct 31 13:17:00.606588 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:00.606583 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d0e4939-0623-427a-9592-d438b52a2a5d" containerName="payload-processing" Oct 31 13:17:00.606828 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:00.606629 5693 cpu_manager.go:414] "containerMap: removing container" podUID="91a9e9c0-9683-4456-803a-ce6e233e2c31" containerName="payload-processing" Oct 31 13:17:00.606828 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:00.606657 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91a9e9c0-9683-4456-803a-ce6e233e2c31" containerName="payload-processing" Oct 31 13:17:00.607001 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:00.606887 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d0e4939-0623-427a-9592-d438b52a2a5d" containerName="payload-processing" Oct 31 13:17:00.607086 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:00.607026 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="91a9e9c0-9683-4456-803a-ce6e233e2c31" containerName="payload-processing" Oct 31 13:17:00.631948 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:00.631914 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl"] Oct 31 13:17:00.632276 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:00.632253 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl" Oct 31 13:17:00.647433 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:00.645601 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-5dc570\"/\"payload-processing-external-model-e2e--ef24cfc7-dockercfg-8djwq\"" Oct 31 13:17:00.647433 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:00.646454 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5dc570\"/\"routing-overlay\"" Oct 31 13:17:00.647433 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:00.646545 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5dc570\"/\"openshift-service-ca.crt\"" Oct 31 13:17:00.647433 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:00.646669 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5dc570\"/\"kube-root-ca.crt\"" Oct 31 13:17:00.647433 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:00.646731 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5dc570\"/\"payload-processing-external-model-plugins-e2e-extmod-a-5dc570\"" Oct 31 13:17:00.647433 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:00.646813 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-5dc570\"/\"e2e-cmp-legacy-69bfde-provider-cred\"" Oct 31 13:17:00.670400 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:00.670362 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f674e1e-7ff4-4923-b32a-fab0fa51d390-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl\" (UID: \"2f674e1e-7ff4-4923-b32a-fab0fa51d390\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl" Oct 31 13:17:00.670848 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:00.670822 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f674e1e-7ff4-4923-b32a-fab0fa51d390-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl\" (UID: \"2f674e1e-7ff4-4923-b32a-fab0fa51d390\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl" Oct 31 13:17:00.671129 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:00.671111 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f674e1e-7ff4-4923-b32a-fab0fa51d390-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl\" (UID: \"2f674e1e-7ff4-4923-b32a-fab0fa51d390\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl" Oct 31 13:17:00.671305 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:00.671290 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56cw4\" (UniqueName: \"kubernetes.io/projected/2f674e1e-7ff4-4923-b32a-fab0fa51d390-kube-api-access-56cw4\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl\" (UID: \"2f674e1e-7ff4-4923-b32a-fab0fa51d390\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl" Oct 31 13:17:00.791042 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:00.781191 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f674e1e-7ff4-4923-b32a-fab0fa51d390-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl\" (UID: \"2f674e1e-7ff4-4923-b32a-fab0fa51d390\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl" Oct 31 13:17:00.869113 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:00.854937 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f674e1e-7ff4-4923-b32a-fab0fa51d390-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl\" (UID: \"2f674e1e-7ff4-4923-b32a-fab0fa51d390\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl" Oct 31 13:17:00.869113 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:00.855363 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f674e1e-7ff4-4923-b32a-fab0fa51d390-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl\" (UID: \"2f674e1e-7ff4-4923-b32a-fab0fa51d390\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl" Oct 31 13:17:00.869113 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:00.855456 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f674e1e-7ff4-4923-b32a-fab0fa51d390-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl\" (UID: \"2f674e1e-7ff4-4923-b32a-fab0fa51d390\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl" Oct 31 13:17:00.869113 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:00.855569 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56cw4\" (UniqueName: \"kubernetes.io/projected/2f674e1e-7ff4-4923-b32a-fab0fa51d390-kube-api-access-56cw4\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl\" (UID: \"2f674e1e-7ff4-4923-b32a-fab0fa51d390\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl" Oct 31 13:17:00.869113 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:00.856289 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f674e1e-7ff4-4923-b32a-fab0fa51d390-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl\" (UID: \"2f674e1e-7ff4-4923-b32a-fab0fa51d390\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl" Oct 31 13:17:00.919393 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:00.919360 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f674e1e-7ff4-4923-b32a-fab0fa51d390-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl\" (UID: \"2f674e1e-7ff4-4923-b32a-fab0fa51d390\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl" Oct 31 13:17:00.932704 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:00.927733 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56cw4\" (UniqueName: \"kubernetes.io/projected/2f674e1e-7ff4-4923-b32a-fab0fa51d390-kube-api-access-56cw4\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl\" (UID: \"2f674e1e-7ff4-4923-b32a-fab0fa51d390\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl" Oct 31 13:17:01.014655 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:01.014595 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl" Oct 31 13:17:01.623686 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:01.623447 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl"] Oct 31 13:17:02.017536 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:02.017501 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl" event={"ID":"2f674e1e-7ff4-4923-b32a-fab0fa51d390","Type":"ContainerStarted","Data":"303bd09edb31c0cc907deb55bfbf5d1e3ca06c3ba6d8afddf6c482986723506c"} Oct 31 13:17:02.017751 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:02.017568 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl" Oct 31 13:17:02.017751 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:02.017591 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl" event={"ID":"2f674e1e-7ff4-4923-b32a-fab0fa51d390","Type":"ContainerStarted","Data":"96d4c2eefb32c1b8b33eb91290457c5bd218e07324a3206d25661725ff286ffa"} Oct 31 13:17:02.040945 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:02.040884 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl" podStartSLOduration=2.040865645 podStartE2EDuration="2.040865645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:17:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:17:02.040738747 +0000 UTC m=+2672.748325042" watchObservedRunningTime="2026-10-31 13:17:02.040865645 +0000 UTC m=+2672.748451941" Oct 31 13:17:02.244948 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:02.244909 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg"] Oct 31 13:17:04.534321 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:04.534284 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2"] Oct 31 13:17:04.539424 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:04.539402 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" Oct 31 13:17:04.546030 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:04.546011 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-29e3a0-openshift-default-dockercfg-4zmv8\"" Oct 31 13:17:04.554187 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:04.554160 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2"] Oct 31 13:17:04.611223 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:04.611176 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af589ec1-de60-43ff-9bbb-0b298d9c99cc-workload-socket\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2\" (UID: \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" Oct 31 13:17:04.611404 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:04.611241 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af589ec1-de60-43ff-9bbb-0b298d9c99cc-workload-certs\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2\" (UID: \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" Oct 31 13:17:04.611448 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:04.611401 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af589ec1-de60-43ff-9bbb-0b298d9c99cc-istiod-ca-cert\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2\" (UID: \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" Oct 31 13:17:04.611540 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:04.611519 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx6v5\" (UniqueName: \"kubernetes.io/projected/af589ec1-de60-43ff-9bbb-0b298d9c99cc-kube-api-access-lx6v5\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2\" (UID: \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" Oct 31 13:17:04.611584 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:04.611559 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af589ec1-de60-43ff-9bbb-0b298d9c99cc-istio-podinfo\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2\" (UID: \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" Oct 31 13:17:04.611620 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:04.611582 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af589ec1-de60-43ff-9bbb-0b298d9c99cc-istio-token\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2\" (UID: \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" Oct 31 13:17:04.611766 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:04.611751 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af589ec1-de60-43ff-9bbb-0b298d9c99cc-istio-envoy\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2\" (UID: \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" Oct 31 13:17:04.611837 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:04.611802 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af589ec1-de60-43ff-9bbb-0b298d9c99cc-istio-data\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2\" (UID: \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" Oct 31 13:17:04.611837 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:04.611824 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af589ec1-de60-43ff-9bbb-0b298d9c99cc-credential-socket\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2\" (UID: \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" Oct 31 13:17:04.713747 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:04.713699 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af589ec1-de60-43ff-9bbb-0b298d9c99cc-istio-envoy\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2\" (UID: \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" Oct 31 13:17:04.713941 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:04.713788 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af589ec1-de60-43ff-9bbb-0b298d9c99cc-istio-data\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2\" (UID: \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" Oct 31 13:17:04.713941 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:04.713831 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af589ec1-de60-43ff-9bbb-0b298d9c99cc-credential-socket\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2\" (UID: \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" Oct 31 13:17:04.714021 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:04.713975 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af589ec1-de60-43ff-9bbb-0b298d9c99cc-workload-socket\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2\" (UID: \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" Oct 31 13:17:04.714062 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:04.714027 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af589ec1-de60-43ff-9bbb-0b298d9c99cc-workload-certs\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2\" (UID: \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" Oct 31 13:17:04.714109 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:04.714094 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af589ec1-de60-43ff-9bbb-0b298d9c99cc-istiod-ca-cert\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2\" (UID: \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" Oct 31 13:17:04.714195 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:04.714174 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx6v5\" (UniqueName: \"kubernetes.io/projected/af589ec1-de60-43ff-9bbb-0b298d9c99cc-kube-api-access-lx6v5\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2\" (UID: \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" Oct 31 13:17:04.714392 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:04.714370 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af589ec1-de60-43ff-9bbb-0b298d9c99cc-istio-podinfo\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2\" (UID: \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" Oct 31 13:17:04.714794 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:04.714515 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af589ec1-de60-43ff-9bbb-0b298d9c99cc-istio-token\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2\" (UID: \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" Oct 31 13:17:04.714794 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:04.714558 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af589ec1-de60-43ff-9bbb-0b298d9c99cc-workload-socket\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2\" (UID: \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" Oct 31 13:17:04.714794 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:04.714459 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af589ec1-de60-43ff-9bbb-0b298d9c99cc-credential-socket\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2\" (UID: \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" Oct 31 13:17:04.714794 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:04.714469 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af589ec1-de60-43ff-9bbb-0b298d9c99cc-istio-data\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2\" (UID: \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" Oct 31 13:17:04.715003 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:04.714878 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af589ec1-de60-43ff-9bbb-0b298d9c99cc-workload-certs\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2\" (UID: \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" Oct 31 13:17:04.715374 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:04.715343 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af589ec1-de60-43ff-9bbb-0b298d9c99cc-istiod-ca-cert\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2\" (UID: \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" Oct 31 13:17:04.716833 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:04.716809 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af589ec1-de60-43ff-9bbb-0b298d9c99cc-istio-envoy\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2\" (UID: \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" Oct 31 13:17:04.717287 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:04.717260 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af589ec1-de60-43ff-9bbb-0b298d9c99cc-istio-podinfo\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2\" (UID: \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" Oct 31 13:17:04.729181 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:04.729147 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx6v5\" (UniqueName: \"kubernetes.io/projected/af589ec1-de60-43ff-9bbb-0b298d9c99cc-kube-api-access-lx6v5\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2\" (UID: \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" Oct 31 13:17:04.729310 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:04.729181 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af589ec1-de60-43ff-9bbb-0b298d9c99cc-istio-token\") pod \"e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2\" (UID: \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\") " pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" Oct 31 13:17:04.859151 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:04.859059 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" Oct 31 13:17:05.152843 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:05.152808 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2"] Oct 31 13:17:05.153204 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:17:05.153177 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf589ec1_de60_43ff_9bbb_0b298d9c99cc.slice/crio-3dccf12ce236749964d6d63efcd9a6365161d2fc5d0a56fb215b711e1ef6259c WatchSource:0}: Error finding container 3dccf12ce236749964d6d63efcd9a6365161d2fc5d0a56fb215b711e1ef6259c: Status 404 returned error can't find the container with id 3dccf12ce236749964d6d63efcd9a6365161d2fc5d0a56fb215b711e1ef6259c Oct 31 13:17:05.156707 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:05.156623 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:17:05.156801 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:05.156757 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:17:05.156845 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:05.156799 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:17:06.062864 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:06.062834 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" event={"ID":"af589ec1-de60-43ff-9bbb-0b298d9c99cc","Type":"ContainerStarted","Data":"d93345d85a287d60125356684d0bc61b194ee595c01ec4fa30d4af9f720ab9c5"} Oct 31 13:17:06.063287 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:06.062887 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" event={"ID":"af589ec1-de60-43ff-9bbb-0b298d9c99cc","Type":"ContainerStarted","Data":"3dccf12ce236749964d6d63efcd9a6365161d2fc5d0a56fb215b711e1ef6259c"} Oct 31 13:17:06.104678 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:06.103726 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" podStartSLOduration=2.1037048130000002 podStartE2EDuration="2.103704813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:17:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:17:06.101138168 +0000 UTC m=+2676.808724466" watchObservedRunningTime="2026-10-31 13:17:06.103704813 +0000 UTC m=+2676.811291125" Oct 31 13:17:06.859850 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:06.859769 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" Oct 31 13:17:06.862037 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:06.862008 5693 patch_prober.go:29] interesting pod/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 31 13:17:06.862186 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:06.862070 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" podUID="af589ec1-de60-43ff-9bbb-0b298d9c99cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 31 13:17:07.246213 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:07.246172 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg" podUID="19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d" containerName="payload-processing" containerID="cri-o://37d7880daae9d87360fff3481cf9e7581bbb9ed703315dd481d4fe15b2b333ba" gracePeriod=25 Oct 31 13:17:07.705524 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:07.705492 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg" Oct 31 13:17:07.812211 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:07.812172 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d-provider-credentials\") pod \"19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d\" (UID: \"19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d\") " Oct 31 13:17:07.815144 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:07.815120 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d-routing-overlay-volume\") pod \"19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d\" (UID: \"19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d\") " Oct 31 13:17:07.815302 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:07.815278 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d-kube-api-access-tvzts\" (UniqueName: \"kubernetes.io/projected/19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d-kube-api-access-tvzts\") pod \"19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d\" (UID: \"19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d\") " Oct 31 13:17:07.815397 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:07.815379 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d-plugins-config-volume\") pod \"19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d\" (UID: \"19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d\") " Oct 31 13:17:07.816302 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:07.816271 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d-plugins-config-volume" pod "19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d" (UID: "19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:17:07.817003 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:07.816978 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d-routing-overlay-volume" pod "19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d" (UID: "19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:17:07.817170 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:07.817143 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d-provider-credentials" pod "19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d" (UID: "19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:17:07.820166 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:07.820141 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d-kube-api-access-tvzts" pod "19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d" (UID: "19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d"). InnerVolumeSpecName "kube-api-access-tvzts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:17:07.860127 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:07.860091 5693 patch_prober.go:29] interesting pod/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 31 13:17:07.860332 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:07.860181 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" podUID="af589ec1-de60-43ff-9bbb-0b298d9c99cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 31 13:17:07.918630 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:07.918591 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d-provider-credentials\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:17:07.918630 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:07.918624 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d-routing-overlay-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:17:07.918872 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:07.918634 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvzts\" (UniqueName: \"kubernetes.io/projected/19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d-kube-api-access-tvzts\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:17:07.918872 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:07.918668 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:17:08.108323 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:08.108290 5693 generic.go:356] "Generic (PLEG): container finished" podID="19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d" containerID="37d7880daae9d87360fff3481cf9e7581bbb9ed703315dd481d4fe15b2b333ba" exitCode=0 Oct 31 13:17:08.108505 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:08.108342 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg" Oct 31 13:17:08.108505 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:08.108381 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg" event={"ID":"19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d","Type":"ContainerDied","Data":"37d7880daae9d87360fff3481cf9e7581bbb9ed703315dd481d4fe15b2b333ba"} Oct 31 13:17:08.108505 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:08.108448 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg" event={"ID":"19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d","Type":"ContainerDied","Data":"863ed64fa5085ee59a8a29140a42866306578a0a91056d2d88ade944209aec41"} Oct 31 13:17:08.108505 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:08.108478 5693 scope.go:122] "RemoveContainer" containerID="37d7880daae9d87360fff3481cf9e7581bbb9ed703315dd481d4fe15b2b333ba" Oct 31 13:17:08.121247 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:08.121230 5693 scope.go:122] "RemoveContainer" containerID="37d7880daae9d87360fff3481cf9e7581bbb9ed703315dd481d4fe15b2b333ba" Oct 31 13:17:08.121507 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:17:08.121482 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37d7880daae9d87360fff3481cf9e7581bbb9ed703315dd481d4fe15b2b333ba\": container with ID starting with 37d7880daae9d87360fff3481cf9e7581bbb9ed703315dd481d4fe15b2b333ba not found: ID does not exist" containerID="37d7880daae9d87360fff3481cf9e7581bbb9ed703315dd481d4fe15b2b333ba" Oct 31 13:17:08.121599 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:08.121515 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37d7880daae9d87360fff3481cf9e7581bbb9ed703315dd481d4fe15b2b333ba"} err="failed to get container status \"37d7880daae9d87360fff3481cf9e7581bbb9ed703315dd481d4fe15b2b333ba\": rpc error: code = NotFound desc = could not find container \"37d7880daae9d87360fff3481cf9e7581bbb9ed703315dd481d4fe15b2b333ba\": container with ID starting with 37d7880daae9d87360fff3481cf9e7581bbb9ed703315dd481d4fe15b2b333ba not found: ID does not exist" Oct 31 13:17:08.135441 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:08.135349 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg"] Oct 31 13:17:08.139694 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:08.139670 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-64987c2vg"] Oct 31 13:17:08.859450 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:08.859418 5693 patch_prober.go:29] interesting pod/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 31 13:17:08.859867 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:08.859468 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" podUID="af589ec1-de60-43ff-9bbb-0b298d9c99cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 31 13:17:09.417047 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:09.417006 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d" path="/var/lib/kubelet/pods/19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d/volumes" Oct 31 13:17:09.859619 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:09.859535 5693 patch_prober.go:29] interesting pod/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 31 13:17:09.859619 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:09.859594 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" podUID="af589ec1-de60-43ff-9bbb-0b298d9c99cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 31 13:17:10.859714 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:10.859680 5693 patch_prober.go:29] interesting pod/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 31 13:17:10.860307 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:10.860272 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" podUID="af589ec1-de60-43ff-9bbb-0b298d9c99cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 31 13:17:11.889547 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:11.889513 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" Oct 31 13:17:11.890531 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:11.890506 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" Oct 31 13:17:11.891845 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:11.891800 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" Oct 31 13:17:13.048573 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:13.048543 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl" Oct 31 13:17:15.565494 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:15.565465 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4"] Oct 31 13:17:15.566387 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:15.566371 5693 cpu_manager.go:414] "containerMap: removing container" podUID="19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d" containerName="payload-processing" Oct 31 13:17:15.566431 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:15.566388 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d" containerName="payload-processing" Oct 31 13:17:15.566521 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:15.566513 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="19dbd9d7-5d07-47c9-9dab-4c1450fa8f8d" containerName="payload-processing" Oct 31 13:17:15.570378 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:15.570361 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4" Oct 31 13:17:15.578748 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:15.578710 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-29e3a0\"/\"agc-cred-c4174febc8d5\"" Oct 31 13:17:15.578882 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:15.578776 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-29e3a0\"/\"routing-overlay\"" Oct 31 13:17:15.578882 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:15.578840 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-29e3a0\"/\"kube-root-ca.crt\"" Oct 31 13:17:15.579102 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:15.579079 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-29e3a0\"/\"payload-processing-external-model-plugins-e2e-extmod-b-29e3a0\"" Oct 31 13:17:15.579839 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:15.579791 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-29e3a0\"/\"payload-processing-external-model-e2e--e205175e-dockercfg-ssrqv\"" Oct 31 13:17:15.579974 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:15.579957 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-29e3a0\"/\"openshift-service-ca.crt\"" Oct 31 13:17:15.585404 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:15.585377 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4"] Oct 31 13:17:15.657937 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:15.657897 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcqb7\" (UniqueName: \"kubernetes.io/projected/e6d3b9e2-832d-49f0-8023-8211b4a96124-kube-api-access-qcqb7\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4\" (UID: \"e6d3b9e2-832d-49f0-8023-8211b4a96124\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4" Oct 31 13:17:15.658107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:15.658001 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e6d3b9e2-832d-49f0-8023-8211b4a96124-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4\" (UID: \"e6d3b9e2-832d-49f0-8023-8211b4a96124\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4" Oct 31 13:17:15.658107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:15.658042 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e6d3b9e2-832d-49f0-8023-8211b4a96124-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4\" (UID: \"e6d3b9e2-832d-49f0-8023-8211b4a96124\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4" Oct 31 13:17:15.658107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:15.658103 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6d3b9e2-832d-49f0-8023-8211b4a96124-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4\" (UID: \"e6d3b9e2-832d-49f0-8023-8211b4a96124\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4" Oct 31 13:17:15.759660 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:15.759613 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcqb7\" (UniqueName: \"kubernetes.io/projected/e6d3b9e2-832d-49f0-8023-8211b4a96124-kube-api-access-qcqb7\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4\" (UID: \"e6d3b9e2-832d-49f0-8023-8211b4a96124\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4" Oct 31 13:17:15.759858 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:15.759718 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e6d3b9e2-832d-49f0-8023-8211b4a96124-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4\" (UID: \"e6d3b9e2-832d-49f0-8023-8211b4a96124\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4" Oct 31 13:17:15.759858 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:15.759755 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e6d3b9e2-832d-49f0-8023-8211b4a96124-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4\" (UID: \"e6d3b9e2-832d-49f0-8023-8211b4a96124\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4" Oct 31 13:17:15.759858 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:15.759823 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6d3b9e2-832d-49f0-8023-8211b4a96124-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4\" (UID: \"e6d3b9e2-832d-49f0-8023-8211b4a96124\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4" Oct 31 13:17:15.760481 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:15.760442 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e6d3b9e2-832d-49f0-8023-8211b4a96124-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4\" (UID: \"e6d3b9e2-832d-49f0-8023-8211b4a96124\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4" Oct 31 13:17:15.760994 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:15.760968 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6d3b9e2-832d-49f0-8023-8211b4a96124-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4\" (UID: \"e6d3b9e2-832d-49f0-8023-8211b4a96124\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4" Oct 31 13:17:15.762537 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:15.762519 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e6d3b9e2-832d-49f0-8023-8211b4a96124-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4\" (UID: \"e6d3b9e2-832d-49f0-8023-8211b4a96124\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4" Oct 31 13:17:15.777672 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:15.777625 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcqb7\" (UniqueName: \"kubernetes.io/projected/e6d3b9e2-832d-49f0-8023-8211b4a96124-kube-api-access-qcqb7\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4\" (UID: \"e6d3b9e2-832d-49f0-8023-8211b4a96124\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4" Oct 31 13:17:15.889779 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:15.889633 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4" Oct 31 13:17:16.196331 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:16.196302 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4"] Oct 31 13:17:16.196474 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:17:16.196322 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6d3b9e2_832d_49f0_8023_8211b4a96124.slice/crio-cc6b220933423598460810e584d818abd69bec85191e25cd33c5e7014c5500ac WatchSource:0}: Error finding container cc6b220933423598460810e584d818abd69bec85191e25cd33c5e7014c5500ac: Status 404 returned error can't find the container with id cc6b220933423598460810e584d818abd69bec85191e25cd33c5e7014c5500ac Oct 31 13:17:17.198567 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:17.198513 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4" event={"ID":"e6d3b9e2-832d-49f0-8023-8211b4a96124","Type":"ContainerStarted","Data":"f8b406dbfbc4c28fe817a00ea856622a2d18498a7d0c1de2235d6dd6bd787441"} Oct 31 13:17:17.199080 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:17.198582 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4" event={"ID":"e6d3b9e2-832d-49f0-8023-8211b4a96124","Type":"ContainerStarted","Data":"cc6b220933423598460810e584d818abd69bec85191e25cd33c5e7014c5500ac"} Oct 31 13:17:17.199080 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:17.198689 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4" Oct 31 13:17:17.230862 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:17.230788 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4" podStartSLOduration=2.230774573 podStartE2EDuration="2.230774573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:17:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:17:17.225341591 +0000 UTC m=+2687.932927923" watchObservedRunningTime="2026-10-31 13:17:17.230774573 +0000 UTC m=+2687.938360867" Oct 31 13:17:18.692433 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:17:18.692392 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:17:27.222456 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:27.222422 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4"] Oct 31 13:17:27.225673 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:27.225627 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4" Oct 31 13:17:32.223484 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:32.223433 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4" podUID="e6d3b9e2-832d-49f0-8023-8211b4a96124" containerName="payload-processing" containerID="cri-o://f8b406dbfbc4c28fe817a00ea856622a2d18498a7d0c1de2235d6dd6bd787441" gracePeriod=25 Oct 31 13:17:32.469548 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:32.469513 5693 generic.go:356] "Generic (PLEG): container finished" podID="e6d3b9e2-832d-49f0-8023-8211b4a96124" containerID="f8b406dbfbc4c28fe817a00ea856622a2d18498a7d0c1de2235d6dd6bd787441" exitCode=0 Oct 31 13:17:32.469795 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:32.469725 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4" event={"ID":"e6d3b9e2-832d-49f0-8023-8211b4a96124","Type":"ContainerDied","Data":"f8b406dbfbc4c28fe817a00ea856622a2d18498a7d0c1de2235d6dd6bd787441"} Oct 31 13:17:32.691958 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:32.691933 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4" Oct 31 13:17:32.767736 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:32.767707 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6d3b9e2-832d-49f0-8023-8211b4a96124-kube-api-access-qcqb7\" (UniqueName: \"kubernetes.io/projected/e6d3b9e2-832d-49f0-8023-8211b4a96124-kube-api-access-qcqb7\") pod \"e6d3b9e2-832d-49f0-8023-8211b4a96124\" (UID: \"e6d3b9e2-832d-49f0-8023-8211b4a96124\") " Oct 31 13:17:32.767949 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:32.767796 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6d3b9e2-832d-49f0-8023-8211b4a96124-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6d3b9e2-832d-49f0-8023-8211b4a96124-plugins-config-volume\") pod \"e6d3b9e2-832d-49f0-8023-8211b4a96124\" (UID: \"e6d3b9e2-832d-49f0-8023-8211b4a96124\") " Oct 31 13:17:32.767949 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:32.767881 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6d3b9e2-832d-49f0-8023-8211b4a96124-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e6d3b9e2-832d-49f0-8023-8211b4a96124-routing-overlay-volume\") pod \"e6d3b9e2-832d-49f0-8023-8211b4a96124\" (UID: \"e6d3b9e2-832d-49f0-8023-8211b4a96124\") " Oct 31 13:17:32.768122 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:32.768098 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6d3b9e2-832d-49f0-8023-8211b4a96124-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e6d3b9e2-832d-49f0-8023-8211b4a96124-provider-credentials\") pod \"e6d3b9e2-832d-49f0-8023-8211b4a96124\" (UID: \"e6d3b9e2-832d-49f0-8023-8211b4a96124\") " Oct 31 13:17:32.768256 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:32.768233 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6d3b9e2-832d-49f0-8023-8211b4a96124-routing-overlay-volume" pod "e6d3b9e2-832d-49f0-8023-8211b4a96124" (UID: "e6d3b9e2-832d-49f0-8023-8211b4a96124"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:17:32.768322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:32.768252 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6d3b9e2-832d-49f0-8023-8211b4a96124-plugins-config-volume" pod "e6d3b9e2-832d-49f0-8023-8211b4a96124" (UID: "e6d3b9e2-832d-49f0-8023-8211b4a96124"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:17:32.768855 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:32.768828 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6d3b9e2-832d-49f0-8023-8211b4a96124-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:17:32.768962 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:32.768874 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e6d3b9e2-832d-49f0-8023-8211b4a96124-routing-overlay-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:17:32.770710 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:32.770685 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6d3b9e2-832d-49f0-8023-8211b4a96124-provider-credentials" pod "e6d3b9e2-832d-49f0-8023-8211b4a96124" (UID: "e6d3b9e2-832d-49f0-8023-8211b4a96124"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:17:32.770918 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:32.770895 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6d3b9e2-832d-49f0-8023-8211b4a96124-kube-api-access-qcqb7" pod "e6d3b9e2-832d-49f0-8023-8211b4a96124" (UID: "e6d3b9e2-832d-49f0-8023-8211b4a96124"). InnerVolumeSpecName "kube-api-access-qcqb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:17:32.919437 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:32.919404 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e6d3b9e2-832d-49f0-8023-8211b4a96124-provider-credentials\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:17:32.919779 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:32.919749 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcqb7\" (UniqueName: \"kubernetes.io/projected/e6d3b9e2-832d-49f0-8023-8211b4a96124-kube-api-access-qcqb7\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:17:33.411347 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:33.411316 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg"] Oct 31 13:17:33.412363 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:33.412339 5693 cpu_manager.go:414] "containerMap: removing container" podUID="e6d3b9e2-832d-49f0-8023-8211b4a96124" containerName="payload-processing" Oct 31 13:17:33.412363 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:33.412363 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6d3b9e2-832d-49f0-8023-8211b4a96124" containerName="payload-processing" Oct 31 13:17:33.412598 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:33.412583 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6d3b9e2-832d-49f0-8023-8211b4a96124" containerName="payload-processing" Oct 31 13:17:33.416589 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:33.416568 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg" Oct 31 13:17:33.429756 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:33.429732 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg"] Oct 31 13:17:33.484142 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:33.484105 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4" event={"ID":"e6d3b9e2-832d-49f0-8023-8211b4a96124","Type":"ContainerDied","Data":"cc6b220933423598460810e584d818abd69bec85191e25cd33c5e7014c5500ac"} Oct 31 13:17:33.484142 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:33.484131 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4" Oct 31 13:17:33.484418 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:33.484183 5693 scope.go:122] "RemoveContainer" containerID="f8b406dbfbc4c28fe817a00ea856622a2d18498a7d0c1de2235d6dd6bd787441" Oct 31 13:17:33.517426 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:33.516003 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4"] Oct 31 13:17:33.525337 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:33.525302 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958qhgd4"] Oct 31 13:17:33.539266 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:33.539235 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c26eb386-0d37-4259-9f88-4fb02f7164cd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg\" (UID: \"c26eb386-0d37-4259-9f88-4fb02f7164cd\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg" Oct 31 13:17:33.539421 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:33.539277 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c26eb386-0d37-4259-9f88-4fb02f7164cd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg\" (UID: \"c26eb386-0d37-4259-9f88-4fb02f7164cd\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg" Oct 31 13:17:33.539460 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:33.539444 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c26eb386-0d37-4259-9f88-4fb02f7164cd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg\" (UID: \"c26eb386-0d37-4259-9f88-4fb02f7164cd\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg" Oct 31 13:17:33.539674 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:33.539625 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc6xs\" (UniqueName: \"kubernetes.io/projected/c26eb386-0d37-4259-9f88-4fb02f7164cd-kube-api-access-gc6xs\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg\" (UID: \"c26eb386-0d37-4259-9f88-4fb02f7164cd\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg" Oct 31 13:17:33.641622 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:33.641582 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc6xs\" (UniqueName: \"kubernetes.io/projected/c26eb386-0d37-4259-9f88-4fb02f7164cd-kube-api-access-gc6xs\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg\" (UID: \"c26eb386-0d37-4259-9f88-4fb02f7164cd\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg" Oct 31 13:17:33.641868 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:33.641847 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c26eb386-0d37-4259-9f88-4fb02f7164cd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg\" (UID: \"c26eb386-0d37-4259-9f88-4fb02f7164cd\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg" Oct 31 13:17:33.642004 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:33.641892 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c26eb386-0d37-4259-9f88-4fb02f7164cd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg\" (UID: \"c26eb386-0d37-4259-9f88-4fb02f7164cd\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg" Oct 31 13:17:33.642076 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:33.642014 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c26eb386-0d37-4259-9f88-4fb02f7164cd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg\" (UID: \"c26eb386-0d37-4259-9f88-4fb02f7164cd\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg" Oct 31 13:17:33.642533 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:33.642509 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c26eb386-0d37-4259-9f88-4fb02f7164cd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg\" (UID: \"c26eb386-0d37-4259-9f88-4fb02f7164cd\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg" Oct 31 13:17:33.642689 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:33.642671 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c26eb386-0d37-4259-9f88-4fb02f7164cd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg\" (UID: \"c26eb386-0d37-4259-9f88-4fb02f7164cd\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg" Oct 31 13:17:33.644845 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:33.644823 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c26eb386-0d37-4259-9f88-4fb02f7164cd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg\" (UID: \"c26eb386-0d37-4259-9f88-4fb02f7164cd\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg" Oct 31 13:17:33.655984 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:33.655961 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc6xs\" (UniqueName: \"kubernetes.io/projected/c26eb386-0d37-4259-9f88-4fb02f7164cd-kube-api-access-gc6xs\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg\" (UID: \"c26eb386-0d37-4259-9f88-4fb02f7164cd\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg" Oct 31 13:17:33.741427 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:33.741335 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg" Oct 31 13:17:34.048049 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:34.048023 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg"] Oct 31 13:17:34.048534 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:17:34.048509 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc26eb386_0d37_4259_9f88_4fb02f7164cd.slice/crio-b94c223240971a56efeab447a2669206fc042aad74e45460dc18e859892abd7f WatchSource:0}: Error finding container b94c223240971a56efeab447a2669206fc042aad74e45460dc18e859892abd7f: Status 404 returned error can't find the container with id b94c223240971a56efeab447a2669206fc042aad74e45460dc18e859892abd7f Oct 31 13:17:34.494555 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:34.494517 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg" event={"ID":"c26eb386-0d37-4259-9f88-4fb02f7164cd","Type":"ContainerStarted","Data":"69a4be1a1e6ed6305f387382e481726774ed173d25fc545dc282379bccd68db8"} Oct 31 13:17:34.495004 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:34.494578 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg" event={"ID":"c26eb386-0d37-4259-9f88-4fb02f7164cd","Type":"ContainerStarted","Data":"b94c223240971a56efeab447a2669206fc042aad74e45460dc18e859892abd7f"} Oct 31 13:17:34.495004 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:34.494611 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg" Oct 31 13:17:34.526090 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:34.526033 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg" podStartSLOduration=1.526013234 podStartE2EDuration="1.526013234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:17:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:17:34.524410008 +0000 UTC m=+2705.231996309" watchObservedRunningTime="2026-10-31 13:17:34.526013234 +0000 UTC m=+2705.233599530" Oct 31 13:17:35.411988 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:35.411958 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6d3b9e2-832d-49f0-8023-8211b4a96124" path="/var/lib/kubelet/pods/e6d3b9e2-832d-49f0-8023-8211b4a96124/volumes" Oct 31 13:17:45.547290 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:45.546767 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg" Oct 31 13:17:45.697172 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:45.697135 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt"] Oct 31 13:17:48.695199 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:17:48.695161 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:17:50.698241 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:50.698201 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt" podUID="4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e" containerName="payload-processing" containerID="cri-o://3d0aa9576f987ba3bbc65eb4e5f06208f8c1da49b96b58245b6c2205fde28d40" gracePeriod=25 Oct 31 13:17:51.040232 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:51.040207 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt" Oct 31 13:17:51.081803 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:51.081769 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e-plugins-config-volume\") pod \"4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e\" (UID: \"4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e\") " Oct 31 13:17:51.081999 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:51.081874 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e-kube-api-access-t27pt\" (UniqueName: \"kubernetes.io/projected/4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e-kube-api-access-t27pt\") pod \"4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e\" (UID: \"4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e\") " Oct 31 13:17:51.082077 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:51.082019 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e-provider-credentials\") pod \"4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e\" (UID: \"4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e\") " Oct 31 13:17:51.082077 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:51.082047 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e-routing-overlay-volume\") pod \"4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e\" (UID: \"4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e\") " Oct 31 13:17:51.082970 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:51.082567 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e-routing-overlay-volume" pod "4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e" (UID: "4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:17:51.082970 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:51.082939 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e-routing-overlay-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:17:51.082970 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:51.082963 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e-plugins-config-volume" pod "4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e" (UID: "4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:17:51.085106 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:51.085068 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e-provider-credentials" pod "4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e" (UID: "4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:17:51.087473 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:51.087448 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e-kube-api-access-t27pt" pod "4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e" (UID: "4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e"). InnerVolumeSpecName "kube-api-access-t27pt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:17:51.184981 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:51.184944 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t27pt\" (UniqueName: \"kubernetes.io/projected/4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e-kube-api-access-t27pt\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:17:51.184981 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:51.184980 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e-provider-credentials\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:17:51.185203 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:51.184994 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:17:51.803903 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:51.803770 5693 generic.go:356] "Generic (PLEG): container finished" podID="4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e" containerID="3d0aa9576f987ba3bbc65eb4e5f06208f8c1da49b96b58245b6c2205fde28d40" exitCode=0 Oct 31 13:17:51.803903 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:51.803839 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt" event={"ID":"4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e","Type":"ContainerDied","Data":"3d0aa9576f987ba3bbc65eb4e5f06208f8c1da49b96b58245b6c2205fde28d40"} Oct 31 13:17:51.803903 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:51.803885 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt" Oct 31 13:17:51.804434 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:51.803915 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt" event={"ID":"4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e","Type":"ContainerDied","Data":"db5ec724ec3e7a6f4ee97c2eb611ecd2529b4ec1ca2f534907bc27da45958305"} Oct 31 13:17:51.804434 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:51.803946 5693 scope.go:122] "RemoveContainer" containerID="3d0aa9576f987ba3bbc65eb4e5f06208f8c1da49b96b58245b6c2205fde28d40" Oct 31 13:17:51.818598 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:51.818579 5693 scope.go:122] "RemoveContainer" containerID="3d0aa9576f987ba3bbc65eb4e5f06208f8c1da49b96b58245b6c2205fde28d40" Oct 31 13:17:51.818996 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:17:51.818972 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d0aa9576f987ba3bbc65eb4e5f06208f8c1da49b96b58245b6c2205fde28d40\": container with ID starting with 3d0aa9576f987ba3bbc65eb4e5f06208f8c1da49b96b58245b6c2205fde28d40 not found: ID does not exist" containerID="3d0aa9576f987ba3bbc65eb4e5f06208f8c1da49b96b58245b6c2205fde28d40" Oct 31 13:17:51.819101 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:51.819002 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d0aa9576f987ba3bbc65eb4e5f06208f8c1da49b96b58245b6c2205fde28d40"} err="failed to get container status \"3d0aa9576f987ba3bbc65eb4e5f06208f8c1da49b96b58245b6c2205fde28d40\": rpc error: code = NotFound desc = could not find container \"3d0aa9576f987ba3bbc65eb4e5f06208f8c1da49b96b58245b6c2205fde28d40\": container with ID starting with 3d0aa9576f987ba3bbc65eb4e5f06208f8c1da49b96b58245b6c2205fde28d40 not found: ID does not exist" Oct 31 13:17:51.869078 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:51.869050 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt"] Oct 31 13:17:51.878330 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:51.878298 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7d78v47xt"] Oct 31 13:17:53.411597 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:53.411565 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e" path="/var/lib/kubelet/pods/4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e/volumes" Oct 31 13:17:55.463323 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:17:55.463292 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-0c1fbb/agc-cred-5db32c229bfc: secret "agc-cred-5db32c229bfc" not found Oct 31 13:17:55.463323 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:17:55.463325 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48: secret "agc-cred-5db32c229bfc" not found Oct 31 13:17:55.463885 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:17:55.463399 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8e9166c-4c50-4ae0-ba02-30396f4468c3-provider-credentials podName:b8e9166c-4c50-4ae0-ba02-30396f4468c3 nodeName:}" failed. No retries permitted until 2026-10-31 13:17:55.963377482 +0000 UTC m=+2726.670963770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b8e9166c-4c50-4ae0-ba02-30396f4468c3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48" (UID: "b8e9166c-4c50-4ae0-ba02-30396f4468c3") : secret "agc-cred-5db32c229bfc" not found Oct 31 13:17:55.977845 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:17:55.977813 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-0c1fbb/agc-cred-5db32c229bfc: secret "agc-cred-5db32c229bfc" not found Oct 31 13:17:55.977845 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:17:55.977841 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48: secret "agc-cred-5db32c229bfc" not found Oct 31 13:17:55.978066 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:17:55.977897 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8e9166c-4c50-4ae0-ba02-30396f4468c3-provider-credentials podName:b8e9166c-4c50-4ae0-ba02-30396f4468c3 nodeName:}" failed. No retries permitted until 2026-10-31 13:17:56.977882621 +0000 UTC m=+2727.685468897 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b8e9166c-4c50-4ae0-ba02-30396f4468c3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48" (UID: "b8e9166c-4c50-4ae0-ba02-30396f4468c3") : secret "agc-cred-5db32c229bfc" not found Oct 31 13:17:56.993937 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:17:56.993899 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-0c1fbb/agc-cred-5db32c229bfc: secret "agc-cred-5db32c229bfc" not found Oct 31 13:17:56.993937 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:17:56.993929 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48: secret "agc-cred-5db32c229bfc" not found Oct 31 13:17:56.994398 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:17:56.994081 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8e9166c-4c50-4ae0-ba02-30396f4468c3-provider-credentials podName:b8e9166c-4c50-4ae0-ba02-30396f4468c3 nodeName:}" failed. No retries permitted until 2026-10-31 13:17:58.99406431 +0000 UTC m=+2729.701650595 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b8e9166c-4c50-4ae0-ba02-30396f4468c3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48" (UID: "b8e9166c-4c50-4ae0-ba02-30396f4468c3") : secret "agc-cred-5db32c229bfc" not found Oct 31 13:17:57.443598 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:57.443559 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48"] Oct 31 13:17:59.045697 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:17:59.045664 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-0c1fbb/agc-cred-5db32c229bfc: secret "agc-cred-5db32c229bfc" not found Oct 31 13:17:59.045697 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:17:59.045696 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-0c1fbb/agc-cred-e4da4b834193: secret "agc-cred-e4da4b834193" not found Oct 31 13:17:59.046131 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:17:59.045711 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48: [secret "agc-cred-5db32c229bfc" not found, secret "agc-cred-e4da4b834193" not found] Oct 31 13:17:59.046131 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:17:59.045787 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8e9166c-4c50-4ae0-ba02-30396f4468c3-provider-credentials podName:b8e9166c-4c50-4ae0-ba02-30396f4468c3 nodeName:}" failed. No retries permitted until 2026-10-31 13:18:03.045767554 +0000 UTC m=+2733.753353832 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b8e9166c-4c50-4ae0-ba02-30396f4468c3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48" (UID: "b8e9166c-4c50-4ae0-ba02-30396f4468c3") : [secret "agc-cred-5db32c229bfc" not found, secret "agc-cred-e4da4b834193" not found] Oct 31 13:17:59.854268 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:17:59.854234 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg"] Oct 31 13:18:02.449667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:02.448301 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48" podUID="b8e9166c-4c50-4ae0-ba02-30396f4468c3" containerName="payload-processing" containerID="cri-o://22c2ae36537df5d5bb0e9d8412beb6dfa498d1b55a69b6d7741ad7fc46a23f89" gracePeriod=25 Oct 31 13:18:02.760114 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:02.760088 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48" Oct 31 13:18:02.866946 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:02.866920 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8e9166c-4c50-4ae0-ba02-30396f4468c3-kube-api-access-2thv7\" (UniqueName: \"kubernetes.io/projected/b8e9166c-4c50-4ae0-ba02-30396f4468c3-kube-api-access-2thv7\") pod \"b8e9166c-4c50-4ae0-ba02-30396f4468c3\" (UID: \"b8e9166c-4c50-4ae0-ba02-30396f4468c3\") " Oct 31 13:18:02.867117 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:02.866986 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8e9166c-4c50-4ae0-ba02-30396f4468c3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b8e9166c-4c50-4ae0-ba02-30396f4468c3-routing-overlay-volume\") pod \"b8e9166c-4c50-4ae0-ba02-30396f4468c3\" (UID: \"b8e9166c-4c50-4ae0-ba02-30396f4468c3\") " Oct 31 13:18:02.867160 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:02.867123 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8e9166c-4c50-4ae0-ba02-30396f4468c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8e9166c-4c50-4ae0-ba02-30396f4468c3-plugins-config-volume\") pod \"b8e9166c-4c50-4ae0-ba02-30396f4468c3\" (UID: \"b8e9166c-4c50-4ae0-ba02-30396f4468c3\") " Oct 31 13:18:02.867335 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:02.867317 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8e9166c-4c50-4ae0-ba02-30396f4468c3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b8e9166c-4c50-4ae0-ba02-30396f4468c3-provider-credentials\") pod \"b8e9166c-4c50-4ae0-ba02-30396f4468c3\" (UID: \"b8e9166c-4c50-4ae0-ba02-30396f4468c3\") " Oct 31 13:18:02.867452 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:02.867309 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8e9166c-4c50-4ae0-ba02-30396f4468c3-routing-overlay-volume" pod "b8e9166c-4c50-4ae0-ba02-30396f4468c3" (UID: "b8e9166c-4c50-4ae0-ba02-30396f4468c3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:18:02.867513 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:02.867471 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8e9166c-4c50-4ae0-ba02-30396f4468c3-plugins-config-volume" pod "b8e9166c-4c50-4ae0-ba02-30396f4468c3" (UID: "b8e9166c-4c50-4ae0-ba02-30396f4468c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:18:02.867903 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:02.867882 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b8e9166c-4c50-4ae0-ba02-30396f4468c3-routing-overlay-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:02.867903 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:02.867902 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8e9166c-4c50-4ae0-ba02-30396f4468c3-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:02.869562 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:02.869536 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8e9166c-4c50-4ae0-ba02-30396f4468c3-kube-api-access-2thv7" pod "b8e9166c-4c50-4ae0-ba02-30396f4468c3" (UID: "b8e9166c-4c50-4ae0-ba02-30396f4468c3"). InnerVolumeSpecName "kube-api-access-2thv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:18:02.869711 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:02.869694 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8e9166c-4c50-4ae0-ba02-30396f4468c3-provider-credentials" pod "b8e9166c-4c50-4ae0-ba02-30396f4468c3" (UID: "b8e9166c-4c50-4ae0-ba02-30396f4468c3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:18:02.912026 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:02.911992 5693 generic.go:356] "Generic (PLEG): container finished" podID="b8e9166c-4c50-4ae0-ba02-30396f4468c3" containerID="22c2ae36537df5d5bb0e9d8412beb6dfa498d1b55a69b6d7741ad7fc46a23f89" exitCode=0 Oct 31 13:18:02.912165 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:02.912032 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48" Oct 31 13:18:02.912165 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:02.912079 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48" event={"ID":"b8e9166c-4c50-4ae0-ba02-30396f4468c3","Type":"ContainerDied","Data":"22c2ae36537df5d5bb0e9d8412beb6dfa498d1b55a69b6d7741ad7fc46a23f89"} Oct 31 13:18:02.912165 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:02.912131 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48" event={"ID":"b8e9166c-4c50-4ae0-ba02-30396f4468c3","Type":"ContainerDied","Data":"6b0bd7f939f2d671ba365b2ebd70b064199b1cb77ca4c2c9328b673187717d2f"} Oct 31 13:18:02.912165 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:02.912160 5693 scope.go:122] "RemoveContainer" containerID="22c2ae36537df5d5bb0e9d8412beb6dfa498d1b55a69b6d7741ad7fc46a23f89" Oct 31 13:18:02.924163 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:02.924139 5693 scope.go:122] "RemoveContainer" containerID="22c2ae36537df5d5bb0e9d8412beb6dfa498d1b55a69b6d7741ad7fc46a23f89" Oct 31 13:18:02.924508 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:18:02.924459 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22c2ae36537df5d5bb0e9d8412beb6dfa498d1b55a69b6d7741ad7fc46a23f89\": container with ID starting with 22c2ae36537df5d5bb0e9d8412beb6dfa498d1b55a69b6d7741ad7fc46a23f89 not found: ID does not exist" containerID="22c2ae36537df5d5bb0e9d8412beb6dfa498d1b55a69b6d7741ad7fc46a23f89" Oct 31 13:18:02.924592 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:02.924518 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22c2ae36537df5d5bb0e9d8412beb6dfa498d1b55a69b6d7741ad7fc46a23f89"} err="failed to get container status \"22c2ae36537df5d5bb0e9d8412beb6dfa498d1b55a69b6d7741ad7fc46a23f89\": rpc error: code = NotFound desc = could not find container \"22c2ae36537df5d5bb0e9d8412beb6dfa498d1b55a69b6d7741ad7fc46a23f89\": container with ID starting with 22c2ae36537df5d5bb0e9d8412beb6dfa498d1b55a69b6d7741ad7fc46a23f89 not found: ID does not exist" Oct 31 13:18:02.939973 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:02.939946 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48"] Oct 31 13:18:02.944015 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:02.943992 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7678qrt48"] Oct 31 13:18:02.969347 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:02.969319 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b8e9166c-4c50-4ae0-ba02-30396f4468c3-provider-credentials\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:02.969347 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:02.969346 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2thv7\" (UniqueName: \"kubernetes.io/projected/b8e9166c-4c50-4ae0-ba02-30396f4468c3-kube-api-access-2thv7\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:03.410850 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:03.410820 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8e9166c-4c50-4ae0-ba02-30396f4468c3" path="/var/lib/kubelet/pods/b8e9166c-4c50-4ae0-ba02-30396f4468c3/volumes" Oct 31 13:18:04.880781 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:04.880732 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg" podUID="c26eb386-0d37-4259-9f88-4fb02f7164cd" containerName="payload-processing" containerID="cri-o://69a4be1a1e6ed6305f387382e481726774ed173d25fc545dc282379bccd68db8" gracePeriod=25 Oct 31 13:18:05.196938 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:05.196912 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg" Oct 31 13:18:05.324778 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:05.324745 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c26eb386-0d37-4259-9f88-4fb02f7164cd-kube-api-access-gc6xs\" (UniqueName: \"kubernetes.io/projected/c26eb386-0d37-4259-9f88-4fb02f7164cd-kube-api-access-gc6xs\") pod \"c26eb386-0d37-4259-9f88-4fb02f7164cd\" (UID: \"c26eb386-0d37-4259-9f88-4fb02f7164cd\") " Oct 31 13:18:05.324973 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:05.324882 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c26eb386-0d37-4259-9f88-4fb02f7164cd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c26eb386-0d37-4259-9f88-4fb02f7164cd-routing-overlay-volume\") pod \"c26eb386-0d37-4259-9f88-4fb02f7164cd\" (UID: \"c26eb386-0d37-4259-9f88-4fb02f7164cd\") " Oct 31 13:18:05.326241 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:05.325217 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c26eb386-0d37-4259-9f88-4fb02f7164cd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c26eb386-0d37-4259-9f88-4fb02f7164cd-provider-credentials\") pod \"c26eb386-0d37-4259-9f88-4fb02f7164cd\" (UID: \"c26eb386-0d37-4259-9f88-4fb02f7164cd\") " Oct 31 13:18:05.326241 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:05.325292 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c26eb386-0d37-4259-9f88-4fb02f7164cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c26eb386-0d37-4259-9f88-4fb02f7164cd-plugins-config-volume\") pod \"c26eb386-0d37-4259-9f88-4fb02f7164cd\" (UID: \"c26eb386-0d37-4259-9f88-4fb02f7164cd\") " Oct 31 13:18:05.333845 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:05.332960 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c26eb386-0d37-4259-9f88-4fb02f7164cd-plugins-config-volume" pod "c26eb386-0d37-4259-9f88-4fb02f7164cd" (UID: "c26eb386-0d37-4259-9f88-4fb02f7164cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:18:05.334312 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:05.334239 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c26eb386-0d37-4259-9f88-4fb02f7164cd-kube-api-access-gc6xs" pod "c26eb386-0d37-4259-9f88-4fb02f7164cd" (UID: "c26eb386-0d37-4259-9f88-4fb02f7164cd"). InnerVolumeSpecName "kube-api-access-gc6xs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:18:05.334312 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:05.334244 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c26eb386-0d37-4259-9f88-4fb02f7164cd-routing-overlay-volume" pod "c26eb386-0d37-4259-9f88-4fb02f7164cd" (UID: "c26eb386-0d37-4259-9f88-4fb02f7164cd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:18:05.336695 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:05.336672 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c26eb386-0d37-4259-9f88-4fb02f7164cd-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:05.336798 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:05.336703 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc6xs\" (UniqueName: \"kubernetes.io/projected/c26eb386-0d37-4259-9f88-4fb02f7164cd-kube-api-access-gc6xs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:05.336798 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:05.336721 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c26eb386-0d37-4259-9f88-4fb02f7164cd-routing-overlay-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:05.337417 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:05.337393 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c26eb386-0d37-4259-9f88-4fb02f7164cd-provider-credentials" pod "c26eb386-0d37-4259-9f88-4fb02f7164cd" (UID: "c26eb386-0d37-4259-9f88-4fb02f7164cd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:18:05.438540 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:05.438508 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c26eb386-0d37-4259-9f88-4fb02f7164cd-provider-credentials\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:05.976842 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:05.976808 5693 generic.go:356] "Generic (PLEG): container finished" podID="c26eb386-0d37-4259-9f88-4fb02f7164cd" containerID="69a4be1a1e6ed6305f387382e481726774ed173d25fc545dc282379bccd68db8" exitCode=0 Oct 31 13:18:05.977297 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:05.976901 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg" event={"ID":"c26eb386-0d37-4259-9f88-4fb02f7164cd","Type":"ContainerDied","Data":"69a4be1a1e6ed6305f387382e481726774ed173d25fc545dc282379bccd68db8"} Oct 31 13:18:05.977297 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:05.976936 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg" Oct 31 13:18:05.977297 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:05.976978 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg" event={"ID":"c26eb386-0d37-4259-9f88-4fb02f7164cd","Type":"ContainerDied","Data":"b94c223240971a56efeab447a2669206fc042aad74e45460dc18e859892abd7f"} Oct 31 13:18:05.977297 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:05.977003 5693 scope.go:122] "RemoveContainer" containerID="69a4be1a1e6ed6305f387382e481726774ed173d25fc545dc282379bccd68db8" Oct 31 13:18:05.994355 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:05.994332 5693 scope.go:122] "RemoveContainer" containerID="69a4be1a1e6ed6305f387382e481726774ed173d25fc545dc282379bccd68db8" Oct 31 13:18:05.994744 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:18:05.994722 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69a4be1a1e6ed6305f387382e481726774ed173d25fc545dc282379bccd68db8\": container with ID starting with 69a4be1a1e6ed6305f387382e481726774ed173d25fc545dc282379bccd68db8 not found: ID does not exist" containerID="69a4be1a1e6ed6305f387382e481726774ed173d25fc545dc282379bccd68db8" Oct 31 13:18:05.994820 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:05.994754 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69a4be1a1e6ed6305f387382e481726774ed173d25fc545dc282379bccd68db8"} err="failed to get container status \"69a4be1a1e6ed6305f387382e481726774ed173d25fc545dc282379bccd68db8\": rpc error: code = NotFound desc = could not find container \"69a4be1a1e6ed6305f387382e481726774ed173d25fc545dc282379bccd68db8\": container with ID starting with 69a4be1a1e6ed6305f387382e481726774ed173d25fc545dc282379bccd68db8 not found: ID does not exist" Oct 31 13:18:06.017348 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:06.017312 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg"] Oct 31 13:18:06.027716 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:06.027688 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b658ldbg"] Oct 31 13:18:07.411425 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:07.411393 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c26eb386-0d37-4259-9f88-4fb02f7164cd" path="/var/lib/kubelet/pods/c26eb386-0d37-4259-9f88-4fb02f7164cd/volumes" Oct 31 13:18:15.856031 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:15.855992 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc"] Oct 31 13:18:15.856941 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:15.856906 5693 cpu_manager.go:414] "containerMap: removing container" podUID="4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e" containerName="payload-processing" Oct 31 13:18:15.856941 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:15.856938 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e" containerName="payload-processing" Oct 31 13:18:15.856941 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:15.856952 5693 cpu_manager.go:414] "containerMap: removing container" podUID="c26eb386-0d37-4259-9f88-4fb02f7164cd" containerName="payload-processing" Oct 31 13:18:15.856941 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:15.856957 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c26eb386-0d37-4259-9f88-4fb02f7164cd" containerName="payload-processing" Oct 31 13:18:15.857443 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:15.856977 5693 cpu_manager.go:414] "containerMap: removing container" podUID="b8e9166c-4c50-4ae0-ba02-30396f4468c3" containerName="payload-processing" Oct 31 13:18:15.857443 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:15.856982 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8e9166c-4c50-4ae0-ba02-30396f4468c3" containerName="payload-processing" Oct 31 13:18:15.857443 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:15.857111 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d6f3ba8-fc0b-41ba-9e8e-9c67724dc17e" containerName="payload-processing" Oct 31 13:18:15.857443 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:15.857119 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="c26eb386-0d37-4259-9f88-4fb02f7164cd" containerName="payload-processing" Oct 31 13:18:15.857443 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:15.857128 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8e9166c-4c50-4ae0-ba02-30396f4468c3" containerName="payload-processing" Oct 31 13:18:15.862751 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:15.862728 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc" Oct 31 13:18:15.877729 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:15.877703 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-54a16b\"/\"payload-processing-external-model-e2e--427d574a-dockercfg-xc26m\"" Oct 31 13:18:15.880282 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:15.880259 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-54a16b\"/\"agc-cred-93157c2e27cd\"" Oct 31 13:18:15.881350 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:15.881331 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-54a16b\"/\"openshift-service-ca.crt\"" Oct 31 13:18:15.881537 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:15.881522 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-54a16b\"/\"payload-processing-external-model-plugins-e2e-extmod-a-54a16b\"" Oct 31 13:18:15.881629 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:15.881599 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-54a16b\"/\"routing-overlay\"" Oct 31 13:18:15.881806 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:15.881790 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-54a16b\"/\"kube-root-ca.crt\"" Oct 31 13:18:15.882596 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:15.882576 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc"] Oct 31 13:18:15.973109 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:15.972232 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/be867635-1ac8-4f6e-9802-98a1d4aba540-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc\" (UID: \"be867635-1ac8-4f6e-9802-98a1d4aba540\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc" Oct 31 13:18:15.973109 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:15.972300 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be867635-1ac8-4f6e-9802-98a1d4aba540-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc\" (UID: \"be867635-1ac8-4f6e-9802-98a1d4aba540\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc" Oct 31 13:18:15.973109 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:15.972379 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts4fh\" (UniqueName: \"kubernetes.io/projected/be867635-1ac8-4f6e-9802-98a1d4aba540-kube-api-access-ts4fh\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc\" (UID: \"be867635-1ac8-4f6e-9802-98a1d4aba540\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc" Oct 31 13:18:15.973109 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:15.972600 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/be867635-1ac8-4f6e-9802-98a1d4aba540-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc\" (UID: \"be867635-1ac8-4f6e-9802-98a1d4aba540\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc" Oct 31 13:18:16.094665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:16.094350 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/be867635-1ac8-4f6e-9802-98a1d4aba540-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc\" (UID: \"be867635-1ac8-4f6e-9802-98a1d4aba540\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc" Oct 31 13:18:16.094665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:16.094413 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be867635-1ac8-4f6e-9802-98a1d4aba540-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc\" (UID: \"be867635-1ac8-4f6e-9802-98a1d4aba540\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc" Oct 31 13:18:16.094665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:16.094487 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts4fh\" (UniqueName: \"kubernetes.io/projected/be867635-1ac8-4f6e-9802-98a1d4aba540-kube-api-access-ts4fh\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc\" (UID: \"be867635-1ac8-4f6e-9802-98a1d4aba540\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc" Oct 31 13:18:16.094994 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:16.094698 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/be867635-1ac8-4f6e-9802-98a1d4aba540-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc\" (UID: \"be867635-1ac8-4f6e-9802-98a1d4aba540\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc" Oct 31 13:18:16.097891 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:16.096247 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be867635-1ac8-4f6e-9802-98a1d4aba540-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc\" (UID: \"be867635-1ac8-4f6e-9802-98a1d4aba540\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc" Oct 31 13:18:16.097891 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:16.096855 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/be867635-1ac8-4f6e-9802-98a1d4aba540-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc\" (UID: \"be867635-1ac8-4f6e-9802-98a1d4aba540\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc" Oct 31 13:18:16.180686 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:16.180129 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts4fh\" (UniqueName: \"kubernetes.io/projected/be867635-1ac8-4f6e-9802-98a1d4aba540-kube-api-access-ts4fh\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc\" (UID: \"be867635-1ac8-4f6e-9802-98a1d4aba540\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc" Oct 31 13:18:16.203800 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:16.203707 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/be867635-1ac8-4f6e-9802-98a1d4aba540-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc\" (UID: \"be867635-1ac8-4f6e-9802-98a1d4aba540\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc" Oct 31 13:18:16.480822 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:16.480750 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc" Oct 31 13:18:16.545148 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:16.545109 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj"] Oct 31 13:18:16.551728 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:16.551704 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj" Oct 31 13:18:16.568153 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:16.560849 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj"] Oct 31 13:18:16.568153 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:16.561170 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-54a16b\"/\"agc-cred-44a9af9b2905\"" Oct 31 13:18:16.706490 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:16.706457 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz9sz\" (UniqueName: \"kubernetes.io/projected/ec1e3d94-c558-4967-95c5-88b63444e8e6-kube-api-access-lz9sz\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj\" (UID: \"ec1e3d94-c558-4967-95c5-88b63444e8e6\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj" Oct 31 13:18:16.706674 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:16.706528 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ec1e3d94-c558-4967-95c5-88b63444e8e6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj\" (UID: \"ec1e3d94-c558-4967-95c5-88b63444e8e6\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj" Oct 31 13:18:16.706674 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:16.706629 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ec1e3d94-c558-4967-95c5-88b63444e8e6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj\" (UID: \"ec1e3d94-c558-4967-95c5-88b63444e8e6\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj" Oct 31 13:18:16.706903 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:16.706883 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec1e3d94-c558-4967-95c5-88b63444e8e6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj\" (UID: \"ec1e3d94-c558-4967-95c5-88b63444e8e6\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj" Oct 31 13:18:16.782698 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:16.777449 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc"] Oct 31 13:18:16.782698 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:18:16.778266 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe867635_1ac8_4f6e_9802_98a1d4aba540.slice/crio-e4980a888d879369dc090dac16c56dd3499c0bdfa7c7ae95904e9c9591fb056d WatchSource:0}: Error finding container e4980a888d879369dc090dac16c56dd3499c0bdfa7c7ae95904e9c9591fb056d: Status 404 returned error can't find the container with id e4980a888d879369dc090dac16c56dd3499c0bdfa7c7ae95904e9c9591fb056d Oct 31 13:18:16.807917 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:16.807884 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ec1e3d94-c558-4967-95c5-88b63444e8e6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj\" (UID: \"ec1e3d94-c558-4967-95c5-88b63444e8e6\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj" Oct 31 13:18:16.808076 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:16.807951 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ec1e3d94-c558-4967-95c5-88b63444e8e6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj\" (UID: \"ec1e3d94-c558-4967-95c5-88b63444e8e6\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj" Oct 31 13:18:16.808211 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:16.808177 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec1e3d94-c558-4967-95c5-88b63444e8e6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj\" (UID: \"ec1e3d94-c558-4967-95c5-88b63444e8e6\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj" Oct 31 13:18:16.808343 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:16.808266 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz9sz\" (UniqueName: \"kubernetes.io/projected/ec1e3d94-c558-4967-95c5-88b63444e8e6-kube-api-access-lz9sz\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj\" (UID: \"ec1e3d94-c558-4967-95c5-88b63444e8e6\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj" Oct 31 13:18:16.808619 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:16.808594 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ec1e3d94-c558-4967-95c5-88b63444e8e6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj\" (UID: \"ec1e3d94-c558-4967-95c5-88b63444e8e6\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj" Oct 31 13:18:16.808999 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:16.808980 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec1e3d94-c558-4967-95c5-88b63444e8e6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj\" (UID: \"ec1e3d94-c558-4967-95c5-88b63444e8e6\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj" Oct 31 13:18:16.810565 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:16.810547 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ec1e3d94-c558-4967-95c5-88b63444e8e6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj\" (UID: \"ec1e3d94-c558-4967-95c5-88b63444e8e6\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj" Oct 31 13:18:16.822467 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:16.822446 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz9sz\" (UniqueName: \"kubernetes.io/projected/ec1e3d94-c558-4967-95c5-88b63444e8e6-kube-api-access-lz9sz\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj\" (UID: \"ec1e3d94-c558-4967-95c5-88b63444e8e6\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj" Oct 31 13:18:16.889331 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:16.889295 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj" Oct 31 13:18:17.191597 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:17.191556 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc" event={"ID":"be867635-1ac8-4f6e-9802-98a1d4aba540","Type":"ContainerStarted","Data":"16e69beb185d1e6768e173aaf19a4f076193ecc12006c9ec7d5a5b713ca374e8"} Oct 31 13:18:17.191822 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:17.191615 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc" event={"ID":"be867635-1ac8-4f6e-9802-98a1d4aba540","Type":"ContainerStarted","Data":"e4980a888d879369dc090dac16c56dd3499c0bdfa7c7ae95904e9c9591fb056d"} Oct 31 13:18:17.191822 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:17.191683 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc" Oct 31 13:18:17.198811 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:17.198772 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj"] Oct 31 13:18:17.199512 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:18:17.199489 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec1e3d94_c558_4967_95c5_88b63444e8e6.slice/crio-75f9118a94251997083b1f410b89f990ee90642e01182f1d88ddb66f2c9a93f2 WatchSource:0}: Error finding container 75f9118a94251997083b1f410b89f990ee90642e01182f1d88ddb66f2c9a93f2: Status 404 returned error can't find the container with id 75f9118a94251997083b1f410b89f990ee90642e01182f1d88ddb66f2c9a93f2 Oct 31 13:18:17.219832 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:17.219764 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc" podStartSLOduration=2.219746173 podStartE2EDuration="2.219746173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:18:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:18:17.210365626 +0000 UTC m=+2747.917951925" watchObservedRunningTime="2026-10-31 13:18:17.219746173 +0000 UTC m=+2747.927332468" Oct 31 13:18:18.201532 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:18.201474 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj" event={"ID":"ec1e3d94-c558-4967-95c5-88b63444e8e6","Type":"ContainerStarted","Data":"925c97f128e04b4190aeda5e068e3efa927437d2f4a14b3c19e59a1abb67fe85"} Oct 31 13:18:18.202078 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:18.201563 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj" event={"ID":"ec1e3d94-c558-4967-95c5-88b63444e8e6","Type":"ContainerStarted","Data":"75f9118a94251997083b1f410b89f990ee90642e01182f1d88ddb66f2c9a93f2"} Oct 31 13:18:18.202078 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:18.201817 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj" Oct 31 13:18:18.222413 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:18.222358 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj" podStartSLOduration=2.222344945 podStartE2EDuration="2.222344945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:18:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:18:18.221218313 +0000 UTC m=+2748.928804608" watchObservedRunningTime="2026-10-31 13:18:18.222344945 +0000 UTC m=+2748.929931241" Oct 31 13:18:18.692588 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:18:18.692551 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:18:28.205585 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:28.205553 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc" Oct 31 13:18:29.213228 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:29.213200 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj" Oct 31 13:18:29.280695 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:29.280656 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc"] Oct 31 13:18:34.282769 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:34.282715 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc" podUID="be867635-1ac8-4f6e-9802-98a1d4aba540" containerName="payload-processing" containerID="cri-o://16e69beb185d1e6768e173aaf19a4f076193ecc12006c9ec7d5a5b713ca374e8" gracePeriod=25 Oct 31 13:18:34.451424 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:34.451392 5693 generic.go:356] "Generic (PLEG): container finished" podID="be867635-1ac8-4f6e-9802-98a1d4aba540" containerID="16e69beb185d1e6768e173aaf19a4f076193ecc12006c9ec7d5a5b713ca374e8" exitCode=0 Oct 31 13:18:34.451573 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:34.451470 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc" event={"ID":"be867635-1ac8-4f6e-9802-98a1d4aba540","Type":"ContainerDied","Data":"16e69beb185d1e6768e173aaf19a4f076193ecc12006c9ec7d5a5b713ca374e8"} Oct 31 13:18:34.594849 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:34.594817 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc" Oct 31 13:18:34.735381 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:34.735335 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be867635-1ac8-4f6e-9802-98a1d4aba540-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/be867635-1ac8-4f6e-9802-98a1d4aba540-routing-overlay-volume\") pod \"be867635-1ac8-4f6e-9802-98a1d4aba540\" (UID: \"be867635-1ac8-4f6e-9802-98a1d4aba540\") " Oct 31 13:18:34.735583 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:34.735420 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be867635-1ac8-4f6e-9802-98a1d4aba540-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be867635-1ac8-4f6e-9802-98a1d4aba540-plugins-config-volume\") pod \"be867635-1ac8-4f6e-9802-98a1d4aba540\" (UID: \"be867635-1ac8-4f6e-9802-98a1d4aba540\") " Oct 31 13:18:34.735583 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:34.735534 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be867635-1ac8-4f6e-9802-98a1d4aba540-provider-credentials\" (UniqueName: \"kubernetes.io/projected/be867635-1ac8-4f6e-9802-98a1d4aba540-provider-credentials\") pod \"be867635-1ac8-4f6e-9802-98a1d4aba540\" (UID: \"be867635-1ac8-4f6e-9802-98a1d4aba540\") " Oct 31 13:18:34.735747 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:34.735693 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be867635-1ac8-4f6e-9802-98a1d4aba540-kube-api-access-ts4fh\" (UniqueName: \"kubernetes.io/projected/be867635-1ac8-4f6e-9802-98a1d4aba540-kube-api-access-ts4fh\") pod \"be867635-1ac8-4f6e-9802-98a1d4aba540\" (UID: \"be867635-1ac8-4f6e-9802-98a1d4aba540\") " Oct 31 13:18:34.737079 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:34.737050 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be867635-1ac8-4f6e-9802-98a1d4aba540-routing-overlay-volume" pod "be867635-1ac8-4f6e-9802-98a1d4aba540" (UID: "be867635-1ac8-4f6e-9802-98a1d4aba540"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:18:34.737571 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:34.737525 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be867635-1ac8-4f6e-9802-98a1d4aba540-plugins-config-volume" pod "be867635-1ac8-4f6e-9802-98a1d4aba540" (UID: "be867635-1ac8-4f6e-9802-98a1d4aba540"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:18:34.739423 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:34.739398 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be867635-1ac8-4f6e-9802-98a1d4aba540-provider-credentials" pod "be867635-1ac8-4f6e-9802-98a1d4aba540" (UID: "be867635-1ac8-4f6e-9802-98a1d4aba540"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:18:34.739704 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:34.739680 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be867635-1ac8-4f6e-9802-98a1d4aba540-kube-api-access-ts4fh" pod "be867635-1ac8-4f6e-9802-98a1d4aba540" (UID: "be867635-1ac8-4f6e-9802-98a1d4aba540"). InnerVolumeSpecName "kube-api-access-ts4fh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:18:34.840748 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:34.838060 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/be867635-1ac8-4f6e-9802-98a1d4aba540-provider-credentials\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:34.840748 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:34.838100 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts4fh\" (UniqueName: \"kubernetes.io/projected/be867635-1ac8-4f6e-9802-98a1d4aba540-kube-api-access-ts4fh\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:34.840748 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:34.838118 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/be867635-1ac8-4f6e-9802-98a1d4aba540-routing-overlay-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:34.840748 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:34.838133 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be867635-1ac8-4f6e-9802-98a1d4aba540-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:34.865606 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:34.865569 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl"] Oct 31 13:18:35.466449 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:35.466399 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc" event={"ID":"be867635-1ac8-4f6e-9802-98a1d4aba540","Type":"ContainerDied","Data":"e4980a888d879369dc090dac16c56dd3499c0bdfa7c7ae95904e9c9591fb056d"} Oct 31 13:18:35.467213 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:35.466468 5693 scope.go:122] "RemoveContainer" containerID="16e69beb185d1e6768e173aaf19a4f076193ecc12006c9ec7d5a5b713ca374e8" Oct 31 13:18:35.467213 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:35.466689 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc" Oct 31 13:18:35.533844 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:35.520415 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc"] Oct 31 13:18:35.559012 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:35.558982 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f6zcnc"] Oct 31 13:18:35.593320 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:35.593282 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz"] Oct 31 13:18:35.594223 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:35.594187 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" podUID="49a12158-0b73-4dbe-86ee-42b72e883cee" containerName="istio-proxy" containerID="cri-o://a7f0da2e9965bcc2978a2cd7beea63fa718a1a2cad77c5cca8961754e21f978a" gracePeriod=30 Oct 31 13:18:37.436302 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:37.436265 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be867635-1ac8-4f6e-9802-98a1d4aba540" path="/var/lib/kubelet/pods/be867635-1ac8-4f6e-9802-98a1d4aba540/volumes" Oct 31 13:18:37.831595 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:37.831517 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8"] Oct 31 13:18:37.832674 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:37.832619 5693 cpu_manager.go:414] "containerMap: removing container" podUID="be867635-1ac8-4f6e-9802-98a1d4aba540" containerName="payload-processing" Oct 31 13:18:37.832674 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:37.832660 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be867635-1ac8-4f6e-9802-98a1d4aba540" containerName="payload-processing" Oct 31 13:18:37.832849 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:37.832838 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="be867635-1ac8-4f6e-9802-98a1d4aba540" containerName="payload-processing" Oct 31 13:18:37.838232 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:37.838211 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8" Oct 31 13:18:37.845764 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:37.845731 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-5dc570\"/\"agc-cred-b4bf09cb0482\"" Oct 31 13:18:37.845937 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:37.845889 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-5dc570\"/\"payload-processing-external-model-e2e--ef24cfc7-dockercfg-vvmt9\"" Oct 31 13:18:37.855137 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:37.855110 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8"] Oct 31 13:18:37.914514 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:37.914468 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9a8e4aec-7674-424f-8b17-100903e002fd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8\" (UID: \"9a8e4aec-7674-424f-8b17-100903e002fd\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8" Oct 31 13:18:37.919050 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:37.919011 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhw9n\" (UniqueName: \"kubernetes.io/projected/9a8e4aec-7674-424f-8b17-100903e002fd-kube-api-access-nhw9n\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8\" (UID: \"9a8e4aec-7674-424f-8b17-100903e002fd\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8" Oct 31 13:18:37.919406 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:37.919383 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9a8e4aec-7674-424f-8b17-100903e002fd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8\" (UID: \"9a8e4aec-7674-424f-8b17-100903e002fd\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8" Oct 31 13:18:37.919569 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:37.919553 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a8e4aec-7674-424f-8b17-100903e002fd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8\" (UID: \"9a8e4aec-7674-424f-8b17-100903e002fd\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8" Oct 31 13:18:38.026665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:38.024078 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhw9n\" (UniqueName: \"kubernetes.io/projected/9a8e4aec-7674-424f-8b17-100903e002fd-kube-api-access-nhw9n\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8\" (UID: \"9a8e4aec-7674-424f-8b17-100903e002fd\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8" Oct 31 13:18:38.026665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:38.024142 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9a8e4aec-7674-424f-8b17-100903e002fd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8\" (UID: \"9a8e4aec-7674-424f-8b17-100903e002fd\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8" Oct 31 13:18:38.026665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:38.024185 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a8e4aec-7674-424f-8b17-100903e002fd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8\" (UID: \"9a8e4aec-7674-424f-8b17-100903e002fd\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8" Oct 31 13:18:38.026665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:38.024502 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9a8e4aec-7674-424f-8b17-100903e002fd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8\" (UID: \"9a8e4aec-7674-424f-8b17-100903e002fd\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8" Oct 31 13:18:38.027876 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:38.027846 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9a8e4aec-7674-424f-8b17-100903e002fd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8\" (UID: \"9a8e4aec-7674-424f-8b17-100903e002fd\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8" Oct 31 13:18:38.029070 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:38.029045 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a8e4aec-7674-424f-8b17-100903e002fd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8\" (UID: \"9a8e4aec-7674-424f-8b17-100903e002fd\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8" Oct 31 13:18:38.046471 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:38.046438 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhw9n\" (UniqueName: \"kubernetes.io/projected/9a8e4aec-7674-424f-8b17-100903e002fd-kube-api-access-nhw9n\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8\" (UID: \"9a8e4aec-7674-424f-8b17-100903e002fd\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8" Oct 31 13:18:38.051838 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:38.051808 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9a8e4aec-7674-424f-8b17-100903e002fd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8\" (UID: \"9a8e4aec-7674-424f-8b17-100903e002fd\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8" Oct 31 13:18:38.167263 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:38.167230 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8" Oct 31 13:18:38.461792 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:38.461763 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8"] Oct 31 13:18:38.462176 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:18:38.462097 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a8e4aec_7674_424f_8b17_100903e002fd.slice/crio-70d6d05e617176cc4e7ae7961ebe826763587c61a1a6105383e9234ebd95050e WatchSource:0}: Error finding container 70d6d05e617176cc4e7ae7961ebe826763587c61a1a6105383e9234ebd95050e: Status 404 returned error can't find the container with id 70d6d05e617176cc4e7ae7961ebe826763587c61a1a6105383e9234ebd95050e Oct 31 13:18:38.620858 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:38.620822 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8" event={"ID":"9a8e4aec-7674-424f-8b17-100903e002fd","Type":"ContainerStarted","Data":"4abe26b5fd07d2fc0aaa14da8a6e4d7ca39386a4d3bf9bfd4f2033b35de31c2b"} Oct 31 13:18:38.621091 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:38.620893 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8" Oct 31 13:18:38.621091 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:38.620912 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8" event={"ID":"9a8e4aec-7674-424f-8b17-100903e002fd","Type":"ContainerStarted","Data":"70d6d05e617176cc4e7ae7961ebe826763587c61a1a6105383e9234ebd95050e"} Oct 31 13:18:38.643902 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:38.643852 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8" podStartSLOduration=1.643839066 podStartE2EDuration="1.643839066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:18:38.639694565 +0000 UTC m=+2769.347280860" watchObservedRunningTime="2026-10-31 13:18:38.643839066 +0000 UTC m=+2769.351425387" Oct 31 13:18:39.866040 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:39.865996 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl" podUID="2f674e1e-7ff4-4923-b32a-fab0fa51d390" containerName="payload-processing" containerID="cri-o://303bd09edb31c0cc907deb55bfbf5d1e3ca06c3ba6d8afddf6c482986723506c" gracePeriod=25 Oct 31 13:18:40.045666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.044701 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr"] Oct 31 13:18:40.060665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.060584 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr" Oct 31 13:18:40.082666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.079619 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/558af0a5-143e-4842-a180-22280235ab6d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr\" (UID: \"558af0a5-143e-4842-a180-22280235ab6d\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr" Oct 31 13:18:40.082666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.079691 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/558af0a5-143e-4842-a180-22280235ab6d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr\" (UID: \"558af0a5-143e-4842-a180-22280235ab6d\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr" Oct 31 13:18:40.082666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.079757 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmphg\" (UniqueName: \"kubernetes.io/projected/558af0a5-143e-4842-a180-22280235ab6d-kube-api-access-pmphg\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr\" (UID: \"558af0a5-143e-4842-a180-22280235ab6d\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr" Oct 31 13:18:40.082666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.079887 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/558af0a5-143e-4842-a180-22280235ab6d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr\" (UID: \"558af0a5-143e-4842-a180-22280235ab6d\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr" Oct 31 13:18:40.102836 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.093951 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-5dc570\"/\"e2e-cfl-d152ec-provider-cred\"" Oct 31 13:18:40.175944 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.175904 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr"] Oct 31 13:18:40.223319 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.194388 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/558af0a5-143e-4842-a180-22280235ab6d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr\" (UID: \"558af0a5-143e-4842-a180-22280235ab6d\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr" Oct 31 13:18:40.223319 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.214786 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/558af0a5-143e-4842-a180-22280235ab6d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr\" (UID: \"558af0a5-143e-4842-a180-22280235ab6d\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr" Oct 31 13:18:40.223319 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.214899 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmphg\" (UniqueName: \"kubernetes.io/projected/558af0a5-143e-4842-a180-22280235ab6d-kube-api-access-pmphg\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr\" (UID: \"558af0a5-143e-4842-a180-22280235ab6d\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr" Oct 31 13:18:40.223319 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.215122 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/558af0a5-143e-4842-a180-22280235ab6d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr\" (UID: \"558af0a5-143e-4842-a180-22280235ab6d\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr" Oct 31 13:18:40.223319 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.216123 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/558af0a5-143e-4842-a180-22280235ab6d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr\" (UID: \"558af0a5-143e-4842-a180-22280235ab6d\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr" Oct 31 13:18:40.223319 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.216805 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/558af0a5-143e-4842-a180-22280235ab6d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr\" (UID: \"558af0a5-143e-4842-a180-22280235ab6d\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr" Oct 31 13:18:40.230730 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.230693 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/558af0a5-143e-4842-a180-22280235ab6d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr\" (UID: \"558af0a5-143e-4842-a180-22280235ab6d\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr" Oct 31 13:18:40.238390 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.238360 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmphg\" (UniqueName: \"kubernetes.io/projected/558af0a5-143e-4842-a180-22280235ab6d-kube-api-access-pmphg\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr\" (UID: \"558af0a5-143e-4842-a180-22280235ab6d\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr" Oct 31 13:18:40.436164 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.436138 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl" Oct 31 13:18:40.517424 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.517094 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr" Oct 31 13:18:40.524938 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.520997 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f674e1e-7ff4-4923-b32a-fab0fa51d390-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f674e1e-7ff4-4923-b32a-fab0fa51d390-plugins-config-volume\") pod \"2f674e1e-7ff4-4923-b32a-fab0fa51d390\" (UID: \"2f674e1e-7ff4-4923-b32a-fab0fa51d390\") " Oct 31 13:18:40.524938 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.521309 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f674e1e-7ff4-4923-b32a-fab0fa51d390-kube-api-access-56cw4\" (UniqueName: \"kubernetes.io/projected/2f674e1e-7ff4-4923-b32a-fab0fa51d390-kube-api-access-56cw4\") pod \"2f674e1e-7ff4-4923-b32a-fab0fa51d390\" (UID: \"2f674e1e-7ff4-4923-b32a-fab0fa51d390\") " Oct 31 13:18:40.524938 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.521363 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f674e1e-7ff4-4923-b32a-fab0fa51d390-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f674e1e-7ff4-4923-b32a-fab0fa51d390-provider-credentials\") pod \"2f674e1e-7ff4-4923-b32a-fab0fa51d390\" (UID: \"2f674e1e-7ff4-4923-b32a-fab0fa51d390\") " Oct 31 13:18:40.524938 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.521400 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f674e1e-7ff4-4923-b32a-fab0fa51d390-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f674e1e-7ff4-4923-b32a-fab0fa51d390-routing-overlay-volume\") pod \"2f674e1e-7ff4-4923-b32a-fab0fa51d390\" (UID: \"2f674e1e-7ff4-4923-b32a-fab0fa51d390\") " Oct 31 13:18:40.524938 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.522720 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f674e1e-7ff4-4923-b32a-fab0fa51d390-routing-overlay-volume" pod "2f674e1e-7ff4-4923-b32a-fab0fa51d390" (UID: "2f674e1e-7ff4-4923-b32a-fab0fa51d390"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:18:40.524938 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.523128 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f674e1e-7ff4-4923-b32a-fab0fa51d390-plugins-config-volume" pod "2f674e1e-7ff4-4923-b32a-fab0fa51d390" (UID: "2f674e1e-7ff4-4923-b32a-fab0fa51d390"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:18:40.537093 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.536835 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f674e1e-7ff4-4923-b32a-fab0fa51d390-kube-api-access-56cw4" pod "2f674e1e-7ff4-4923-b32a-fab0fa51d390" (UID: "2f674e1e-7ff4-4923-b32a-fab0fa51d390"). InnerVolumeSpecName "kube-api-access-56cw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:18:40.542417 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.542381 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f674e1e-7ff4-4923-b32a-fab0fa51d390-provider-credentials" pod "2f674e1e-7ff4-4923-b32a-fab0fa51d390" (UID: "2f674e1e-7ff4-4923-b32a-fab0fa51d390"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:18:40.625522 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.625422 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56cw4\" (UniqueName: \"kubernetes.io/projected/2f674e1e-7ff4-4923-b32a-fab0fa51d390-kube-api-access-56cw4\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:40.625522 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.625458 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f674e1e-7ff4-4923-b32a-fab0fa51d390-provider-credentials\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:40.625522 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.625474 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f674e1e-7ff4-4923-b32a-fab0fa51d390-routing-overlay-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:40.625522 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.625488 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f674e1e-7ff4-4923-b32a-fab0fa51d390-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:40.714707 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.712857 5693 generic.go:356] "Generic (PLEG): container finished" podID="2f674e1e-7ff4-4923-b32a-fab0fa51d390" containerID="303bd09edb31c0cc907deb55bfbf5d1e3ca06c3ba6d8afddf6c482986723506c" exitCode=0 Oct 31 13:18:40.714707 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.713426 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl" Oct 31 13:18:40.716594 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.716560 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl" event={"ID":"2f674e1e-7ff4-4923-b32a-fab0fa51d390","Type":"ContainerDied","Data":"303bd09edb31c0cc907deb55bfbf5d1e3ca06c3ba6d8afddf6c482986723506c"} Oct 31 13:18:40.716789 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.716630 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl" event={"ID":"2f674e1e-7ff4-4923-b32a-fab0fa51d390","Type":"ContainerDied","Data":"96d4c2eefb32c1b8b33eb91290457c5bd218e07324a3206d25661725ff286ffa"} Oct 31 13:18:40.716789 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.716682 5693 scope.go:122] "RemoveContainer" containerID="303bd09edb31c0cc907deb55bfbf5d1e3ca06c3ba6d8afddf6c482986723506c" Oct 31 13:18:40.740412 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.740334 5693 scope.go:122] "RemoveContainer" containerID="303bd09edb31c0cc907deb55bfbf5d1e3ca06c3ba6d8afddf6c482986723506c" Oct 31 13:18:40.742007 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:18:40.741934 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"303bd09edb31c0cc907deb55bfbf5d1e3ca06c3ba6d8afddf6c482986723506c\": container with ID starting with 303bd09edb31c0cc907deb55bfbf5d1e3ca06c3ba6d8afddf6c482986723506c not found: ID does not exist" containerID="303bd09edb31c0cc907deb55bfbf5d1e3ca06c3ba6d8afddf6c482986723506c" Oct 31 13:18:40.742007 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.741974 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"303bd09edb31c0cc907deb55bfbf5d1e3ca06c3ba6d8afddf6c482986723506c"} err="failed to get container status \"303bd09edb31c0cc907deb55bfbf5d1e3ca06c3ba6d8afddf6c482986723506c\": rpc error: code = NotFound desc = could not find container \"303bd09edb31c0cc907deb55bfbf5d1e3ca06c3ba6d8afddf6c482986723506c\": container with ID starting with 303bd09edb31c0cc907deb55bfbf5d1e3ca06c3ba6d8afddf6c482986723506c not found: ID does not exist" Oct 31 13:18:40.760128 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.759877 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl"] Oct 31 13:18:40.768121 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.768050 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7f4bkcpbl"] Oct 31 13:18:40.842818 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.842790 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr"] Oct 31 13:18:40.844519 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:18:40.844491 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod558af0a5_143e_4842_a180_22280235ab6d.slice/crio-a2522fb4301da2c9cafdfe9cbffcb9f1defd1324a6fb9647254bfd8deaf3b07e WatchSource:0}: Error finding container a2522fb4301da2c9cafdfe9cbffcb9f1defd1324a6fb9647254bfd8deaf3b07e: Status 404 returned error can't find the container with id a2522fb4301da2c9cafdfe9cbffcb9f1defd1324a6fb9647254bfd8deaf3b07e Oct 31 13:18:40.957984 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:40.957961 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" Oct 31 13:18:41.035668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.033383 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49a12158-0b73-4dbe-86ee-42b72e883cee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a12158-0b73-4dbe-86ee-42b72e883cee-credential-socket\") pod \"49a12158-0b73-4dbe-86ee-42b72e883cee\" (UID: \"49a12158-0b73-4dbe-86ee-42b72e883cee\") " Oct 31 13:18:41.035668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.033524 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49a12158-0b73-4dbe-86ee-42b72e883cee-kube-api-access-vwszl\" (UniqueName: \"kubernetes.io/projected/49a12158-0b73-4dbe-86ee-42b72e883cee-kube-api-access-vwszl\") pod \"49a12158-0b73-4dbe-86ee-42b72e883cee\" (UID: \"49a12158-0b73-4dbe-86ee-42b72e883cee\") " Oct 31 13:18:41.035668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.033595 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49a12158-0b73-4dbe-86ee-42b72e883cee-istio-token\" (UniqueName: \"kubernetes.io/projected/49a12158-0b73-4dbe-86ee-42b72e883cee-istio-token\") pod \"49a12158-0b73-4dbe-86ee-42b72e883cee\" (UID: \"49a12158-0b73-4dbe-86ee-42b72e883cee\") " Oct 31 13:18:41.035668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.033675 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49a12158-0b73-4dbe-86ee-42b72e883cee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49a12158-0b73-4dbe-86ee-42b72e883cee-istio-data\") pod \"49a12158-0b73-4dbe-86ee-42b72e883cee\" (UID: \"49a12158-0b73-4dbe-86ee-42b72e883cee\") " Oct 31 13:18:41.035668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.033847 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49a12158-0b73-4dbe-86ee-42b72e883cee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49a12158-0b73-4dbe-86ee-42b72e883cee-istio-podinfo\") pod \"49a12158-0b73-4dbe-86ee-42b72e883cee\" (UID: \"49a12158-0b73-4dbe-86ee-42b72e883cee\") " Oct 31 13:18:41.035668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.033916 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49a12158-0b73-4dbe-86ee-42b72e883cee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49a12158-0b73-4dbe-86ee-42b72e883cee-workload-certs\") pod \"49a12158-0b73-4dbe-86ee-42b72e883cee\" (UID: \"49a12158-0b73-4dbe-86ee-42b72e883cee\") " Oct 31 13:18:41.035668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.033973 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49a12158-0b73-4dbe-86ee-42b72e883cee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49a12158-0b73-4dbe-86ee-42b72e883cee-istiod-ca-cert\") pod \"49a12158-0b73-4dbe-86ee-42b72e883cee\" (UID: \"49a12158-0b73-4dbe-86ee-42b72e883cee\") " Oct 31 13:18:41.035668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.034058 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49a12158-0b73-4dbe-86ee-42b72e883cee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a12158-0b73-4dbe-86ee-42b72e883cee-workload-socket\") pod \"49a12158-0b73-4dbe-86ee-42b72e883cee\" (UID: \"49a12158-0b73-4dbe-86ee-42b72e883cee\") " Oct 31 13:18:41.035668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.034236 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49a12158-0b73-4dbe-86ee-42b72e883cee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49a12158-0b73-4dbe-86ee-42b72e883cee-istio-envoy\") pod \"49a12158-0b73-4dbe-86ee-42b72e883cee\" (UID: \"49a12158-0b73-4dbe-86ee-42b72e883cee\") " Oct 31 13:18:41.035668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.034615 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49a12158-0b73-4dbe-86ee-42b72e883cee-credential-socket" pod "49a12158-0b73-4dbe-86ee-42b72e883cee" (UID: "49a12158-0b73-4dbe-86ee-42b72e883cee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:18:41.035668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.035127 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49a12158-0b73-4dbe-86ee-42b72e883cee-istiod-ca-cert" pod "49a12158-0b73-4dbe-86ee-42b72e883cee" (UID: "49a12158-0b73-4dbe-86ee-42b72e883cee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:18:41.035668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.035376 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49a12158-0b73-4dbe-86ee-42b72e883cee-workload-certs" pod "49a12158-0b73-4dbe-86ee-42b72e883cee" (UID: "49a12158-0b73-4dbe-86ee-42b72e883cee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:18:41.036574 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.035835 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49a12158-0b73-4dbe-86ee-42b72e883cee-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:41.036574 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.035858 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49a12158-0b73-4dbe-86ee-42b72e883cee-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:41.036574 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.035881 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a12158-0b73-4dbe-86ee-42b72e883cee-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:41.036574 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.036451 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49a12158-0b73-4dbe-86ee-42b72e883cee-istio-data" pod "49a12158-0b73-4dbe-86ee-42b72e883cee" (UID: "49a12158-0b73-4dbe-86ee-42b72e883cee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:18:41.036828 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.036714 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49a12158-0b73-4dbe-86ee-42b72e883cee-kube-api-access-vwszl" pod "49a12158-0b73-4dbe-86ee-42b72e883cee" (UID: "49a12158-0b73-4dbe-86ee-42b72e883cee"). InnerVolumeSpecName "kube-api-access-vwszl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:18:41.036828 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.036718 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49a12158-0b73-4dbe-86ee-42b72e883cee-workload-socket" pod "49a12158-0b73-4dbe-86ee-42b72e883cee" (UID: "49a12158-0b73-4dbe-86ee-42b72e883cee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:18:41.043040 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.042923 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49a12158-0b73-4dbe-86ee-42b72e883cee-istio-envoy" pod "49a12158-0b73-4dbe-86ee-42b72e883cee" (UID: "49a12158-0b73-4dbe-86ee-42b72e883cee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:18:41.043040 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.043002 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49a12158-0b73-4dbe-86ee-42b72e883cee-istio-podinfo" pod "49a12158-0b73-4dbe-86ee-42b72e883cee" (UID: "49a12158-0b73-4dbe-86ee-42b72e883cee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:18:41.044617 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.044599 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49a12158-0b73-4dbe-86ee-42b72e883cee-istio-token" pod "49a12158-0b73-4dbe-86ee-42b72e883cee" (UID: "49a12158-0b73-4dbe-86ee-42b72e883cee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:18:41.158671 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.157609 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49a12158-0b73-4dbe-86ee-42b72e883cee-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:41.158671 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.157663 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49a12158-0b73-4dbe-86ee-42b72e883cee-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:41.158671 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.157679 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a12158-0b73-4dbe-86ee-42b72e883cee-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:41.158671 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.157691 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49a12158-0b73-4dbe-86ee-42b72e883cee-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:41.158671 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.157704 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwszl\" (UniqueName: \"kubernetes.io/projected/49a12158-0b73-4dbe-86ee-42b72e883cee-kube-api-access-vwszl\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:41.158671 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.157720 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49a12158-0b73-4dbe-86ee-42b72e883cee-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:41.415300 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.415259 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f674e1e-7ff4-4923-b32a-fab0fa51d390" path="/var/lib/kubelet/pods/2f674e1e-7ff4-4923-b32a-fab0fa51d390/volumes" Oct 31 13:18:41.734762 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.734596 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr" event={"ID":"558af0a5-143e-4842-a180-22280235ab6d","Type":"ContainerStarted","Data":"28c9cbe7e71617ddcda7162babbe2585e98ef2821aefe5f6bf304894c769a222"} Oct 31 13:18:41.734762 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.734700 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr" event={"ID":"558af0a5-143e-4842-a180-22280235ab6d","Type":"ContainerStarted","Data":"a2522fb4301da2c9cafdfe9cbffcb9f1defd1324a6fb9647254bfd8deaf3b07e"} Oct 31 13:18:41.737022 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.734864 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr" Oct 31 13:18:41.738824 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.738798 5693 generic.go:356] "Generic (PLEG): container finished" podID="49a12158-0b73-4dbe-86ee-42b72e883cee" containerID="a7f0da2e9965bcc2978a2cd7beea63fa718a1a2cad77c5cca8961754e21f978a" exitCode=0 Oct 31 13:18:41.738943 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.738857 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" Oct 31 13:18:41.738943 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.738874 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" event={"ID":"49a12158-0b73-4dbe-86ee-42b72e883cee","Type":"ContainerDied","Data":"a7f0da2e9965bcc2978a2cd7beea63fa718a1a2cad77c5cca8961754e21f978a"} Oct 31 13:18:41.738943 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.738926 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz" event={"ID":"49a12158-0b73-4dbe-86ee-42b72e883cee","Type":"ContainerDied","Data":"22a4fbc0d8850ff8cb3b3d1c8d45a8dfb532d29aacb8c67c0d62180e91ce604f"} Oct 31 13:18:41.739119 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.738950 5693 scope.go:122] "RemoveContainer" containerID="a7f0da2e9965bcc2978a2cd7beea63fa718a1a2cad77c5cca8961754e21f978a" Oct 31 13:18:41.751433 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.751413 5693 scope.go:122] "RemoveContainer" containerID="e4724401ffda94aaa10ed5c3cb501889a22d50f9b1ba1e9ba5fe6d132089d8b0" Oct 31 13:18:41.761780 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.761368 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr" podStartSLOduration=2.76135359 podStartE2EDuration="2.76135359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:18:41.755139576 +0000 UTC m=+2772.462725871" watchObservedRunningTime="2026-10-31 13:18:41.76135359 +0000 UTC m=+2772.468939885" Oct 31 13:18:41.774762 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.774628 5693 scope.go:122] "RemoveContainer" containerID="a7f0da2e9965bcc2978a2cd7beea63fa718a1a2cad77c5cca8961754e21f978a" Oct 31 13:18:41.782700 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:18:41.779298 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7f0da2e9965bcc2978a2cd7beea63fa718a1a2cad77c5cca8961754e21f978a\": container with ID starting with a7f0da2e9965bcc2978a2cd7beea63fa718a1a2cad77c5cca8961754e21f978a not found: ID does not exist" containerID="a7f0da2e9965bcc2978a2cd7beea63fa718a1a2cad77c5cca8961754e21f978a" Oct 31 13:18:41.782700 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.779340 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7f0da2e9965bcc2978a2cd7beea63fa718a1a2cad77c5cca8961754e21f978a"} err="failed to get container status \"a7f0da2e9965bcc2978a2cd7beea63fa718a1a2cad77c5cca8961754e21f978a\": rpc error: code = NotFound desc = could not find container \"a7f0da2e9965bcc2978a2cd7beea63fa718a1a2cad77c5cca8961754e21f978a\": container with ID starting with a7f0da2e9965bcc2978a2cd7beea63fa718a1a2cad77c5cca8961754e21f978a not found: ID does not exist" Oct 31 13:18:41.782700 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.779366 5693 scope.go:122] "RemoveContainer" containerID="e4724401ffda94aaa10ed5c3cb501889a22d50f9b1ba1e9ba5fe6d132089d8b0" Oct 31 13:18:41.782700 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:18:41.779906 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4724401ffda94aaa10ed5c3cb501889a22d50f9b1ba1e9ba5fe6d132089d8b0\": container with ID starting with e4724401ffda94aaa10ed5c3cb501889a22d50f9b1ba1e9ba5fe6d132089d8b0 not found: ID does not exist" containerID="e4724401ffda94aaa10ed5c3cb501889a22d50f9b1ba1e9ba5fe6d132089d8b0" Oct 31 13:18:41.782700 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.779939 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4724401ffda94aaa10ed5c3cb501889a22d50f9b1ba1e9ba5fe6d132089d8b0"} err="failed to get container status \"e4724401ffda94aaa10ed5c3cb501889a22d50f9b1ba1e9ba5fe6d132089d8b0\": rpc error: code = NotFound desc = could not find container \"e4724401ffda94aaa10ed5c3cb501889a22d50f9b1ba1e9ba5fe6d132089d8b0\": container with ID starting with e4724401ffda94aaa10ed5c3cb501889a22d50f9b1ba1e9ba5fe6d132089d8b0 not found: ID does not exist" Oct 31 13:18:41.794128 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.794095 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz"] Oct 31 13:18:41.798585 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:41.798558 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-nhnnz"] Oct 31 13:18:43.411493 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:43.411447 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49a12158-0b73-4dbe-86ee-42b72e883cee" path="/var/lib/kubelet/pods/49a12158-0b73-4dbe-86ee-42b72e883cee/volumes" Oct 31 13:18:47.135906 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:47.135866 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj"] Oct 31 13:18:47.137287 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:47.137259 5693 cpu_manager.go:414] "containerMap: removing container" podUID="2f674e1e-7ff4-4923-b32a-fab0fa51d390" containerName="payload-processing" Oct 31 13:18:47.137287 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:47.137285 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f674e1e-7ff4-4923-b32a-fab0fa51d390" containerName="payload-processing" Oct 31 13:18:47.137524 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:47.137309 5693 cpu_manager.go:414] "containerMap: removing container" podUID="49a12158-0b73-4dbe-86ee-42b72e883cee" containerName="istio-proxy" Oct 31 13:18:47.137524 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:47.137318 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49a12158-0b73-4dbe-86ee-42b72e883cee" containerName="istio-proxy" Oct 31 13:18:47.137524 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:47.137349 5693 cpu_manager.go:414] "containerMap: removing container" podUID="49a12158-0b73-4dbe-86ee-42b72e883cee" containerName="istio-proxy" Oct 31 13:18:47.137524 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:47.137361 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49a12158-0b73-4dbe-86ee-42b72e883cee" containerName="istio-proxy" Oct 31 13:18:47.137727 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:47.137679 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f674e1e-7ff4-4923-b32a-fab0fa51d390" containerName="payload-processing" Oct 31 13:18:47.137727 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:47.137697 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="49a12158-0b73-4dbe-86ee-42b72e883cee" containerName="istio-proxy" Oct 31 13:18:47.137727 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:47.137707 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="49a12158-0b73-4dbe-86ee-42b72e883cee" containerName="istio-proxy" Oct 31 13:18:47.144527 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:47.144502 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj" Oct 31 13:18:47.153717 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:47.152062 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj"] Oct 31 13:18:47.165901 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:47.165864 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-842ac9\"/\"payload-processing-external-model-plugins-e2e-extmod-a-842ac9\"" Oct 31 13:18:47.166077 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:47.165940 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-842ac9\"/\"kube-root-ca.crt\"" Oct 31 13:18:47.166077 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:47.166000 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-842ac9\"/\"openshift-service-ca.crt\"" Oct 31 13:18:47.166077 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:47.166070 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-842ac9\"/\"payload-processing-external-model-e2e--64249880-dockercfg-7lb6g\"" Oct 31 13:18:47.166273 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:47.166123 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-842ac9\"/\"routing-overlay\"" Oct 31 13:18:47.166273 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:47.166158 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-842ac9\"/\"e2e-cmp-legacy-95e093-provider-cred\"" Oct 31 13:18:47.224665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:47.224613 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxcfk\" (UniqueName: \"kubernetes.io/projected/77ebd766-a4ad-4701-a3d3-76f8c06b9e45-kube-api-access-cxcfk\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj\" (UID: \"77ebd766-a4ad-4701-a3d3-76f8c06b9e45\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj" Oct 31 13:18:47.224865 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:47.224823 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/77ebd766-a4ad-4701-a3d3-76f8c06b9e45-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj\" (UID: \"77ebd766-a4ad-4701-a3d3-76f8c06b9e45\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj" Oct 31 13:18:47.224984 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:47.224960 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/77ebd766-a4ad-4701-a3d3-76f8c06b9e45-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj\" (UID: \"77ebd766-a4ad-4701-a3d3-76f8c06b9e45\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj" Oct 31 13:18:47.225128 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:47.225109 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77ebd766-a4ad-4701-a3d3-76f8c06b9e45-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj\" (UID: \"77ebd766-a4ad-4701-a3d3-76f8c06b9e45\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj" Oct 31 13:18:47.327692 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:47.327013 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxcfk\" (UniqueName: \"kubernetes.io/projected/77ebd766-a4ad-4701-a3d3-76f8c06b9e45-kube-api-access-cxcfk\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj\" (UID: \"77ebd766-a4ad-4701-a3d3-76f8c06b9e45\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj" Oct 31 13:18:47.327692 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:47.327100 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/77ebd766-a4ad-4701-a3d3-76f8c06b9e45-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj\" (UID: \"77ebd766-a4ad-4701-a3d3-76f8c06b9e45\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj" Oct 31 13:18:47.327692 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:47.327199 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/77ebd766-a4ad-4701-a3d3-76f8c06b9e45-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj\" (UID: \"77ebd766-a4ad-4701-a3d3-76f8c06b9e45\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj" Oct 31 13:18:47.327692 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:47.327308 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77ebd766-a4ad-4701-a3d3-76f8c06b9e45-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj\" (UID: \"77ebd766-a4ad-4701-a3d3-76f8c06b9e45\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj" Oct 31 13:18:47.328136 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:47.328107 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/77ebd766-a4ad-4701-a3d3-76f8c06b9e45-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj\" (UID: \"77ebd766-a4ad-4701-a3d3-76f8c06b9e45\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj" Oct 31 13:18:47.330672 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:47.329573 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77ebd766-a4ad-4701-a3d3-76f8c06b9e45-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj\" (UID: \"77ebd766-a4ad-4701-a3d3-76f8c06b9e45\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj" Oct 31 13:18:47.346693 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:47.332535 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/77ebd766-a4ad-4701-a3d3-76f8c06b9e45-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj\" (UID: \"77ebd766-a4ad-4701-a3d3-76f8c06b9e45\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj" Oct 31 13:18:47.364660 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:47.363702 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxcfk\" (UniqueName: \"kubernetes.io/projected/77ebd766-a4ad-4701-a3d3-76f8c06b9e45-kube-api-access-cxcfk\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj\" (UID: \"77ebd766-a4ad-4701-a3d3-76f8c06b9e45\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj" Oct 31 13:18:47.481665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:47.479983 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj" Oct 31 13:18:47.908849 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:47.908821 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj"] Oct 31 13:18:48.707607 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:18:48.707568 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:18:48.857694 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:48.857623 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj" event={"ID":"77ebd766-a4ad-4701-a3d3-76f8c06b9e45","Type":"ContainerStarted","Data":"1b3b176b782be38d8ff6a3d452f728fdbaaa5e00dc3cfbe67985fa77035b1a00"} Oct 31 13:18:48.857876 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:48.857722 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj" event={"ID":"77ebd766-a4ad-4701-a3d3-76f8c06b9e45","Type":"ContainerStarted","Data":"ce21386962fdc5688e10f3cb5144fbf50a557573d43ecf3ea4b7cb28e4044bb0"} Oct 31 13:18:48.857876 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:48.857774 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj" Oct 31 13:18:48.905367 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:48.905298 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj" podStartSLOduration=1.905278094 podStartE2EDuration="1.905278094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:18:48.901975662 +0000 UTC m=+2779.609561958" watchObservedRunningTime="2026-10-31 13:18:48.905278094 +0000 UTC m=+2779.612864390" Oct 31 13:18:49.703694 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:49.703601 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8" Oct 31 13:18:49.837912 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:18:49.837863 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-5dc570/agc-cred-b4bf09cb0482: secret "agc-cred-b4bf09cb0482" not found Oct 31 13:18:49.838684 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:18:49.838610 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8: secret "agc-cred-b4bf09cb0482" not found Oct 31 13:18:49.838816 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:18:49.838752 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a8e4aec-7674-424f-8b17-100903e002fd-provider-credentials podName:9a8e4aec-7674-424f-8b17-100903e002fd nodeName:}" failed. No retries permitted until 2026-10-31 13:18:50.338728902 +0000 UTC m=+2781.046315176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9a8e4aec-7674-424f-8b17-100903e002fd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8" (UID: "9a8e4aec-7674-424f-8b17-100903e002fd") : secret "agc-cred-b4bf09cb0482" not found Oct 31 13:18:50.371797 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:18:50.370361 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-5dc570/agc-cred-b4bf09cb0482: secret "agc-cred-b4bf09cb0482" not found Oct 31 13:18:50.371797 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:18:50.370395 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8: secret "agc-cred-b4bf09cb0482" not found Oct 31 13:18:50.371797 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:18:50.371727 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a8e4aec-7674-424f-8b17-100903e002fd-provider-credentials podName:9a8e4aec-7674-424f-8b17-100903e002fd nodeName:}" failed. No retries permitted until 2026-10-31 13:18:51.371412672 +0000 UTC m=+2782.078998947 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9a8e4aec-7674-424f-8b17-100903e002fd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8" (UID: "9a8e4aec-7674-424f-8b17-100903e002fd") : secret "agc-cred-b4bf09cb0482" not found Oct 31 13:18:51.408768 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:18:51.408734 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-5dc570/agc-cred-b4bf09cb0482: secret "agc-cred-b4bf09cb0482" not found Oct 31 13:18:51.408768 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:18:51.408757 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8: secret "agc-cred-b4bf09cb0482" not found Oct 31 13:18:51.409254 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:18:51.408816 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a8e4aec-7674-424f-8b17-100903e002fd-provider-credentials podName:9a8e4aec-7674-424f-8b17-100903e002fd nodeName:}" failed. No retries permitted until 2026-10-31 13:18:53.408803021 +0000 UTC m=+2784.116389295 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9a8e4aec-7674-424f-8b17-100903e002fd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8" (UID: "9a8e4aec-7674-424f-8b17-100903e002fd") : secret "agc-cred-b4bf09cb0482" not found Oct 31 13:18:52.802258 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:52.802222 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr" Oct 31 13:18:52.874062 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:52.874025 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8"] Oct 31 13:18:53.453337 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:18:53.453302 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-5dc570/agc-cred-b4bf09cb0482: secret "agc-cred-b4bf09cb0482" not found Oct 31 13:18:53.453337 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:18:53.453329 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8: secret "agc-cred-b4bf09cb0482" not found Oct 31 13:18:53.453579 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:18:53.453398 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a8e4aec-7674-424f-8b17-100903e002fd-provider-credentials podName:9a8e4aec-7674-424f-8b17-100903e002fd nodeName:}" failed. No retries permitted until 2026-10-31 13:18:57.453382682 +0000 UTC m=+2788.160968955 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9a8e4aec-7674-424f-8b17-100903e002fd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8" (UID: "9a8e4aec-7674-424f-8b17-100903e002fd") : secret "agc-cred-b4bf09cb0482" not found Oct 31 13:18:56.046673 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:56.031312 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg"] Oct 31 13:18:56.112691 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:56.107281 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg" Oct 31 13:18:56.112691 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:56.111097 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg"] Oct 31 13:18:56.124662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:56.117883 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-29e3a0\"/\"openshift-service-ca.crt\"" Oct 31 13:18:56.133663 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:56.127630 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-29e3a0\"/\"payload-processing-external-model-e2e--e205175e-dockercfg-ssrqv\"" Oct 31 13:18:56.133663 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:56.128840 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4340ef1f-61ea-4077-830d-514d4d2e4a25-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg\" (UID: \"4340ef1f-61ea-4077-830d-514d4d2e4a25\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg" Oct 31 13:18:56.133663 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:56.128970 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4340ef1f-61ea-4077-830d-514d4d2e4a25-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg\" (UID: \"4340ef1f-61ea-4077-830d-514d4d2e4a25\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg" Oct 31 13:18:56.133663 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:56.129023 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4340ef1f-61ea-4077-830d-514d4d2e4a25-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg\" (UID: \"4340ef1f-61ea-4077-830d-514d4d2e4a25\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg" Oct 31 13:18:56.133663 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:56.129118 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9v95\" (UniqueName: \"kubernetes.io/projected/4340ef1f-61ea-4077-830d-514d4d2e4a25-kube-api-access-t9v95\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg\" (UID: \"4340ef1f-61ea-4077-830d-514d4d2e4a25\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg" Oct 31 13:18:56.133663 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:56.129711 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-29e3a0\"/\"payload-processing-external-model-plugins-e2e-extmod-b-29e3a0\"" Oct 31 13:18:56.215696 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:56.215667 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-29e3a0\"/\"routing-overlay\"" Oct 31 13:18:56.215882 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:56.215803 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-29e3a0\"/\"kube-root-ca.crt\"" Oct 31 13:18:56.215882 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:56.215826 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-29e3a0\"/\"agc-cred-c4174febc8d5\"" Oct 31 13:18:56.230767 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:56.230739 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4340ef1f-61ea-4077-830d-514d4d2e4a25-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg\" (UID: \"4340ef1f-61ea-4077-830d-514d4d2e4a25\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg" Oct 31 13:18:56.230961 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:56.230811 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4340ef1f-61ea-4077-830d-514d4d2e4a25-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg\" (UID: \"4340ef1f-61ea-4077-830d-514d4d2e4a25\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg" Oct 31 13:18:56.230961 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:56.230842 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4340ef1f-61ea-4077-830d-514d4d2e4a25-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg\" (UID: \"4340ef1f-61ea-4077-830d-514d4d2e4a25\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg" Oct 31 13:18:56.230961 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:56.230896 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9v95\" (UniqueName: \"kubernetes.io/projected/4340ef1f-61ea-4077-830d-514d4d2e4a25-kube-api-access-t9v95\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg\" (UID: \"4340ef1f-61ea-4077-830d-514d4d2e4a25\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg" Oct 31 13:18:56.231677 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:56.231635 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4340ef1f-61ea-4077-830d-514d4d2e4a25-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg\" (UID: \"4340ef1f-61ea-4077-830d-514d4d2e4a25\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg" Oct 31 13:18:56.231677 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:56.231659 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4340ef1f-61ea-4077-830d-514d4d2e4a25-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg\" (UID: \"4340ef1f-61ea-4077-830d-514d4d2e4a25\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg" Oct 31 13:18:56.233883 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:56.233858 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4340ef1f-61ea-4077-830d-514d4d2e4a25-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg\" (UID: \"4340ef1f-61ea-4077-830d-514d4d2e4a25\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg" Oct 31 13:18:56.248942 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:56.248914 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9v95\" (UniqueName: \"kubernetes.io/projected/4340ef1f-61ea-4077-830d-514d4d2e4a25-kube-api-access-t9v95\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg\" (UID: \"4340ef1f-61ea-4077-830d-514d4d2e4a25\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg" Oct 31 13:18:56.516019 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:56.514019 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg" Oct 31 13:18:56.960870 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:18:56.960844 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4340ef1f_61ea_4077_830d_514d4d2e4a25.slice/crio-701d902523b266a7dfbf19090b243c2ad37b51499bf40f5f4a3c33cac974100f WatchSource:0}: Error finding container 701d902523b266a7dfbf19090b243c2ad37b51499bf40f5f4a3c33cac974100f: Status 404 returned error can't find the container with id 701d902523b266a7dfbf19090b243c2ad37b51499bf40f5f4a3c33cac974100f Oct 31 13:18:56.961512 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:56.961490 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg"] Oct 31 13:18:56.975309 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:56.975281 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg" event={"ID":"4340ef1f-61ea-4077-830d-514d4d2e4a25","Type":"ContainerStarted","Data":"701d902523b266a7dfbf19090b243c2ad37b51499bf40f5f4a3c33cac974100f"} Oct 31 13:18:57.457715 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:18:57.457678 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-5dc570/agc-cred-b4bf09cb0482: secret "agc-cred-b4bf09cb0482" not found Oct 31 13:18:57.457715 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:18:57.457709 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8: secret "agc-cred-b4bf09cb0482" not found Oct 31 13:18:57.458158 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:18:57.457782 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a8e4aec-7674-424f-8b17-100903e002fd-provider-credentials podName:9a8e4aec-7674-424f-8b17-100903e002fd nodeName:}" failed. No retries permitted until 2026-10-31 13:19:05.457765994 +0000 UTC m=+2796.165352269 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9a8e4aec-7674-424f-8b17-100903e002fd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8" (UID: "9a8e4aec-7674-424f-8b17-100903e002fd") : secret "agc-cred-b4bf09cb0482" not found Oct 31 13:18:57.649826 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:57.649787 5693 patch_prober.go:29] interesting pod/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:18:57.650015 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:57.649853 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" podUID="a022831f-b584-43fe-91a7-7f541c066ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:18:57.875068 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:57.874977 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8" podUID="9a8e4aec-7674-424f-8b17-100903e002fd" containerName="payload-processing" containerID="cri-o://4abe26b5fd07d2fc0aaa14da8a6e4d7ca39386a4d3bf9bfd4f2033b35de31c2b" gracePeriod=25 Oct 31 13:18:57.988098 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:57.988064 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg" event={"ID":"4340ef1f-61ea-4077-830d-514d4d2e4a25","Type":"ContainerStarted","Data":"598cc398864270cac424087f8e56e18f1d2483cb8d718d3f52a06722880ebe65"} Oct 31 13:18:57.988331 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:57.988151 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg" Oct 31 13:18:58.017537 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:58.017484 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg" podStartSLOduration=3.017468347 podStartE2EDuration="3.017468347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:18:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:18:58.010695146 +0000 UTC m=+2788.718281440" watchObservedRunningTime="2026-10-31 13:18:58.017468347 +0000 UTC m=+2788.725054662" Oct 31 13:18:58.209472 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:58.209438 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8" Oct 31 13:18:58.270769 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:58.270735 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a8e4aec-7674-424f-8b17-100903e002fd-kube-api-access-nhw9n\" (UniqueName: \"kubernetes.io/projected/9a8e4aec-7674-424f-8b17-100903e002fd-kube-api-access-nhw9n\") pod \"9a8e4aec-7674-424f-8b17-100903e002fd\" (UID: \"9a8e4aec-7674-424f-8b17-100903e002fd\") " Oct 31 13:18:58.594802 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:58.270832 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a8e4aec-7674-424f-8b17-100903e002fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a8e4aec-7674-424f-8b17-100903e002fd-plugins-config-volume\") pod \"9a8e4aec-7674-424f-8b17-100903e002fd\" (UID: \"9a8e4aec-7674-424f-8b17-100903e002fd\") " Oct 31 13:18:58.594802 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:58.270874 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a8e4aec-7674-424f-8b17-100903e002fd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9a8e4aec-7674-424f-8b17-100903e002fd-routing-overlay-volume\") pod \"9a8e4aec-7674-424f-8b17-100903e002fd\" (UID: \"9a8e4aec-7674-424f-8b17-100903e002fd\") " Oct 31 13:18:58.594802 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:58.271122 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a8e4aec-7674-424f-8b17-100903e002fd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9a8e4aec-7674-424f-8b17-100903e002fd-provider-credentials\") pod \"9a8e4aec-7674-424f-8b17-100903e002fd\" (UID: \"9a8e4aec-7674-424f-8b17-100903e002fd\") " Oct 31 13:18:58.594802 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:58.271269 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a8e4aec-7674-424f-8b17-100903e002fd-routing-overlay-volume" pod "9a8e4aec-7674-424f-8b17-100903e002fd" (UID: "9a8e4aec-7674-424f-8b17-100903e002fd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:18:58.594802 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:58.271279 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a8e4aec-7674-424f-8b17-100903e002fd-plugins-config-volume" pod "9a8e4aec-7674-424f-8b17-100903e002fd" (UID: "9a8e4aec-7674-424f-8b17-100903e002fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:18:58.594802 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:58.271744 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a8e4aec-7674-424f-8b17-100903e002fd-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:58.594802 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:58.271760 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9a8e4aec-7674-424f-8b17-100903e002fd-routing-overlay-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:58.594802 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:58.273358 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a8e4aec-7674-424f-8b17-100903e002fd-provider-credentials" pod "9a8e4aec-7674-424f-8b17-100903e002fd" (UID: "9a8e4aec-7674-424f-8b17-100903e002fd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:18:58.594802 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:58.273413 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a8e4aec-7674-424f-8b17-100903e002fd-kube-api-access-nhw9n" pod "9a8e4aec-7674-424f-8b17-100903e002fd" (UID: "9a8e4aec-7674-424f-8b17-100903e002fd"). InnerVolumeSpecName "kube-api-access-nhw9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:18:58.594802 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:58.373513 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9a8e4aec-7674-424f-8b17-100903e002fd-provider-credentials\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:58.594802 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:58.373537 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhw9n\" (UniqueName: \"kubernetes.io/projected/9a8e4aec-7674-424f-8b17-100903e002fd-kube-api-access-nhw9n\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:58.999236 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:58.999206 5693 generic.go:356] "Generic (PLEG): container finished" podID="9a8e4aec-7674-424f-8b17-100903e002fd" containerID="4abe26b5fd07d2fc0aaa14da8a6e4d7ca39386a4d3bf9bfd4f2033b35de31c2b" exitCode=0 Oct 31 13:18:58.999464 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:58.999255 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8" Oct 31 13:18:58.999464 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:58.999303 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8" event={"ID":"9a8e4aec-7674-424f-8b17-100903e002fd","Type":"ContainerDied","Data":"4abe26b5fd07d2fc0aaa14da8a6e4d7ca39386a4d3bf9bfd4f2033b35de31c2b"} Oct 31 13:18:58.999464 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:58.999363 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8" event={"ID":"9a8e4aec-7674-424f-8b17-100903e002fd","Type":"ContainerDied","Data":"70d6d05e617176cc4e7ae7961ebe826763587c61a1a6105383e9234ebd95050e"} Oct 31 13:18:58.999464 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:58.999392 5693 scope.go:122] "RemoveContainer" containerID="4abe26b5fd07d2fc0aaa14da8a6e4d7ca39386a4d3bf9bfd4f2033b35de31c2b" Oct 31 13:18:59.001730 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:59.001696 5693 generic.go:356] "Generic (PLEG): container finished" podID="a022831f-b584-43fe-91a7-7f541c066ace" containerID="c140569b2bd55c4082acba10ca650cd3758e8d04deb825997848a11005d1a9f8" exitCode=0 Oct 31 13:18:59.001834 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:59.001770 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" event={"ID":"a022831f-b584-43fe-91a7-7f541c066ace","Type":"ContainerDied","Data":"c140569b2bd55c4082acba10ca650cd3758e8d04deb825997848a11005d1a9f8"} Oct 31 13:18:59.002631 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:59.002598 5693 scope.go:122] "RemoveContainer" containerID="c140569b2bd55c4082acba10ca650cd3758e8d04deb825997848a11005d1a9f8" Oct 31 13:18:59.008715 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:59.007092 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:18:59.008715 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:59.007186 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:18:59.008715 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:59.007233 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:18:59.018082 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:59.018060 5693 scope.go:122] "RemoveContainer" containerID="4abe26b5fd07d2fc0aaa14da8a6e4d7ca39386a4d3bf9bfd4f2033b35de31c2b" Oct 31 13:18:59.018415 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:18:59.018392 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4abe26b5fd07d2fc0aaa14da8a6e4d7ca39386a4d3bf9bfd4f2033b35de31c2b\": container with ID starting with 4abe26b5fd07d2fc0aaa14da8a6e4d7ca39386a4d3bf9bfd4f2033b35de31c2b not found: ID does not exist" containerID="4abe26b5fd07d2fc0aaa14da8a6e4d7ca39386a4d3bf9bfd4f2033b35de31c2b" Oct 31 13:18:59.018495 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:59.018425 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4abe26b5fd07d2fc0aaa14da8a6e4d7ca39386a4d3bf9bfd4f2033b35de31c2b"} err="failed to get container status \"4abe26b5fd07d2fc0aaa14da8a6e4d7ca39386a4d3bf9bfd4f2033b35de31c2b\": rpc error: code = NotFound desc = could not find container \"4abe26b5fd07d2fc0aaa14da8a6e4d7ca39386a4d3bf9bfd4f2033b35de31c2b\": container with ID starting with 4abe26b5fd07d2fc0aaa14da8a6e4d7ca39386a4d3bf9bfd4f2033b35de31c2b not found: ID does not exist" Oct 31 13:18:59.065435 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:59.064019 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8"] Oct 31 13:18:59.070898 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:59.070867 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8fqb4k8"] Oct 31 13:18:59.412350 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:59.412317 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a8e4aec-7674-424f-8b17-100903e002fd" path="/var/lib/kubelet/pods/9a8e4aec-7674-424f-8b17-100903e002fd/volumes" Oct 31 13:18:59.869546 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:18:59.869465 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj" Oct 31 13:19:00.022666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:00.018969 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" event={"ID":"a022831f-b584-43fe-91a7-7f541c066ace","Type":"ContainerStarted","Data":"e2d6bcbc7bf35b16702d1f2705a242775af5ab8cd6e0569ad46313a327d6b8b2"} Oct 31 13:19:00.807845 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:00.807813 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" Oct 31 13:19:00.808523 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:00.808495 5693 patch_prober.go:29] interesting pod/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 31 13:19:00.808573 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:00.808554 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" podUID="a022831f-b584-43fe-91a7-7f541c066ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 31 13:19:01.808529 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:01.808491 5693 patch_prober.go:29] interesting pod/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 31 13:19:01.809067 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:01.808552 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" podUID="a022831f-b584-43fe-91a7-7f541c066ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 31 13:19:02.808601 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:02.808568 5693 patch_prober.go:29] interesting pod/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 31 13:19:02.809047 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:02.808623 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" podUID="a022831f-b584-43fe-91a7-7f541c066ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 31 13:19:03.809845 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:03.808491 5693 patch_prober.go:29] interesting pod/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 31 13:19:03.809845 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:03.808556 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" podUID="a022831f-b584-43fe-91a7-7f541c066ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 31 13:19:04.867670 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:04.859408 5693 patch_prober.go:29] interesting pod/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 31 13:19:04.867670 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:04.859472 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" podUID="a022831f-b584-43fe-91a7-7f541c066ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 31 13:19:05.809049 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:05.809007 5693 patch_prober.go:29] interesting pod/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 31 13:19:05.809257 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:05.809091 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" podUID="a022831f-b584-43fe-91a7-7f541c066ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 31 13:19:06.812970 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:06.812935 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" Oct 31 13:19:06.813695 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:06.813634 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" Oct 31 13:19:06.814708 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:06.814687 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" Oct 31 13:19:09.005778 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:09.005676 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg" Oct 31 13:19:09.243413 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:09.243377 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5"] Oct 31 13:19:09.244312 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:09.244288 5693 cpu_manager.go:414] "containerMap: removing container" podUID="9a8e4aec-7674-424f-8b17-100903e002fd" containerName="payload-processing" Oct 31 13:19:09.244312 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:09.244310 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a8e4aec-7674-424f-8b17-100903e002fd" containerName="payload-processing" Oct 31 13:19:09.244653 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:09.244621 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a8e4aec-7674-424f-8b17-100903e002fd" containerName="payload-processing" Oct 31 13:19:09.250012 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:09.249992 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5" Oct 31 13:19:09.260866 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:09.260843 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2ef686\"/\"agc-cred-5104fff0eb2b\"" Oct 31 13:19:09.260969 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:09.260950 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2ef686\"/\"kube-root-ca.crt\"" Oct 31 13:19:09.261033 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:09.261015 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2ef686\"/\"routing-overlay\"" Oct 31 13:19:09.266730 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:09.261100 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2ef686\"/\"openshift-service-ca.crt\"" Oct 31 13:19:09.266730 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:09.261156 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2ef686\"/\"payload-processing-external-model-plugins-e2e-extmod-a-2ef686\"" Oct 31 13:19:09.266730 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:09.261251 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2ef686\"/\"payload-processing-external-model-e2e--362fde4e-dockercfg-l7p2q\"" Oct 31 13:19:09.266730 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:09.261305 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2ef686\"/\"agc-cred-811c6a0d5a4c\"" Oct 31 13:19:09.266730 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:09.263752 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5"] Oct 31 13:19:09.330542 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:09.330506 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17b2a30d-fd18-4144-9212-7ef5586a629e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5\" (UID: \"17b2a30d-fd18-4144-9212-7ef5586a629e\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5" Oct 31 13:19:09.330737 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:09.330551 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/17b2a30d-fd18-4144-9212-7ef5586a629e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5\" (UID: \"17b2a30d-fd18-4144-9212-7ef5586a629e\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5" Oct 31 13:19:09.330737 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:09.330693 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jqzr\" (UniqueName: \"kubernetes.io/projected/17b2a30d-fd18-4144-9212-7ef5586a629e-kube-api-access-4jqzr\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5\" (UID: \"17b2a30d-fd18-4144-9212-7ef5586a629e\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5" Oct 31 13:19:09.330832 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:09.330752 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/17b2a30d-fd18-4144-9212-7ef5586a629e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5\" (UID: \"17b2a30d-fd18-4144-9212-7ef5586a629e\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5" Oct 31 13:19:09.433041 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:09.433007 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17b2a30d-fd18-4144-9212-7ef5586a629e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5\" (UID: \"17b2a30d-fd18-4144-9212-7ef5586a629e\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5" Oct 31 13:19:09.433041 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:09.433044 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/17b2a30d-fd18-4144-9212-7ef5586a629e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5\" (UID: \"17b2a30d-fd18-4144-9212-7ef5586a629e\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5" Oct 31 13:19:09.433320 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:09.433068 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jqzr\" (UniqueName: \"kubernetes.io/projected/17b2a30d-fd18-4144-9212-7ef5586a629e-kube-api-access-4jqzr\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5\" (UID: \"17b2a30d-fd18-4144-9212-7ef5586a629e\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5" Oct 31 13:19:09.433320 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:09.433119 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/17b2a30d-fd18-4144-9212-7ef5586a629e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5\" (UID: \"17b2a30d-fd18-4144-9212-7ef5586a629e\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5" Oct 31 13:19:09.433872 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:09.433847 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/17b2a30d-fd18-4144-9212-7ef5586a629e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5\" (UID: \"17b2a30d-fd18-4144-9212-7ef5586a629e\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5" Oct 31 13:19:09.433979 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:09.433960 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17b2a30d-fd18-4144-9212-7ef5586a629e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5\" (UID: \"17b2a30d-fd18-4144-9212-7ef5586a629e\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5" Oct 31 13:19:09.435895 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:09.435878 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/17b2a30d-fd18-4144-9212-7ef5586a629e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5\" (UID: \"17b2a30d-fd18-4144-9212-7ef5586a629e\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5" Oct 31 13:19:09.445764 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:09.445743 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jqzr\" (UniqueName: \"kubernetes.io/projected/17b2a30d-fd18-4144-9212-7ef5586a629e-kube-api-access-4jqzr\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5\" (UID: \"17b2a30d-fd18-4144-9212-7ef5586a629e\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5" Oct 31 13:19:09.573702 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:09.573600 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5" Oct 31 13:19:09.864378 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:19:09.864345 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17b2a30d_fd18_4144_9212_7ef5586a629e.slice/crio-a3d9859fe83d846e162ec4e35e4d2bdc065b9eddd44237ebaeba0a2949f00feb WatchSource:0}: Error finding container a3d9859fe83d846e162ec4e35e4d2bdc065b9eddd44237ebaeba0a2949f00feb: Status 404 returned error can't find the container with id a3d9859fe83d846e162ec4e35e4d2bdc065b9eddd44237ebaeba0a2949f00feb Oct 31 13:19:09.864533 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:09.864394 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5"] Oct 31 13:19:10.321530 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:10.321493 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5" event={"ID":"17b2a30d-fd18-4144-9212-7ef5586a629e","Type":"ContainerStarted","Data":"f7e31a2b4ca898fd3c4d1a7296df9111b9a4d77ee305826307febbfd8dee39bb"} Oct 31 13:19:10.321959 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:10.321554 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5" event={"ID":"17b2a30d-fd18-4144-9212-7ef5586a629e","Type":"ContainerStarted","Data":"a3d9859fe83d846e162ec4e35e4d2bdc065b9eddd44237ebaeba0a2949f00feb"} Oct 31 13:19:10.321959 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:10.321583 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5" Oct 31 13:19:10.372239 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:10.372177 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5" podStartSLOduration=1.372156132 podStartE2EDuration="1.372156132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:19:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:19:10.359512782 +0000 UTC m=+2801.067099081" watchObservedRunningTime="2026-10-31 13:19:10.372156132 +0000 UTC m=+2801.079742429" Oct 31 13:19:18.569461 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:18.568907 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6"] Oct 31 13:19:18.569461 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:18.569262 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" podUID="8afd9c8e-49db-4c62-8af3-4fbed8d2d954" containerName="istio-proxy" containerID="cri-o://2dd0d980673356e52c7665d3ff54c25de45a867171a0ae907a1a844f8a6b6445" gracePeriod=30 Oct 31 13:19:18.693319 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:19:18.693211 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:19:21.335446 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:21.335417 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5" Oct 31 13:19:23.919037 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:23.919005 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" Oct 31 13:19:24.028350 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:24.028314 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-workload-socket\") pod \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\" (UID: \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\") " Oct 31 13:19:24.028549 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:24.028472 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-istio-token\" (UniqueName: \"kubernetes.io/projected/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-istio-token\") pod \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\" (UID: \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\") " Oct 31 13:19:24.028625 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:24.028604 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-istio-data\") pod \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\" (UID: \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\") " Oct 31 13:19:24.028625 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:24.028605 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-workload-socket" pod "8afd9c8e-49db-4c62-8af3-4fbed8d2d954" (UID: "8afd9c8e-49db-4c62-8af3-4fbed8d2d954"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:19:24.028802 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:24.028735 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-credential-socket\") pod \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\" (UID: \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\") " Oct 31 13:19:24.028802 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:24.028793 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-istiod-ca-cert\") pod \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\" (UID: \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\") " Oct 31 13:19:24.028926 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:24.028854 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-workload-certs\") pod \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\" (UID: \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\") " Oct 31 13:19:24.028926 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:24.028855 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-istio-data" pod "8afd9c8e-49db-4c62-8af3-4fbed8d2d954" (UID: "8afd9c8e-49db-4c62-8af3-4fbed8d2d954"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:19:24.028926 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:24.028901 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-istio-envoy\") pod \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\" (UID: \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\") " Oct 31 13:19:24.029083 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:24.028959 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-istio-podinfo\") pod \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\" (UID: \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\") " Oct 31 13:19:24.029083 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:24.028964 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-credential-socket" pod "8afd9c8e-49db-4c62-8af3-4fbed8d2d954" (UID: "8afd9c8e-49db-4c62-8af3-4fbed8d2d954"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:19:24.029195 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:24.029118 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-kube-api-access-sxsbc\" (UniqueName: \"kubernetes.io/projected/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-kube-api-access-sxsbc\") pod \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\" (UID: \"8afd9c8e-49db-4c62-8af3-4fbed8d2d954\") " Oct 31 13:19:24.029195 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:24.029142 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-workload-certs" pod "8afd9c8e-49db-4c62-8af3-4fbed8d2d954" (UID: "8afd9c8e-49db-4c62-8af3-4fbed8d2d954"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:19:24.029195 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:24.029146 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-istiod-ca-cert" pod "8afd9c8e-49db-4c62-8af3-4fbed8d2d954" (UID: "8afd9c8e-49db-4c62-8af3-4fbed8d2d954"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:19:24.030455 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:24.030426 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:24.030455 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:24.030457 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:24.030700 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:24.030472 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:24.030700 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:24.030486 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:24.030700 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:24.030500 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:24.030938 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:24.030916 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-istio-token" pod "8afd9c8e-49db-4c62-8af3-4fbed8d2d954" (UID: "8afd9c8e-49db-4c62-8af3-4fbed8d2d954"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:19:24.031140 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:24.031113 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-istio-envoy" pod "8afd9c8e-49db-4c62-8af3-4fbed8d2d954" (UID: "8afd9c8e-49db-4c62-8af3-4fbed8d2d954"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:19:24.031348 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:24.031327 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-istio-podinfo" pod "8afd9c8e-49db-4c62-8af3-4fbed8d2d954" (UID: "8afd9c8e-49db-4c62-8af3-4fbed8d2d954"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:19:24.031669 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:24.031631 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-kube-api-access-sxsbc" pod "8afd9c8e-49db-4c62-8af3-4fbed8d2d954" (UID: "8afd9c8e-49db-4c62-8af3-4fbed8d2d954"). InnerVolumeSpecName "kube-api-access-sxsbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:19:24.132348 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:24.132249 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:24.132348 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:24.132297 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:24.132348 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:24.132308 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxsbc\" (UniqueName: \"kubernetes.io/projected/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-kube-api-access-sxsbc\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:24.132348 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:24.132319 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8afd9c8e-49db-4c62-8af3-4fbed8d2d954-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:24.464746 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:24.464705 5693 generic.go:356] "Generic (PLEG): container finished" podID="8afd9c8e-49db-4c62-8af3-4fbed8d2d954" containerID="2dd0d980673356e52c7665d3ff54c25de45a867171a0ae907a1a844f8a6b6445" exitCode=0 Oct 31 13:19:24.464953 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:24.464768 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" Oct 31 13:19:24.464953 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:24.464802 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" event={"ID":"8afd9c8e-49db-4c62-8af3-4fbed8d2d954","Type":"ContainerDied","Data":"2dd0d980673356e52c7665d3ff54c25de45a867171a0ae907a1a844f8a6b6445"} Oct 31 13:19:24.464953 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:24.464874 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6" event={"ID":"8afd9c8e-49db-4c62-8af3-4fbed8d2d954","Type":"ContainerDied","Data":"b7df648282208532807f8099472f86141e695ab55b5d0d35c4c318806c14dbaa"} Oct 31 13:19:24.464953 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:24.464905 5693 scope.go:122] "RemoveContainer" containerID="2dd0d980673356e52c7665d3ff54c25de45a867171a0ae907a1a844f8a6b6445" Oct 31 13:19:24.477867 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:24.477841 5693 scope.go:122] "RemoveContainer" containerID="2dd0d980673356e52c7665d3ff54c25de45a867171a0ae907a1a844f8a6b6445" Oct 31 13:19:24.478194 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:19:24.478172 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dd0d980673356e52c7665d3ff54c25de45a867171a0ae907a1a844f8a6b6445\": container with ID starting with 2dd0d980673356e52c7665d3ff54c25de45a867171a0ae907a1a844f8a6b6445 not found: ID does not exist" containerID="2dd0d980673356e52c7665d3ff54c25de45a867171a0ae907a1a844f8a6b6445" Oct 31 13:19:24.478276 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:24.478206 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd0d980673356e52c7665d3ff54c25de45a867171a0ae907a1a844f8a6b6445"} err="failed to get container status \"2dd0d980673356e52c7665d3ff54c25de45a867171a0ae907a1a844f8a6b6445\": rpc error: code = NotFound desc = could not find container \"2dd0d980673356e52c7665d3ff54c25de45a867171a0ae907a1a844f8a6b6445\": container with ID starting with 2dd0d980673356e52c7665d3ff54c25de45a867171a0ae907a1a844f8a6b6445 not found: ID does not exist" Oct 31 13:19:24.491611 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:24.491507 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6"] Oct 31 13:19:24.496979 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:24.496956 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-qtcm6"] Oct 31 13:19:25.411716 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:25.411682 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8afd9c8e-49db-4c62-8af3-4fbed8d2d954" path="/var/lib/kubelet/pods/8afd9c8e-49db-4c62-8af3-4fbed8d2d954/volumes" Oct 31 13:19:48.688854 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:19:48.688815 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:19:50.706245 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:50.706210 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg"] Oct 31 13:19:55.706584 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:55.706531 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg" podUID="4340ef1f-61ea-4077-830d-514d4d2e4a25" containerName="payload-processing" containerID="cri-o://598cc398864270cac424087f8e56e18f1d2483cb8d718d3f52a06722880ebe65" gracePeriod=25 Oct 31 13:19:56.005729 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:56.005703 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg" Oct 31 13:19:56.103292 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:56.103257 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4340ef1f-61ea-4077-830d-514d4d2e4a25-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4340ef1f-61ea-4077-830d-514d4d2e4a25-routing-overlay-volume\") pod \"4340ef1f-61ea-4077-830d-514d4d2e4a25\" (UID: \"4340ef1f-61ea-4077-830d-514d4d2e4a25\") " Oct 31 13:19:56.103487 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:56.103388 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4340ef1f-61ea-4077-830d-514d4d2e4a25-kube-api-access-t9v95\" (UniqueName: \"kubernetes.io/projected/4340ef1f-61ea-4077-830d-514d4d2e4a25-kube-api-access-t9v95\") pod \"4340ef1f-61ea-4077-830d-514d4d2e4a25\" (UID: \"4340ef1f-61ea-4077-830d-514d4d2e4a25\") " Oct 31 13:19:56.103487 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:56.103416 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4340ef1f-61ea-4077-830d-514d4d2e4a25-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4340ef1f-61ea-4077-830d-514d4d2e4a25-plugins-config-volume\") pod \"4340ef1f-61ea-4077-830d-514d4d2e4a25\" (UID: \"4340ef1f-61ea-4077-830d-514d4d2e4a25\") " Oct 31 13:19:56.103712 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:56.103686 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4340ef1f-61ea-4077-830d-514d4d2e4a25-routing-overlay-volume" pod "4340ef1f-61ea-4077-830d-514d4d2e4a25" (UID: "4340ef1f-61ea-4077-830d-514d4d2e4a25"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:19:56.103810 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:56.103761 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4340ef1f-61ea-4077-830d-514d4d2e4a25-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4340ef1f-61ea-4077-830d-514d4d2e4a25-provider-credentials\") pod \"4340ef1f-61ea-4077-830d-514d4d2e4a25\" (UID: \"4340ef1f-61ea-4077-830d-514d4d2e4a25\") " Oct 31 13:19:56.103977 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:56.103949 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4340ef1f-61ea-4077-830d-514d4d2e4a25-plugins-config-volume" pod "4340ef1f-61ea-4077-830d-514d4d2e4a25" (UID: "4340ef1f-61ea-4077-830d-514d4d2e4a25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:19:56.104567 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:56.104500 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4340ef1f-61ea-4077-830d-514d4d2e4a25-routing-overlay-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:56.104567 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:56.104522 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4340ef1f-61ea-4077-830d-514d4d2e4a25-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:56.105854 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:56.105831 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4340ef1f-61ea-4077-830d-514d4d2e4a25-kube-api-access-t9v95" pod "4340ef1f-61ea-4077-830d-514d4d2e4a25" (UID: "4340ef1f-61ea-4077-830d-514d4d2e4a25"). InnerVolumeSpecName "kube-api-access-t9v95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:19:56.106174 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:56.106149 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4340ef1f-61ea-4077-830d-514d4d2e4a25-provider-credentials" pod "4340ef1f-61ea-4077-830d-514d4d2e4a25" (UID: "4340ef1f-61ea-4077-830d-514d4d2e4a25"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:19:56.210303 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:56.210263 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4340ef1f-61ea-4077-830d-514d4d2e4a25-provider-credentials\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:56.210303 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:56.210304 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9v95\" (UniqueName: \"kubernetes.io/projected/4340ef1f-61ea-4077-830d-514d4d2e4a25-kube-api-access-t9v95\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:56.796858 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:56.796821 5693 generic.go:356] "Generic (PLEG): container finished" podID="4340ef1f-61ea-4077-830d-514d4d2e4a25" containerID="598cc398864270cac424087f8e56e18f1d2483cb8d718d3f52a06722880ebe65" exitCode=0 Oct 31 13:19:56.797394 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:56.796862 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg" Oct 31 13:19:56.797394 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:56.796896 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg" event={"ID":"4340ef1f-61ea-4077-830d-514d4d2e4a25","Type":"ContainerDied","Data":"598cc398864270cac424087f8e56e18f1d2483cb8d718d3f52a06722880ebe65"} Oct 31 13:19:56.797394 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:56.796953 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg" event={"ID":"4340ef1f-61ea-4077-830d-514d4d2e4a25","Type":"ContainerDied","Data":"701d902523b266a7dfbf19090b243c2ad37b51499bf40f5f4a3c33cac974100f"} Oct 31 13:19:56.797394 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:56.796984 5693 scope.go:122] "RemoveContainer" containerID="598cc398864270cac424087f8e56e18f1d2483cb8d718d3f52a06722880ebe65" Oct 31 13:19:56.816666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:56.810963 5693 scope.go:122] "RemoveContainer" containerID="598cc398864270cac424087f8e56e18f1d2483cb8d718d3f52a06722880ebe65" Oct 31 13:19:56.816666 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:19:56.811884 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"598cc398864270cac424087f8e56e18f1d2483cb8d718d3f52a06722880ebe65\": container with ID starting with 598cc398864270cac424087f8e56e18f1d2483cb8d718d3f52a06722880ebe65 not found: ID does not exist" containerID="598cc398864270cac424087f8e56e18f1d2483cb8d718d3f52a06722880ebe65" Oct 31 13:19:56.816666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:56.812022 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"598cc398864270cac424087f8e56e18f1d2483cb8d718d3f52a06722880ebe65"} err="failed to get container status \"598cc398864270cac424087f8e56e18f1d2483cb8d718d3f52a06722880ebe65\": rpc error: code = NotFound desc = could not find container \"598cc398864270cac424087f8e56e18f1d2483cb8d718d3f52a06722880ebe65\": container with ID starting with 598cc398864270cac424087f8e56e18f1d2483cb8d718d3f52a06722880ebe65 not found: ID does not exist" Oct 31 13:19:56.833274 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:56.833243 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg"] Oct 31 13:19:56.838146 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:56.838116 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6958dmprg"] Oct 31 13:19:57.423188 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:19:57.423159 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4340ef1f-61ea-4077-830d-514d4d2e4a25" path="/var/lib/kubelet/pods/4340ef1f-61ea-4077-830d-514d4d2e4a25/volumes" Oct 31 13:20:03.788944 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:03.788907 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5"] Oct 31 13:20:08.789300 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:08.789229 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5" podUID="17b2a30d-fd18-4144-9212-7ef5586a629e" containerName="payload-processing" containerID="cri-o://f7e31a2b4ca898fd3c4d1a7296df9111b9a4d77ee305826307febbfd8dee39bb" gracePeriod=25 Oct 31 13:20:09.007986 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:09.007953 5693 generic.go:356] "Generic (PLEG): container finished" podID="17b2a30d-fd18-4144-9212-7ef5586a629e" containerID="f7e31a2b4ca898fd3c4d1a7296df9111b9a4d77ee305826307febbfd8dee39bb" exitCode=0 Oct 31 13:20:09.008179 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:09.008135 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5" event={"ID":"17b2a30d-fd18-4144-9212-7ef5586a629e","Type":"ContainerDied","Data":"f7e31a2b4ca898fd3c4d1a7296df9111b9a4d77ee305826307febbfd8dee39bb"} Oct 31 13:20:09.111175 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:09.111148 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5" Oct 31 13:20:09.207843 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:09.207813 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17b2a30d-fd18-4144-9212-7ef5586a629e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/17b2a30d-fd18-4144-9212-7ef5586a629e-routing-overlay-volume\") pod \"17b2a30d-fd18-4144-9212-7ef5586a629e\" (UID: \"17b2a30d-fd18-4144-9212-7ef5586a629e\") " Oct 31 13:20:09.208019 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:09.207983 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17b2a30d-fd18-4144-9212-7ef5586a629e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17b2a30d-fd18-4144-9212-7ef5586a629e-plugins-config-volume\") pod \"17b2a30d-fd18-4144-9212-7ef5586a629e\" (UID: \"17b2a30d-fd18-4144-9212-7ef5586a629e\") " Oct 31 13:20:09.208019 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:09.208017 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17b2a30d-fd18-4144-9212-7ef5586a629e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/17b2a30d-fd18-4144-9212-7ef5586a629e-provider-credentials\") pod \"17b2a30d-fd18-4144-9212-7ef5586a629e\" (UID: \"17b2a30d-fd18-4144-9212-7ef5586a629e\") " Oct 31 13:20:09.208181 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:09.208158 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17b2a30d-fd18-4144-9212-7ef5586a629e-routing-overlay-volume" pod "17b2a30d-fd18-4144-9212-7ef5586a629e" (UID: "17b2a30d-fd18-4144-9212-7ef5586a629e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:20:09.208256 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:09.208192 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17b2a30d-fd18-4144-9212-7ef5586a629e-kube-api-access-4jqzr\" (UniqueName: \"kubernetes.io/projected/17b2a30d-fd18-4144-9212-7ef5586a629e-kube-api-access-4jqzr\") pod \"17b2a30d-fd18-4144-9212-7ef5586a629e\" (UID: \"17b2a30d-fd18-4144-9212-7ef5586a629e\") " Oct 31 13:20:09.208428 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:09.208408 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17b2a30d-fd18-4144-9212-7ef5586a629e-plugins-config-volume" pod "17b2a30d-fd18-4144-9212-7ef5586a629e" (UID: "17b2a30d-fd18-4144-9212-7ef5586a629e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:20:09.209187 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:09.209161 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/17b2a30d-fd18-4144-9212-7ef5586a629e-routing-overlay-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:20:09.209315 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:09.209192 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17b2a30d-fd18-4144-9212-7ef5586a629e-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:20:09.210754 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:09.210725 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17b2a30d-fd18-4144-9212-7ef5586a629e-provider-credentials" pod "17b2a30d-fd18-4144-9212-7ef5586a629e" (UID: "17b2a30d-fd18-4144-9212-7ef5586a629e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:20:09.210879 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:09.210784 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17b2a30d-fd18-4144-9212-7ef5586a629e-kube-api-access-4jqzr" pod "17b2a30d-fd18-4144-9212-7ef5586a629e" (UID: "17b2a30d-fd18-4144-9212-7ef5586a629e"). InnerVolumeSpecName "kube-api-access-4jqzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:20:09.310999 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:09.310963 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/17b2a30d-fd18-4144-9212-7ef5586a629e-provider-credentials\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:20:09.310999 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:09.310992 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jqzr\" (UniqueName: \"kubernetes.io/projected/17b2a30d-fd18-4144-9212-7ef5586a629e-kube-api-access-4jqzr\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:20:10.018802 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:10.018767 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5" Oct 31 13:20:10.018802 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:10.018799 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5" event={"ID":"17b2a30d-fd18-4144-9212-7ef5586a629e","Type":"ContainerDied","Data":"a3d9859fe83d846e162ec4e35e4d2bdc065b9eddd44237ebaeba0a2949f00feb"} Oct 31 13:20:10.019392 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:10.018867 5693 scope.go:122] "RemoveContainer" containerID="f7e31a2b4ca898fd3c4d1a7296df9111b9a4d77ee305826307febbfd8dee39bb" Oct 31 13:20:10.064001 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:10.063963 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5"] Oct 31 13:20:10.071713 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:10.071687 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7d8bm2rg5"] Oct 31 13:20:11.411175 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:11.411146 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17b2a30d-fd18-4144-9212-7ef5586a629e" path="/var/lib/kubelet/pods/17b2a30d-fd18-4144-9212-7ef5586a629e/volumes" Oct 31 13:20:12.019943 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:12.019911 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr"] Oct 31 13:20:17.020610 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:17.020506 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr" podUID="558af0a5-143e-4842-a180-22280235ab6d" containerName="payload-processing" containerID="cri-o://28c9cbe7e71617ddcda7162babbe2585e98ef2821aefe5f6bf304894c769a222" gracePeriod=25 Oct 31 13:20:17.318103 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:17.318068 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr" Oct 31 13:20:17.426931 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:17.426899 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/558af0a5-143e-4842-a180-22280235ab6d-kube-api-access-pmphg\" (UniqueName: \"kubernetes.io/projected/558af0a5-143e-4842-a180-22280235ab6d-kube-api-access-pmphg\") pod \"558af0a5-143e-4842-a180-22280235ab6d\" (UID: \"558af0a5-143e-4842-a180-22280235ab6d\") " Oct 31 13:20:17.427091 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:17.427002 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/558af0a5-143e-4842-a180-22280235ab6d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/558af0a5-143e-4842-a180-22280235ab6d-provider-credentials\") pod \"558af0a5-143e-4842-a180-22280235ab6d\" (UID: \"558af0a5-143e-4842-a180-22280235ab6d\") " Oct 31 13:20:17.427231 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:17.427215 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/558af0a5-143e-4842-a180-22280235ab6d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/558af0a5-143e-4842-a180-22280235ab6d-plugins-config-volume\") pod \"558af0a5-143e-4842-a180-22280235ab6d\" (UID: \"558af0a5-143e-4842-a180-22280235ab6d\") " Oct 31 13:20:17.427373 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:17.427356 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/558af0a5-143e-4842-a180-22280235ab6d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/558af0a5-143e-4842-a180-22280235ab6d-routing-overlay-volume\") pod \"558af0a5-143e-4842-a180-22280235ab6d\" (UID: \"558af0a5-143e-4842-a180-22280235ab6d\") " Oct 31 13:20:17.427695 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:17.427669 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/558af0a5-143e-4842-a180-22280235ab6d-plugins-config-volume" pod "558af0a5-143e-4842-a180-22280235ab6d" (UID: "558af0a5-143e-4842-a180-22280235ab6d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:20:17.427944 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:17.427919 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/558af0a5-143e-4842-a180-22280235ab6d-routing-overlay-volume" pod "558af0a5-143e-4842-a180-22280235ab6d" (UID: "558af0a5-143e-4842-a180-22280235ab6d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:20:17.428139 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:17.428120 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/558af0a5-143e-4842-a180-22280235ab6d-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:20:17.428180 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:17.428146 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/558af0a5-143e-4842-a180-22280235ab6d-routing-overlay-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:20:17.429539 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:17.429512 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/558af0a5-143e-4842-a180-22280235ab6d-provider-credentials" pod "558af0a5-143e-4842-a180-22280235ab6d" (UID: "558af0a5-143e-4842-a180-22280235ab6d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:20:17.429809 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:17.429783 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/558af0a5-143e-4842-a180-22280235ab6d-kube-api-access-pmphg" pod "558af0a5-143e-4842-a180-22280235ab6d" (UID: "558af0a5-143e-4842-a180-22280235ab6d"). InnerVolumeSpecName "kube-api-access-pmphg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:20:17.529553 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:17.529448 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/558af0a5-143e-4842-a180-22280235ab6d-provider-credentials\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:20:17.529553 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:17.529483 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmphg\" (UniqueName: \"kubernetes.io/projected/558af0a5-143e-4842-a180-22280235ab6d-kube-api-access-pmphg\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:20:18.132716 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:18.132682 5693 generic.go:356] "Generic (PLEG): container finished" podID="558af0a5-143e-4842-a180-22280235ab6d" containerID="28c9cbe7e71617ddcda7162babbe2585e98ef2821aefe5f6bf304894c769a222" exitCode=0 Oct 31 13:20:18.133138 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:18.132731 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr" Oct 31 13:20:18.133138 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:18.132763 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr" event={"ID":"558af0a5-143e-4842-a180-22280235ab6d","Type":"ContainerDied","Data":"28c9cbe7e71617ddcda7162babbe2585e98ef2821aefe5f6bf304894c769a222"} Oct 31 13:20:18.133138 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:18.132817 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr" event={"ID":"558af0a5-143e-4842-a180-22280235ab6d","Type":"ContainerDied","Data":"a2522fb4301da2c9cafdfe9cbffcb9f1defd1324a6fb9647254bfd8deaf3b07e"} Oct 31 13:20:18.133138 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:18.132839 5693 scope.go:122] "RemoveContainer" containerID="28c9cbe7e71617ddcda7162babbe2585e98ef2821aefe5f6bf304894c769a222" Oct 31 13:20:18.144882 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:18.144859 5693 scope.go:122] "RemoveContainer" containerID="28c9cbe7e71617ddcda7162babbe2585e98ef2821aefe5f6bf304894c769a222" Oct 31 13:20:18.145167 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:20:18.145150 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28c9cbe7e71617ddcda7162babbe2585e98ef2821aefe5f6bf304894c769a222\": container with ID starting with 28c9cbe7e71617ddcda7162babbe2585e98ef2821aefe5f6bf304894c769a222 not found: ID does not exist" containerID="28c9cbe7e71617ddcda7162babbe2585e98ef2821aefe5f6bf304894c769a222" Oct 31 13:20:18.145221 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:18.145175 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28c9cbe7e71617ddcda7162babbe2585e98ef2821aefe5f6bf304894c769a222"} err="failed to get container status \"28c9cbe7e71617ddcda7162babbe2585e98ef2821aefe5f6bf304894c769a222\": rpc error: code = NotFound desc = could not find container \"28c9cbe7e71617ddcda7162babbe2585e98ef2821aefe5f6bf304894c769a222\": container with ID starting with 28c9cbe7e71617ddcda7162babbe2585e98ef2821aefe5f6bf304894c769a222 not found: ID does not exist" Oct 31 13:20:18.160556 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:18.160530 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr"] Oct 31 13:20:18.164550 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:18.164526 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-74dd224xr"] Oct 31 13:20:18.694995 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:20:18.694963 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:20:19.410740 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:19.410708 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="558af0a5-143e-4842-a180-22280235ab6d" path="/var/lib/kubelet/pods/558af0a5-143e-4842-a180-22280235ab6d/volumes" Oct 31 13:20:34.420962 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:34.420933 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g"] Oct 31 13:20:34.421808 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:34.421792 5693 cpu_manager.go:414] "containerMap: removing container" podUID="558af0a5-143e-4842-a180-22280235ab6d" containerName="payload-processing" Oct 31 13:20:34.421871 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:34.421810 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="558af0a5-143e-4842-a180-22280235ab6d" containerName="payload-processing" Oct 31 13:20:34.421871 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:34.421829 5693 cpu_manager.go:414] "containerMap: removing container" podUID="17b2a30d-fd18-4144-9212-7ef5586a629e" containerName="payload-processing" Oct 31 13:20:34.421871 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:34.421834 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17b2a30d-fd18-4144-9212-7ef5586a629e" containerName="payload-processing" Oct 31 13:20:34.421871 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:34.421869 5693 cpu_manager.go:414] "containerMap: removing container" podUID="4340ef1f-61ea-4077-830d-514d4d2e4a25" containerName="payload-processing" Oct 31 13:20:34.422010 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:34.421875 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4340ef1f-61ea-4077-830d-514d4d2e4a25" containerName="payload-processing" Oct 31 13:20:34.422010 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:34.421886 5693 cpu_manager.go:414] "containerMap: removing container" podUID="8afd9c8e-49db-4c62-8af3-4fbed8d2d954" containerName="istio-proxy" Oct 31 13:20:34.422010 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:34.421892 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8afd9c8e-49db-4c62-8af3-4fbed8d2d954" containerName="istio-proxy" Oct 31 13:20:34.422161 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:34.422023 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="4340ef1f-61ea-4077-830d-514d4d2e4a25" containerName="payload-processing" Oct 31 13:20:34.422161 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:34.422038 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="8afd9c8e-49db-4c62-8af3-4fbed8d2d954" containerName="istio-proxy" Oct 31 13:20:34.422161 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:34.422055 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="558af0a5-143e-4842-a180-22280235ab6d" containerName="payload-processing" Oct 31 13:20:34.422161 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:34.422066 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="17b2a30d-fd18-4144-9212-7ef5586a629e" containerName="payload-processing" Oct 31 13:20:34.427596 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:34.427577 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g" Oct 31 13:20:34.439903 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:34.439876 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-54a16b\"/\"agc-cred-c3f392fb9a7d\"" Oct 31 13:20:34.442162 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:34.442132 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g"] Oct 31 13:20:34.510125 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:34.510089 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a250e7c9-851e-4caf-8d6f-de4a4014a301-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g\" (UID: \"a250e7c9-851e-4caf-8d6f-de4a4014a301\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g" Oct 31 13:20:34.510337 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:34.510228 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a250e7c9-851e-4caf-8d6f-de4a4014a301-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g\" (UID: \"a250e7c9-851e-4caf-8d6f-de4a4014a301\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g" Oct 31 13:20:34.510406 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:34.510350 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a250e7c9-851e-4caf-8d6f-de4a4014a301-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g\" (UID: \"a250e7c9-851e-4caf-8d6f-de4a4014a301\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g" Oct 31 13:20:34.510463 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:34.510402 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5956f\" (UniqueName: \"kubernetes.io/projected/a250e7c9-851e-4caf-8d6f-de4a4014a301-kube-api-access-5956f\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g\" (UID: \"a250e7c9-851e-4caf-8d6f-de4a4014a301\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g" Oct 31 13:20:34.612253 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:34.612220 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a250e7c9-851e-4caf-8d6f-de4a4014a301-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g\" (UID: \"a250e7c9-851e-4caf-8d6f-de4a4014a301\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g" Oct 31 13:20:34.612465 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:34.612341 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a250e7c9-851e-4caf-8d6f-de4a4014a301-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g\" (UID: \"a250e7c9-851e-4caf-8d6f-de4a4014a301\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g" Oct 31 13:20:34.612465 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:34.612456 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a250e7c9-851e-4caf-8d6f-de4a4014a301-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g\" (UID: \"a250e7c9-851e-4caf-8d6f-de4a4014a301\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g" Oct 31 13:20:34.612587 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:34.612486 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5956f\" (UniqueName: \"kubernetes.io/projected/a250e7c9-851e-4caf-8d6f-de4a4014a301-kube-api-access-5956f\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g\" (UID: \"a250e7c9-851e-4caf-8d6f-de4a4014a301\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g" Oct 31 13:20:34.613238 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:34.613175 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a250e7c9-851e-4caf-8d6f-de4a4014a301-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g\" (UID: \"a250e7c9-851e-4caf-8d6f-de4a4014a301\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g" Oct 31 13:20:34.613367 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:34.613236 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a250e7c9-851e-4caf-8d6f-de4a4014a301-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g\" (UID: \"a250e7c9-851e-4caf-8d6f-de4a4014a301\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g" Oct 31 13:20:34.615225 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:34.615191 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a250e7c9-851e-4caf-8d6f-de4a4014a301-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g\" (UID: \"a250e7c9-851e-4caf-8d6f-de4a4014a301\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g" Oct 31 13:20:34.637818 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:34.637794 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5956f\" (UniqueName: \"kubernetes.io/projected/a250e7c9-851e-4caf-8d6f-de4a4014a301-kube-api-access-5956f\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g\" (UID: \"a250e7c9-851e-4caf-8d6f-de4a4014a301\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g" Oct 31 13:20:34.745198 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:34.745094 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g" Oct 31 13:20:35.044280 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:35.044250 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g"] Oct 31 13:20:35.045194 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:20:35.045169 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda250e7c9_851e_4caf_8d6f_de4a4014a301.slice/crio-62cedcc436a2f9bf905f01cb199584e344dec15160f812a00417215855f44196 WatchSource:0}: Error finding container 62cedcc436a2f9bf905f01cb199584e344dec15160f812a00417215855f44196: Status 404 returned error can't find the container with id 62cedcc436a2f9bf905f01cb199584e344dec15160f812a00417215855f44196 Oct 31 13:20:35.305951 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:35.305859 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g" event={"ID":"a250e7c9-851e-4caf-8d6f-de4a4014a301","Type":"ContainerStarted","Data":"e1d9b071db408dbccbb0afa601f352fe9b5ba231f28f0b42d1fec3796ddd354b"} Oct 31 13:20:35.305951 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:35.305907 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g" event={"ID":"a250e7c9-851e-4caf-8d6f-de4a4014a301","Type":"ContainerStarted","Data":"62cedcc436a2f9bf905f01cb199584e344dec15160f812a00417215855f44196"} Oct 31 13:20:35.306150 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:35.306131 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g" Oct 31 13:20:35.328175 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:35.328124 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g" podStartSLOduration=1.32811006 podStartE2EDuration="1.32811006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:20:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:20:35.325790945 +0000 UTC m=+2886.033377241" watchObservedRunningTime="2026-10-31 13:20:35.32811006 +0000 UTC m=+2886.035696354" Oct 31 13:20:46.318517 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:46.318441 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g" Oct 31 13:20:46.383436 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:46.383400 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj"] Oct 31 13:20:48.689275 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:20:48.689232 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:20:51.388673 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:51.388608 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj" podUID="ec1e3d94-c558-4967-95c5-88b63444e8e6" containerName="payload-processing" containerID="cri-o://925c97f128e04b4190aeda5e068e3efa927437d2f4a14b3c19e59a1abb67fe85" gracePeriod=25 Oct 31 13:20:51.679405 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:51.679377 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj" Oct 31 13:20:51.744245 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:51.744206 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec1e3d94-c558-4967-95c5-88b63444e8e6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ec1e3d94-c558-4967-95c5-88b63444e8e6-provider-credentials\") pod \"ec1e3d94-c558-4967-95c5-88b63444e8e6\" (UID: \"ec1e3d94-c558-4967-95c5-88b63444e8e6\") " Oct 31 13:20:51.744445 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:51.744391 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec1e3d94-c558-4967-95c5-88b63444e8e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec1e3d94-c558-4967-95c5-88b63444e8e6-plugins-config-volume\") pod \"ec1e3d94-c558-4967-95c5-88b63444e8e6\" (UID: \"ec1e3d94-c558-4967-95c5-88b63444e8e6\") " Oct 31 13:20:51.744508 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:51.744467 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec1e3d94-c558-4967-95c5-88b63444e8e6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ec1e3d94-c558-4967-95c5-88b63444e8e6-routing-overlay-volume\") pod \"ec1e3d94-c558-4967-95c5-88b63444e8e6\" (UID: \"ec1e3d94-c558-4967-95c5-88b63444e8e6\") " Oct 31 13:20:51.744695 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:51.744673 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec1e3d94-c558-4967-95c5-88b63444e8e6-kube-api-access-lz9sz\" (UniqueName: \"kubernetes.io/projected/ec1e3d94-c558-4967-95c5-88b63444e8e6-kube-api-access-lz9sz\") pod \"ec1e3d94-c558-4967-95c5-88b63444e8e6\" (UID: \"ec1e3d94-c558-4967-95c5-88b63444e8e6\") " Oct 31 13:20:51.746605 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:51.746578 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec1e3d94-c558-4967-95c5-88b63444e8e6-plugins-config-volume" pod "ec1e3d94-c558-4967-95c5-88b63444e8e6" (UID: "ec1e3d94-c558-4967-95c5-88b63444e8e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:20:51.746763 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:51.746610 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec1e3d94-c558-4967-95c5-88b63444e8e6-routing-overlay-volume" pod "ec1e3d94-c558-4967-95c5-88b63444e8e6" (UID: "ec1e3d94-c558-4967-95c5-88b63444e8e6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:20:51.792770 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:51.792705 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec1e3d94-c558-4967-95c5-88b63444e8e6-kube-api-access-lz9sz" pod "ec1e3d94-c558-4967-95c5-88b63444e8e6" (UID: "ec1e3d94-c558-4967-95c5-88b63444e8e6"). InnerVolumeSpecName "kube-api-access-lz9sz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:20:51.799102 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:51.799050 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec1e3d94-c558-4967-95c5-88b63444e8e6-provider-credentials" pod "ec1e3d94-c558-4967-95c5-88b63444e8e6" (UID: "ec1e3d94-c558-4967-95c5-88b63444e8e6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:20:51.849319 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:51.849276 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec1e3d94-c558-4967-95c5-88b63444e8e6-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:20:51.849728 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:51.849628 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ec1e3d94-c558-4967-95c5-88b63444e8e6-routing-overlay-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:20:51.849728 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:51.849705 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz9sz\" (UniqueName: \"kubernetes.io/projected/ec1e3d94-c558-4967-95c5-88b63444e8e6-kube-api-access-lz9sz\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:20:51.849728 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:51.849716 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ec1e3d94-c558-4967-95c5-88b63444e8e6-provider-credentials\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:20:52.447458 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:52.447424 5693 generic.go:356] "Generic (PLEG): container finished" podID="ec1e3d94-c558-4967-95c5-88b63444e8e6" containerID="925c97f128e04b4190aeda5e068e3efa927437d2f4a14b3c19e59a1abb67fe85" exitCode=0 Oct 31 13:20:52.447991 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:52.447482 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj" Oct 31 13:20:52.447991 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:52.447506 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj" event={"ID":"ec1e3d94-c558-4967-95c5-88b63444e8e6","Type":"ContainerDied","Data":"925c97f128e04b4190aeda5e068e3efa927437d2f4a14b3c19e59a1abb67fe85"} Oct 31 13:20:52.447991 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:52.447574 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj" event={"ID":"ec1e3d94-c558-4967-95c5-88b63444e8e6","Type":"ContainerDied","Data":"75f9118a94251997083b1f410b89f990ee90642e01182f1d88ddb66f2c9a93f2"} Oct 31 13:20:52.447991 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:52.447607 5693 scope.go:122] "RemoveContainer" containerID="925c97f128e04b4190aeda5e068e3efa927437d2f4a14b3c19e59a1abb67fe85" Oct 31 13:20:52.458915 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:52.458897 5693 scope.go:122] "RemoveContainer" containerID="925c97f128e04b4190aeda5e068e3efa927437d2f4a14b3c19e59a1abb67fe85" Oct 31 13:20:52.459215 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:20:52.459196 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"925c97f128e04b4190aeda5e068e3efa927437d2f4a14b3c19e59a1abb67fe85\": container with ID starting with 925c97f128e04b4190aeda5e068e3efa927437d2f4a14b3c19e59a1abb67fe85 not found: ID does not exist" containerID="925c97f128e04b4190aeda5e068e3efa927437d2f4a14b3c19e59a1abb67fe85" Oct 31 13:20:52.459282 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:52.459225 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"925c97f128e04b4190aeda5e068e3efa927437d2f4a14b3c19e59a1abb67fe85"} err="failed to get container status \"925c97f128e04b4190aeda5e068e3efa927437d2f4a14b3c19e59a1abb67fe85\": rpc error: code = NotFound desc = could not find container \"925c97f128e04b4190aeda5e068e3efa927437d2f4a14b3c19e59a1abb67fe85\": container with ID starting with 925c97f128e04b4190aeda5e068e3efa927437d2f4a14b3c19e59a1abb67fe85 not found: ID does not exist" Oct 31 13:20:52.477998 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:52.477973 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj"] Oct 31 13:20:52.481977 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:52.481956 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6khdtj"] Oct 31 13:20:53.417196 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:53.417165 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec1e3d94-c558-4967-95c5-88b63444e8e6" path="/var/lib/kubelet/pods/ec1e3d94-c558-4967-95c5-88b63444e8e6/volumes" Oct 31 13:20:56.376041 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:20:56.376008 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj"] Oct 31 13:21:00.686267 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:00.686232 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-655bb989fc-prw8w"] Oct 31 13:21:00.687057 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:00.687042 5693 cpu_manager.go:414] "containerMap: removing container" podUID="ec1e3d94-c558-4967-95c5-88b63444e8e6" containerName="payload-processing" Oct 31 13:21:00.687122 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:00.687057 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec1e3d94-c558-4967-95c5-88b63444e8e6" containerName="payload-processing" Oct 31 13:21:00.687204 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:00.687195 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec1e3d94-c558-4967-95c5-88b63444e8e6" containerName="payload-processing" Oct 31 13:21:00.690975 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:00.690951 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-655bb989fc-prw8w" Oct 31 13:21:00.700126 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:00.700101 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-655bb989fc-prw8w"] Oct 31 13:21:00.703539 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:00.703517 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 31 13:21:00.706574 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:00.706548 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 31 13:21:00.706755 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:00.706561 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 31 13:21:00.712696 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:00.708539 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-2ccf99ff0d3b\"" Oct 31 13:21:00.712696 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:00.708556 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-3ee368d0531c\"" Oct 31 13:21:00.712696 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:00.708590 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-f42326e45f6c\"" Oct 31 13:21:00.712696 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:00.708591 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-99ph4\"" Oct 31 13:21:00.804774 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:00.804739 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 31 13:21:00.839296 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:00.839256 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/74350b94-36ee-43f4-8019-65b2bd7380ae-provider-credentials\") pod \"payload-processing-external-model-655bb989fc-prw8w\" (UID: \"74350b94-36ee-43f4-8019-65b2bd7380ae\") " pod="models-as-a-service/payload-processing-external-model-655bb989fc-prw8w" Oct 31 13:21:00.839496 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:00.839330 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74350b94-36ee-43f4-8019-65b2bd7380ae-routing-overlay-volume\") pod \"payload-processing-external-model-655bb989fc-prw8w\" (UID: \"74350b94-36ee-43f4-8019-65b2bd7380ae\") " pod="models-as-a-service/payload-processing-external-model-655bb989fc-prw8w" Oct 31 13:21:00.839496 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:00.839409 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74350b94-36ee-43f4-8019-65b2bd7380ae-plugins-config-volume\") pod \"payload-processing-external-model-655bb989fc-prw8w\" (UID: \"74350b94-36ee-43f4-8019-65b2bd7380ae\") " pod="models-as-a-service/payload-processing-external-model-655bb989fc-prw8w" Oct 31 13:21:00.839621 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:00.839582 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6gm7\" (UniqueName: \"kubernetes.io/projected/74350b94-36ee-43f4-8019-65b2bd7380ae-kube-api-access-d6gm7\") pod \"payload-processing-external-model-655bb989fc-prw8w\" (UID: \"74350b94-36ee-43f4-8019-65b2bd7380ae\") " pod="models-as-a-service/payload-processing-external-model-655bb989fc-prw8w" Oct 31 13:21:00.941284 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:00.941194 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/74350b94-36ee-43f4-8019-65b2bd7380ae-provider-credentials\") pod \"payload-processing-external-model-655bb989fc-prw8w\" (UID: \"74350b94-36ee-43f4-8019-65b2bd7380ae\") " pod="models-as-a-service/payload-processing-external-model-655bb989fc-prw8w" Oct 31 13:21:00.941284 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:00.941247 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74350b94-36ee-43f4-8019-65b2bd7380ae-routing-overlay-volume\") pod \"payload-processing-external-model-655bb989fc-prw8w\" (UID: \"74350b94-36ee-43f4-8019-65b2bd7380ae\") " pod="models-as-a-service/payload-processing-external-model-655bb989fc-prw8w" Oct 31 13:21:00.941284 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:00.941269 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74350b94-36ee-43f4-8019-65b2bd7380ae-plugins-config-volume\") pod \"payload-processing-external-model-655bb989fc-prw8w\" (UID: \"74350b94-36ee-43f4-8019-65b2bd7380ae\") " pod="models-as-a-service/payload-processing-external-model-655bb989fc-prw8w" Oct 31 13:21:00.941601 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:00.941372 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6gm7\" (UniqueName: \"kubernetes.io/projected/74350b94-36ee-43f4-8019-65b2bd7380ae-kube-api-access-d6gm7\") pod \"payload-processing-external-model-655bb989fc-prw8w\" (UID: \"74350b94-36ee-43f4-8019-65b2bd7380ae\") " pod="models-as-a-service/payload-processing-external-model-655bb989fc-prw8w" Oct 31 13:21:00.942252 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:00.942224 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74350b94-36ee-43f4-8019-65b2bd7380ae-routing-overlay-volume\") pod \"payload-processing-external-model-655bb989fc-prw8w\" (UID: \"74350b94-36ee-43f4-8019-65b2bd7380ae\") " pod="models-as-a-service/payload-processing-external-model-655bb989fc-prw8w" Oct 31 13:21:00.942428 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:00.942348 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74350b94-36ee-43f4-8019-65b2bd7380ae-plugins-config-volume\") pod \"payload-processing-external-model-655bb989fc-prw8w\" (UID: \"74350b94-36ee-43f4-8019-65b2bd7380ae\") " pod="models-as-a-service/payload-processing-external-model-655bb989fc-prw8w" Oct 31 13:21:00.944426 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:00.944403 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/74350b94-36ee-43f4-8019-65b2bd7380ae-provider-credentials\") pod \"payload-processing-external-model-655bb989fc-prw8w\" (UID: \"74350b94-36ee-43f4-8019-65b2bd7380ae\") " pod="models-as-a-service/payload-processing-external-model-655bb989fc-prw8w" Oct 31 13:21:00.964524 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:00.964497 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6gm7\" (UniqueName: \"kubernetes.io/projected/74350b94-36ee-43f4-8019-65b2bd7380ae-kube-api-access-d6gm7\") pod \"payload-processing-external-model-655bb989fc-prw8w\" (UID: \"74350b94-36ee-43f4-8019-65b2bd7380ae\") " pod="models-as-a-service/payload-processing-external-model-655bb989fc-prw8w" Oct 31 13:21:01.009878 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:01.009840 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-655bb989fc-prw8w" Oct 31 13:21:01.376877 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:01.376772 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj" podUID="77ebd766-a4ad-4701-a3d3-76f8c06b9e45" containerName="payload-processing" containerID="cri-o://1b3b176b782be38d8ff6a3d452f728fdbaaa5e00dc3cfbe67985fa77035b1a00" gracePeriod=25 Oct 31 13:21:01.394853 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:01.394819 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-655bb989fc-prw8w"] Oct 31 13:21:01.540609 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:01.540537 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-655bb989fc-prw8w" event={"ID":"74350b94-36ee-43f4-8019-65b2bd7380ae","Type":"ContainerStarted","Data":"32e75ebd0a82832fc54e814ef0af46285de254a76297d8a199a806bb54dc2b92"} Oct 31 13:21:01.542787 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:01.542766 5693 generic.go:356] "Generic (PLEG): container finished" podID="77ebd766-a4ad-4701-a3d3-76f8c06b9e45" containerID="1b3b176b782be38d8ff6a3d452f728fdbaaa5e00dc3cfbe67985fa77035b1a00" exitCode=0 Oct 31 13:21:01.542963 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:01.542941 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj" event={"ID":"77ebd766-a4ad-4701-a3d3-76f8c06b9e45","Type":"ContainerDied","Data":"1b3b176b782be38d8ff6a3d452f728fdbaaa5e00dc3cfbe67985fa77035b1a00"} Oct 31 13:21:01.656284 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:01.656258 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj" Oct 31 13:21:01.756583 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:01.756544 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77ebd766-a4ad-4701-a3d3-76f8c06b9e45-provider-credentials\" (UniqueName: \"kubernetes.io/projected/77ebd766-a4ad-4701-a3d3-76f8c06b9e45-provider-credentials\") pod \"77ebd766-a4ad-4701-a3d3-76f8c06b9e45\" (UID: \"77ebd766-a4ad-4701-a3d3-76f8c06b9e45\") " Oct 31 13:21:01.757187 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:01.756814 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77ebd766-a4ad-4701-a3d3-76f8c06b9e45-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77ebd766-a4ad-4701-a3d3-76f8c06b9e45-plugins-config-volume\") pod \"77ebd766-a4ad-4701-a3d3-76f8c06b9e45\" (UID: \"77ebd766-a4ad-4701-a3d3-76f8c06b9e45\") " Oct 31 13:21:01.757187 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:01.756858 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77ebd766-a4ad-4701-a3d3-76f8c06b9e45-kube-api-access-cxcfk\" (UniqueName: \"kubernetes.io/projected/77ebd766-a4ad-4701-a3d3-76f8c06b9e45-kube-api-access-cxcfk\") pod \"77ebd766-a4ad-4701-a3d3-76f8c06b9e45\" (UID: \"77ebd766-a4ad-4701-a3d3-76f8c06b9e45\") " Oct 31 13:21:01.757187 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:01.757120 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77ebd766-a4ad-4701-a3d3-76f8c06b9e45-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/77ebd766-a4ad-4701-a3d3-76f8c06b9e45-routing-overlay-volume\") pod \"77ebd766-a4ad-4701-a3d3-76f8c06b9e45\" (UID: \"77ebd766-a4ad-4701-a3d3-76f8c06b9e45\") " Oct 31 13:21:01.757402 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:01.757209 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77ebd766-a4ad-4701-a3d3-76f8c06b9e45-plugins-config-volume" pod "77ebd766-a4ad-4701-a3d3-76f8c06b9e45" (UID: "77ebd766-a4ad-4701-a3d3-76f8c06b9e45"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:21:01.757765 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:01.757736 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77ebd766-a4ad-4701-a3d3-76f8c06b9e45-routing-overlay-volume" pod "77ebd766-a4ad-4701-a3d3-76f8c06b9e45" (UID: "77ebd766-a4ad-4701-a3d3-76f8c06b9e45"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:21:01.758061 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:01.757827 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77ebd766-a4ad-4701-a3d3-76f8c06b9e45-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:01.758061 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:01.757848 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/77ebd766-a4ad-4701-a3d3-76f8c06b9e45-routing-overlay-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:01.761144 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:01.761123 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77ebd766-a4ad-4701-a3d3-76f8c06b9e45-provider-credentials" pod "77ebd766-a4ad-4701-a3d3-76f8c06b9e45" (UID: "77ebd766-a4ad-4701-a3d3-76f8c06b9e45"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:21:01.761387 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:01.761361 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77ebd766-a4ad-4701-a3d3-76f8c06b9e45-kube-api-access-cxcfk" pod "77ebd766-a4ad-4701-a3d3-76f8c06b9e45" (UID: "77ebd766-a4ad-4701-a3d3-76f8c06b9e45"). InnerVolumeSpecName "kube-api-access-cxcfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:21:01.860041 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:01.860008 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/77ebd766-a4ad-4701-a3d3-76f8c06b9e45-provider-credentials\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:01.860243 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:01.860039 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxcfk\" (UniqueName: \"kubernetes.io/projected/77ebd766-a4ad-4701-a3d3-76f8c06b9e45-kube-api-access-cxcfk\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:02.595501 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:02.591865 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj" event={"ID":"77ebd766-a4ad-4701-a3d3-76f8c06b9e45","Type":"ContainerDied","Data":"ce21386962fdc5688e10f3cb5144fbf50a557573d43ecf3ea4b7cb28e4044bb0"} Oct 31 13:21:02.595501 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:02.591979 5693 scope.go:122] "RemoveContainer" containerID="1b3b176b782be38d8ff6a3d452f728fdbaaa5e00dc3cfbe67985fa77035b1a00" Oct 31 13:21:02.595501 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:02.592168 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj" Oct 31 13:21:02.612499 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:02.611556 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-655bb989fc-prw8w" event={"ID":"74350b94-36ee-43f4-8019-65b2bd7380ae","Type":"ContainerStarted","Data":"a8be26739aa95cffd19b0c97b44ce3f693d063016e867fe81031b9193fd63bff"} Oct 31 13:21:02.612499 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:02.612443 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-655bb989fc-prw8w" Oct 31 13:21:02.641774 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:02.641735 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj"] Oct 31 13:21:02.643703 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:02.643673 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-578c7w4tj"] Oct 31 13:21:02.652519 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:02.652325 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-655bb989fc-prw8w" podStartSLOduration=2.652308103 podStartE2EDuration="2.652308103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:21:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:21:02.651723691 +0000 UTC m=+2913.359309992" watchObservedRunningTime="2026-10-31 13:21:02.652308103 +0000 UTC m=+2913.359894399" Oct 31 13:21:03.417241 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:03.417206 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77ebd766-a4ad-4701-a3d3-76f8c06b9e45" path="/var/lib/kubelet/pods/77ebd766-a4ad-4701-a3d3-76f8c06b9e45/volumes" Oct 31 13:21:07.663688 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:07.663655 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-655bb989fc-prw8w"] Oct 31 13:21:07.667118 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:07.667095 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-655bb989fc-prw8w" Oct 31 13:21:12.664556 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:12.664512 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-655bb989fc-prw8w" podUID="74350b94-36ee-43f4-8019-65b2bd7380ae" containerName="payload-processing" containerID="cri-o://a8be26739aa95cffd19b0c97b44ce3f693d063016e867fe81031b9193fd63bff" gracePeriod=25 Oct 31 13:21:13.010883 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:13.010853 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-655bb989fc-prw8w" Oct 31 13:21:13.054736 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:13.054704 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74350b94-36ee-43f4-8019-65b2bd7380ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74350b94-36ee-43f4-8019-65b2bd7380ae-plugins-config-volume\") pod \"74350b94-36ee-43f4-8019-65b2bd7380ae\" (UID: \"74350b94-36ee-43f4-8019-65b2bd7380ae\") " Oct 31 13:21:13.054942 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:13.054800 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74350b94-36ee-43f4-8019-65b2bd7380ae-provider-credentials\" (UniqueName: \"kubernetes.io/projected/74350b94-36ee-43f4-8019-65b2bd7380ae-provider-credentials\") pod \"74350b94-36ee-43f4-8019-65b2bd7380ae\" (UID: \"74350b94-36ee-43f4-8019-65b2bd7380ae\") " Oct 31 13:21:13.054942 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:13.054848 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74350b94-36ee-43f4-8019-65b2bd7380ae-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74350b94-36ee-43f4-8019-65b2bd7380ae-routing-overlay-volume\") pod \"74350b94-36ee-43f4-8019-65b2bd7380ae\" (UID: \"74350b94-36ee-43f4-8019-65b2bd7380ae\") " Oct 31 13:21:13.055079 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:13.055062 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74350b94-36ee-43f4-8019-65b2bd7380ae-kube-api-access-d6gm7\" (UniqueName: \"kubernetes.io/projected/74350b94-36ee-43f4-8019-65b2bd7380ae-kube-api-access-d6gm7\") pod \"74350b94-36ee-43f4-8019-65b2bd7380ae\" (UID: \"74350b94-36ee-43f4-8019-65b2bd7380ae\") " Oct 31 13:21:13.055199 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:13.055172 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74350b94-36ee-43f4-8019-65b2bd7380ae-plugins-config-volume" pod "74350b94-36ee-43f4-8019-65b2bd7380ae" (UID: "74350b94-36ee-43f4-8019-65b2bd7380ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:21:13.055281 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:13.055260 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74350b94-36ee-43f4-8019-65b2bd7380ae-routing-overlay-volume" pod "74350b94-36ee-43f4-8019-65b2bd7380ae" (UID: "74350b94-36ee-43f4-8019-65b2bd7380ae"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:21:13.055877 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:13.055850 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74350b94-36ee-43f4-8019-65b2bd7380ae-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:13.055877 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:13.055879 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74350b94-36ee-43f4-8019-65b2bd7380ae-routing-overlay-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:13.058445 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:13.058400 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74350b94-36ee-43f4-8019-65b2bd7380ae-kube-api-access-d6gm7" pod "74350b94-36ee-43f4-8019-65b2bd7380ae" (UID: "74350b94-36ee-43f4-8019-65b2bd7380ae"). InnerVolumeSpecName "kube-api-access-d6gm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:21:13.058576 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:13.058530 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74350b94-36ee-43f4-8019-65b2bd7380ae-provider-credentials" pod "74350b94-36ee-43f4-8019-65b2bd7380ae" (UID: "74350b94-36ee-43f4-8019-65b2bd7380ae"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:21:13.157771 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:13.157732 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6gm7\" (UniqueName: \"kubernetes.io/projected/74350b94-36ee-43f4-8019-65b2bd7380ae-kube-api-access-d6gm7\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:13.157771 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:13.157765 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/74350b94-36ee-43f4-8019-65b2bd7380ae-provider-credentials\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:13.733696 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:13.733593 5693 generic.go:356] "Generic (PLEG): container finished" podID="74350b94-36ee-43f4-8019-65b2bd7380ae" containerID="a8be26739aa95cffd19b0c97b44ce3f693d063016e867fe81031b9193fd63bff" exitCode=0 Oct 31 13:21:13.733696 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:13.733654 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-655bb989fc-prw8w" Oct 31 13:21:13.734164 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:13.733701 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-655bb989fc-prw8w" event={"ID":"74350b94-36ee-43f4-8019-65b2bd7380ae","Type":"ContainerDied","Data":"a8be26739aa95cffd19b0c97b44ce3f693d063016e867fe81031b9193fd63bff"} Oct 31 13:21:13.734164 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:13.733764 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-655bb989fc-prw8w" event={"ID":"74350b94-36ee-43f4-8019-65b2bd7380ae","Type":"ContainerDied","Data":"32e75ebd0a82832fc54e814ef0af46285de254a76297d8a199a806bb54dc2b92"} Oct 31 13:21:13.734164 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:13.733793 5693 scope.go:122] "RemoveContainer" containerID="a8be26739aa95cffd19b0c97b44ce3f693d063016e867fe81031b9193fd63bff" Oct 31 13:21:13.745175 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:13.745157 5693 scope.go:122] "RemoveContainer" containerID="a8be26739aa95cffd19b0c97b44ce3f693d063016e867fe81031b9193fd63bff" Oct 31 13:21:13.745552 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:21:13.745522 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8be26739aa95cffd19b0c97b44ce3f693d063016e867fe81031b9193fd63bff\": container with ID starting with a8be26739aa95cffd19b0c97b44ce3f693d063016e867fe81031b9193fd63bff not found: ID does not exist" containerID="a8be26739aa95cffd19b0c97b44ce3f693d063016e867fe81031b9193fd63bff" Oct 31 13:21:13.745679 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:13.745557 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8be26739aa95cffd19b0c97b44ce3f693d063016e867fe81031b9193fd63bff"} err="failed to get container status \"a8be26739aa95cffd19b0c97b44ce3f693d063016e867fe81031b9193fd63bff\": rpc error: code = NotFound desc = could not find container \"a8be26739aa95cffd19b0c97b44ce3f693d063016e867fe81031b9193fd63bff\": container with ID starting with a8be26739aa95cffd19b0c97b44ce3f693d063016e867fe81031b9193fd63bff not found: ID does not exist" Oct 31 13:21:13.767949 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:13.767919 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-655bb989fc-prw8w"] Oct 31 13:21:13.772408 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:13.772379 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-655bb989fc-prw8w"] Oct 31 13:21:15.413037 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:15.413000 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74350b94-36ee-43f4-8019-65b2bd7380ae" path="/var/lib/kubelet/pods/74350b94-36ee-43f4-8019-65b2bd7380ae/volumes" Oct 31 13:21:15.455218 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:15.455188 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch"] Oct 31 13:21:15.456088 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:15.456071 5693 cpu_manager.go:414] "containerMap: removing container" podUID="77ebd766-a4ad-4701-a3d3-76f8c06b9e45" containerName="payload-processing" Oct 31 13:21:15.456142 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:15.456090 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77ebd766-a4ad-4701-a3d3-76f8c06b9e45" containerName="payload-processing" Oct 31 13:21:15.456142 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:15.456102 5693 cpu_manager.go:414] "containerMap: removing container" podUID="74350b94-36ee-43f4-8019-65b2bd7380ae" containerName="payload-processing" Oct 31 13:21:15.456142 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:15.456107 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74350b94-36ee-43f4-8019-65b2bd7380ae" containerName="payload-processing" Oct 31 13:21:15.456257 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:15.456247 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="77ebd766-a4ad-4701-a3d3-76f8c06b9e45" containerName="payload-processing" Oct 31 13:21:15.456257 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:15.456256 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="74350b94-36ee-43f4-8019-65b2bd7380ae" containerName="payload-processing" Oct 31 13:21:15.461356 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:15.461334 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch" Oct 31 13:21:15.468953 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:15.468924 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-842ac9\"/\"payload-processing-external-model-plugins-e2e-extmod-a-842ac9\"" Oct 31 13:21:15.470434 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:15.470411 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-842ac9\"/\"agc-cred-7d5d1d5a1265\"" Oct 31 13:21:15.470585 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:15.470493 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-842ac9\"/\"routing-overlay\"" Oct 31 13:21:15.470585 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:15.470508 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-842ac9\"/\"kube-root-ca.crt\"" Oct 31 13:21:15.470744 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:15.470591 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-842ac9\"/\"payload-processing-external-model-e2e--64249880-dockercfg-l2zw7\"" Oct 31 13:21:15.470744 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:15.470697 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-842ac9\"/\"openshift-service-ca.crt\"" Oct 31 13:21:15.473696 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:15.473674 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch"] Oct 31 13:21:15.563387 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:15.563349 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qkjc\" (UniqueName: \"kubernetes.io/projected/9940bb55-a0b2-4d49-80ab-7b369e67394d-kube-api-access-4qkjc\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch\" (UID: \"9940bb55-a0b2-4d49-80ab-7b369e67394d\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch" Oct 31 13:21:15.563560 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:15.563425 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9940bb55-a0b2-4d49-80ab-7b369e67394d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch\" (UID: \"9940bb55-a0b2-4d49-80ab-7b369e67394d\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch" Oct 31 13:21:15.563560 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:15.563552 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9940bb55-a0b2-4d49-80ab-7b369e67394d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch\" (UID: \"9940bb55-a0b2-4d49-80ab-7b369e67394d\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch" Oct 31 13:21:15.563761 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:15.563744 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9940bb55-a0b2-4d49-80ab-7b369e67394d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch\" (UID: \"9940bb55-a0b2-4d49-80ab-7b369e67394d\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch" Oct 31 13:21:15.683319 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:15.683212 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9940bb55-a0b2-4d49-80ab-7b369e67394d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch\" (UID: \"9940bb55-a0b2-4d49-80ab-7b369e67394d\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch" Oct 31 13:21:15.691964 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:15.691929 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qkjc\" (UniqueName: \"kubernetes.io/projected/9940bb55-a0b2-4d49-80ab-7b369e67394d-kube-api-access-4qkjc\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch\" (UID: \"9940bb55-a0b2-4d49-80ab-7b369e67394d\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch" Oct 31 13:21:15.692634 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:15.692605 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9940bb55-a0b2-4d49-80ab-7b369e67394d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch\" (UID: \"9940bb55-a0b2-4d49-80ab-7b369e67394d\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch" Oct 31 13:21:15.693417 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:15.693395 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9940bb55-a0b2-4d49-80ab-7b369e67394d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch\" (UID: \"9940bb55-a0b2-4d49-80ab-7b369e67394d\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch" Oct 31 13:21:15.698677 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:15.688781 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9940bb55-a0b2-4d49-80ab-7b369e67394d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch\" (UID: \"9940bb55-a0b2-4d49-80ab-7b369e67394d\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch" Oct 31 13:21:15.698907 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:15.694191 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9940bb55-a0b2-4d49-80ab-7b369e67394d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch\" (UID: \"9940bb55-a0b2-4d49-80ab-7b369e67394d\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch" Oct 31 13:21:15.722710 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:15.722150 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9940bb55-a0b2-4d49-80ab-7b369e67394d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch\" (UID: \"9940bb55-a0b2-4d49-80ab-7b369e67394d\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch" Oct 31 13:21:15.744946 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:15.744891 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qkjc\" (UniqueName: \"kubernetes.io/projected/9940bb55-a0b2-4d49-80ab-7b369e67394d-kube-api-access-4qkjc\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch\" (UID: \"9940bb55-a0b2-4d49-80ab-7b369e67394d\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch" Oct 31 13:21:15.783665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:15.778972 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch" Oct 31 13:21:16.134276 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:16.134238 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp"] Oct 31 13:21:16.139855 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:16.139831 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp" Oct 31 13:21:16.148779 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:16.148758 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-15a6a2\"/\"kube-root-ca.crt\"" Oct 31 13:21:16.154240 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:16.154200 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-15a6a2\"/\"payload-processing-external-model-e2e--7a720dc0-dockercfg-592fk\"" Oct 31 13:21:16.154386 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:16.154271 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-15a6a2\"/\"agc-cred-e2cf7058e6c2\"" Oct 31 13:21:16.161015 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:16.160674 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp"] Oct 31 13:21:16.243523 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:16.243489 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/809353f9-0eec-4cdc-89f2-e82af19aecd7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp\" (UID: \"809353f9-0eec-4cdc-89f2-e82af19aecd7\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp" Oct 31 13:21:16.243713 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:16.243553 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/809353f9-0eec-4cdc-89f2-e82af19aecd7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp\" (UID: \"809353f9-0eec-4cdc-89f2-e82af19aecd7\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp" Oct 31 13:21:16.243767 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:16.243748 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l94sw\" (UniqueName: \"kubernetes.io/projected/809353f9-0eec-4cdc-89f2-e82af19aecd7-kube-api-access-l94sw\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp\" (UID: \"809353f9-0eec-4cdc-89f2-e82af19aecd7\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp" Oct 31 13:21:16.243863 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:16.243845 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/809353f9-0eec-4cdc-89f2-e82af19aecd7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp\" (UID: \"809353f9-0eec-4cdc-89f2-e82af19aecd7\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp" Oct 31 13:21:16.248922 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:16.248893 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-15a6a2\"/\"openshift-service-ca.crt\"" Oct 31 13:21:16.249058 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:16.248963 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-15a6a2\"/\"payload-processing-external-model-plugins-e2e-extmod-b-15a6a2\"" Oct 31 13:21:16.249058 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:16.248987 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-15a6a2\"/\"routing-overlay\"" Oct 31 13:21:16.252548 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:16.252524 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch"] Oct 31 13:21:16.252951 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:21:16.252928 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9940bb55_a0b2_4d49_80ab_7b369e67394d.slice/crio-ad42778a463a7dbc3a75f1155d987b93d3496dd4861934adef0429851c2974ab WatchSource:0}: Error finding container ad42778a463a7dbc3a75f1155d987b93d3496dd4861934adef0429851c2974ab: Status 404 returned error can't find the container with id ad42778a463a7dbc3a75f1155d987b93d3496dd4861934adef0429851c2974ab Oct 31 13:21:16.356677 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:16.348210 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/809353f9-0eec-4cdc-89f2-e82af19aecd7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp\" (UID: \"809353f9-0eec-4cdc-89f2-e82af19aecd7\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp" Oct 31 13:21:16.356677 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:16.348308 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/809353f9-0eec-4cdc-89f2-e82af19aecd7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp\" (UID: \"809353f9-0eec-4cdc-89f2-e82af19aecd7\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp" Oct 31 13:21:16.356677 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:16.348406 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l94sw\" (UniqueName: \"kubernetes.io/projected/809353f9-0eec-4cdc-89f2-e82af19aecd7-kube-api-access-l94sw\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp\" (UID: \"809353f9-0eec-4cdc-89f2-e82af19aecd7\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp" Oct 31 13:21:16.356677 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:16.348487 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/809353f9-0eec-4cdc-89f2-e82af19aecd7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp\" (UID: \"809353f9-0eec-4cdc-89f2-e82af19aecd7\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp" Oct 31 13:21:16.356677 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:16.349389 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/809353f9-0eec-4cdc-89f2-e82af19aecd7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp\" (UID: \"809353f9-0eec-4cdc-89f2-e82af19aecd7\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp" Oct 31 13:21:16.368676 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:16.358262 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/809353f9-0eec-4cdc-89f2-e82af19aecd7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp\" (UID: \"809353f9-0eec-4cdc-89f2-e82af19aecd7\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp" Oct 31 13:21:16.395763 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:16.390714 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/809353f9-0eec-4cdc-89f2-e82af19aecd7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp\" (UID: \"809353f9-0eec-4cdc-89f2-e82af19aecd7\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp" Oct 31 13:21:16.423218 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:16.423168 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l94sw\" (UniqueName: \"kubernetes.io/projected/809353f9-0eec-4cdc-89f2-e82af19aecd7-kube-api-access-l94sw\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp\" (UID: \"809353f9-0eec-4cdc-89f2-e82af19aecd7\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp" Oct 31 13:21:16.474672 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:16.469323 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp" Oct 31 13:21:16.787047 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:16.786761 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp"] Oct 31 13:21:16.792023 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:21:16.791996 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod809353f9_0eec_4cdc_89f2_e82af19aecd7.slice/crio-c3ed194bc66b2cbd9f8dd2e238e73f3ded2e48ba232ae64eb6e2c501b2003f83 WatchSource:0}: Error finding container c3ed194bc66b2cbd9f8dd2e238e73f3ded2e48ba232ae64eb6e2c501b2003f83: Status 404 returned error can't find the container with id c3ed194bc66b2cbd9f8dd2e238e73f3ded2e48ba232ae64eb6e2c501b2003f83 Oct 31 13:21:16.899303 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:16.899272 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp" event={"ID":"809353f9-0eec-4cdc-89f2-e82af19aecd7","Type":"ContainerStarted","Data":"4dd47c144a029e0a5a7300d219428e1456b55aa52b68e13b4fa7c068f039d94f"} Oct 31 13:21:16.899461 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:16.899330 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp" event={"ID":"809353f9-0eec-4cdc-89f2-e82af19aecd7","Type":"ContainerStarted","Data":"c3ed194bc66b2cbd9f8dd2e238e73f3ded2e48ba232ae64eb6e2c501b2003f83"} Oct 31 13:21:16.899461 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:16.899358 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp" Oct 31 13:21:16.901136 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:16.901110 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch" event={"ID":"9940bb55-a0b2-4d49-80ab-7b369e67394d","Type":"ContainerStarted","Data":"d24d714d00d1963229d86fac43065ae6b8b76d4f373b71db4f282cea2e3408e3"} Oct 31 13:21:16.901254 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:16.901162 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch" event={"ID":"9940bb55-a0b2-4d49-80ab-7b369e67394d","Type":"ContainerStarted","Data":"ad42778a463a7dbc3a75f1155d987b93d3496dd4861934adef0429851c2974ab"} Oct 31 13:21:16.901254 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:16.901190 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch" Oct 31 13:21:16.920832 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:16.920778 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp" podStartSLOduration=0.920760432 podStartE2EDuration="920.760432ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:21:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:21:16.919091302 +0000 UTC m=+2927.626677591" watchObservedRunningTime="2026-10-31 13:21:16.920760432 +0000 UTC m=+2927.628346728" Oct 31 13:21:16.938988 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:16.938943 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch" podStartSLOduration=1.938930281 podStartE2EDuration="1.938930281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:21:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:21:16.937521885 +0000 UTC m=+2927.645108181" watchObservedRunningTime="2026-10-31 13:21:16.938930281 +0000 UTC m=+2927.646516575" Oct 31 13:21:18.689914 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:21:18.689880 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:21:21.981846 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:21.981810 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5d47957896-7c8qs"] Oct 31 13:21:21.987211 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:21.987187 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d47957896-7c8qs" Oct 31 13:21:22.000421 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:22.000379 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 31 13:21:22.000421 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:22.000395 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 31 13:21:22.000421 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:22.000421 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 31 13:21:22.002063 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:22.002021 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 31 13:21:22.002809 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:22.002676 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-2ccf99ff0d3b\"" Oct 31 13:21:22.003343 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:22.003281 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-99ph4\"" Oct 31 13:21:22.008417 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:22.008392 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d47957896-7c8qs"] Oct 31 13:21:22.134684 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:22.134653 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9e64beca-4cd9-4507-a577-349e369ae4b8-routing-overlay-volume\") pod \"payload-processing-external-model-5d47957896-7c8qs\" (UID: \"9e64beca-4cd9-4507-a577-349e369ae4b8\") " pod="models-as-a-service/payload-processing-external-model-5d47957896-7c8qs" Oct 31 13:21:22.134890 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:22.134868 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9e64beca-4cd9-4507-a577-349e369ae4b8-provider-credentials\") pod \"payload-processing-external-model-5d47957896-7c8qs\" (UID: \"9e64beca-4cd9-4507-a577-349e369ae4b8\") " pod="models-as-a-service/payload-processing-external-model-5d47957896-7c8qs" Oct 31 13:21:22.134937 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:22.134923 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e64beca-4cd9-4507-a577-349e369ae4b8-plugins-config-volume\") pod \"payload-processing-external-model-5d47957896-7c8qs\" (UID: \"9e64beca-4cd9-4507-a577-349e369ae4b8\") " pod="models-as-a-service/payload-processing-external-model-5d47957896-7c8qs" Oct 31 13:21:22.135147 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:22.135125 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcwzf\" (UniqueName: \"kubernetes.io/projected/9e64beca-4cd9-4507-a577-349e369ae4b8-kube-api-access-kcwzf\") pod \"payload-processing-external-model-5d47957896-7c8qs\" (UID: \"9e64beca-4cd9-4507-a577-349e369ae4b8\") " pod="models-as-a-service/payload-processing-external-model-5d47957896-7c8qs" Oct 31 13:21:22.236444 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:22.236350 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcwzf\" (UniqueName: \"kubernetes.io/projected/9e64beca-4cd9-4507-a577-349e369ae4b8-kube-api-access-kcwzf\") pod \"payload-processing-external-model-5d47957896-7c8qs\" (UID: \"9e64beca-4cd9-4507-a577-349e369ae4b8\") " pod="models-as-a-service/payload-processing-external-model-5d47957896-7c8qs" Oct 31 13:21:22.236444 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:22.236407 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9e64beca-4cd9-4507-a577-349e369ae4b8-routing-overlay-volume\") pod \"payload-processing-external-model-5d47957896-7c8qs\" (UID: \"9e64beca-4cd9-4507-a577-349e369ae4b8\") " pod="models-as-a-service/payload-processing-external-model-5d47957896-7c8qs" Oct 31 13:21:22.236719 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:22.236565 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9e64beca-4cd9-4507-a577-349e369ae4b8-provider-credentials\") pod \"payload-processing-external-model-5d47957896-7c8qs\" (UID: \"9e64beca-4cd9-4507-a577-349e369ae4b8\") " pod="models-as-a-service/payload-processing-external-model-5d47957896-7c8qs" Oct 31 13:21:22.236719 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:22.236611 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e64beca-4cd9-4507-a577-349e369ae4b8-plugins-config-volume\") pod \"payload-processing-external-model-5d47957896-7c8qs\" (UID: \"9e64beca-4cd9-4507-a577-349e369ae4b8\") " pod="models-as-a-service/payload-processing-external-model-5d47957896-7c8qs" Oct 31 13:21:22.237183 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:22.237154 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9e64beca-4cd9-4507-a577-349e369ae4b8-routing-overlay-volume\") pod \"payload-processing-external-model-5d47957896-7c8qs\" (UID: \"9e64beca-4cd9-4507-a577-349e369ae4b8\") " pod="models-as-a-service/payload-processing-external-model-5d47957896-7c8qs" Oct 31 13:21:22.237303 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:22.237279 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e64beca-4cd9-4507-a577-349e369ae4b8-plugins-config-volume\") pod \"payload-processing-external-model-5d47957896-7c8qs\" (UID: \"9e64beca-4cd9-4507-a577-349e369ae4b8\") " pod="models-as-a-service/payload-processing-external-model-5d47957896-7c8qs" Oct 31 13:21:22.239189 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:22.239167 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9e64beca-4cd9-4507-a577-349e369ae4b8-provider-credentials\") pod \"payload-processing-external-model-5d47957896-7c8qs\" (UID: \"9e64beca-4cd9-4507-a577-349e369ae4b8\") " pod="models-as-a-service/payload-processing-external-model-5d47957896-7c8qs" Oct 31 13:21:22.250744 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:22.250722 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcwzf\" (UniqueName: \"kubernetes.io/projected/9e64beca-4cd9-4507-a577-349e369ae4b8-kube-api-access-kcwzf\") pod \"payload-processing-external-model-5d47957896-7c8qs\" (UID: \"9e64beca-4cd9-4507-a577-349e369ae4b8\") " pod="models-as-a-service/payload-processing-external-model-5d47957896-7c8qs" Oct 31 13:21:22.305969 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:22.305940 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d47957896-7c8qs" Oct 31 13:21:22.636813 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:22.636782 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d47957896-7c8qs"] Oct 31 13:21:22.974556 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:22.974519 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d47957896-7c8qs" event={"ID":"9e64beca-4cd9-4507-a577-349e369ae4b8","Type":"ContainerStarted","Data":"ba6a898d87137b11698cbfdeb235125a186892c773c201c9fc3ba5730ae101a7"} Oct 31 13:21:22.974734 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:22.974573 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d47957896-7c8qs" event={"ID":"9e64beca-4cd9-4507-a577-349e369ae4b8","Type":"ContainerStarted","Data":"18402c3368bec60dbf7b23ed103c8820624e9bb2524d67c9fac07c0784bf8880"} Oct 31 13:21:22.974734 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:22.974603 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5d47957896-7c8qs" Oct 31 13:21:25.217854 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:25.217801 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5d47957896-7c8qs" podStartSLOduration=4.217781087 podStartE2EDuration="4.217781087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:21:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:21:23.037202223 +0000 UTC m=+2933.744788518" watchObservedRunningTime="2026-10-31 13:21:25.217781087 +0000 UTC m=+2935.925367383" Oct 31 13:21:25.222218 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:25.222191 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7"] Oct 31 13:21:25.227222 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:25.227200 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7" Oct 31 13:21:25.235607 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:25.235037 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2ef686\"/\"kube-root-ca.crt\"" Oct 31 13:21:25.235607 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:25.235095 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2ef686\"/\"agc-cred-d3843138879a\"" Oct 31 13:21:25.235607 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:25.235174 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2ef686\"/\"openshift-service-ca.crt\"" Oct 31 13:21:25.235607 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:25.235234 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2ef686\"/\"payload-processing-external-model-plugins-e2e-extmod-a-2ef686\"" Oct 31 13:21:25.235607 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:25.235249 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2ef686\"/\"routing-overlay\"" Oct 31 13:21:25.241058 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:25.241036 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7"] Oct 31 13:21:25.279085 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:25.279042 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5bee661a-1cac-4b73-9d3b-05cf50e47813-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7\" (UID: \"5bee661a-1cac-4b73-9d3b-05cf50e47813\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7" Oct 31 13:21:25.279275 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:25.279188 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bee661a-1cac-4b73-9d3b-05cf50e47813-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7\" (UID: \"5bee661a-1cac-4b73-9d3b-05cf50e47813\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7" Oct 31 13:21:25.279430 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:25.279410 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc8xg\" (UniqueName: \"kubernetes.io/projected/5bee661a-1cac-4b73-9d3b-05cf50e47813-kube-api-access-wc8xg\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7\" (UID: \"5bee661a-1cac-4b73-9d3b-05cf50e47813\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7" Oct 31 13:21:25.279704 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:25.279678 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5bee661a-1cac-4b73-9d3b-05cf50e47813-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7\" (UID: \"5bee661a-1cac-4b73-9d3b-05cf50e47813\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7" Oct 31 13:21:25.381048 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:25.381005 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5bee661a-1cac-4b73-9d3b-05cf50e47813-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7\" (UID: \"5bee661a-1cac-4b73-9d3b-05cf50e47813\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7" Oct 31 13:21:25.381252 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:25.381137 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bee661a-1cac-4b73-9d3b-05cf50e47813-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7\" (UID: \"5bee661a-1cac-4b73-9d3b-05cf50e47813\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7" Oct 31 13:21:25.381321 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:25.381264 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc8xg\" (UniqueName: \"kubernetes.io/projected/5bee661a-1cac-4b73-9d3b-05cf50e47813-kube-api-access-wc8xg\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7\" (UID: \"5bee661a-1cac-4b73-9d3b-05cf50e47813\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7" Oct 31 13:21:25.381381 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:25.381364 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5bee661a-1cac-4b73-9d3b-05cf50e47813-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7\" (UID: \"5bee661a-1cac-4b73-9d3b-05cf50e47813\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7" Oct 31 13:21:25.382089 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:25.382062 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bee661a-1cac-4b73-9d3b-05cf50e47813-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7\" (UID: \"5bee661a-1cac-4b73-9d3b-05cf50e47813\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7" Oct 31 13:21:25.382212 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:25.382065 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5bee661a-1cac-4b73-9d3b-05cf50e47813-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7\" (UID: \"5bee661a-1cac-4b73-9d3b-05cf50e47813\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7" Oct 31 13:21:25.383859 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:25.383837 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5bee661a-1cac-4b73-9d3b-05cf50e47813-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7\" (UID: \"5bee661a-1cac-4b73-9d3b-05cf50e47813\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7" Oct 31 13:21:25.393384 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:25.393359 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc8xg\" (UniqueName: \"kubernetes.io/projected/5bee661a-1cac-4b73-9d3b-05cf50e47813-kube-api-access-wc8xg\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7\" (UID: \"5bee661a-1cac-4b73-9d3b-05cf50e47813\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7" Oct 31 13:21:25.548728 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:25.548429 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7" Oct 31 13:21:25.990409 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:25.990364 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7"] Oct 31 13:21:25.990570 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:21:25.990497 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bee661a_1cac_4b73_9d3b_05cf50e47813.slice/crio-7afb3e8b323745ad0f70ea17ac8df42aaba6988020b6644e5f99b853871fcb9b WatchSource:0}: Error finding container 7afb3e8b323745ad0f70ea17ac8df42aaba6988020b6644e5f99b853871fcb9b: Status 404 returned error can't find the container with id 7afb3e8b323745ad0f70ea17ac8df42aaba6988020b6644e5f99b853871fcb9b Oct 31 13:21:27.009399 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:27.009363 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7" event={"ID":"5bee661a-1cac-4b73-9d3b-05cf50e47813","Type":"ContainerStarted","Data":"3c110a86502eea8e0cdfef98a2b53bb1d048e3fd9d1e187e37c6372eeff472da"} Oct 31 13:21:27.009856 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:27.009418 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7" event={"ID":"5bee661a-1cac-4b73-9d3b-05cf50e47813","Type":"ContainerStarted","Data":"7afb3e8b323745ad0f70ea17ac8df42aaba6988020b6644e5f99b853871fcb9b"} Oct 31 13:21:27.010395 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:27.010377 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7" Oct 31 13:21:27.080756 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:27.076429 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7" podStartSLOduration=2.076407215 podStartE2EDuration="2.076407215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:21:27.056539505 +0000 UTC m=+2937.764125818" watchObservedRunningTime="2026-10-31 13:21:27.076407215 +0000 UTC m=+2937.783993511" Oct 31 13:21:27.912535 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:27.912507 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch" Oct 31 13:21:27.912725 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:27.912584 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp" Oct 31 13:21:34.005078 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:34.004747 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5d47957896-7c8qs" Oct 31 13:21:34.225711 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:34.225677 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g"] Oct 31 13:21:35.550941 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:35.550908 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg"] Oct 31 13:21:35.557347 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:35.557326 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" Oct 31 13:21:35.574424 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:35.574399 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg"] Oct 31 13:21:35.615353 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:35.615318 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-vtqkg\" (UID: \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" Oct 31 13:21:35.615627 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:35.615599 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-vtqkg\" (UID: \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" Oct 31 13:21:35.615627 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:35.615659 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-vtqkg\" (UID: \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" Oct 31 13:21:35.615870 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:35.615831 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-vtqkg\" (UID: \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" Oct 31 13:21:35.615963 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:35.615947 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-vtqkg\" (UID: \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" Oct 31 13:21:35.616027 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:35.615970 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-vtqkg\" (UID: \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" Oct 31 13:21:35.616027 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:35.615990 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-vtqkg\" (UID: \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" Oct 31 13:21:35.616147 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:35.616126 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-vtqkg\" (UID: \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" Oct 31 13:21:35.616591 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:35.616231 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dtb2\" (UniqueName: \"kubernetes.io/projected/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-kube-api-access-8dtb2\") pod \"maas-default-gateway-openshift-default-fc8c56547-vtqkg\" (UID: \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" Oct 31 13:21:35.723727 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:35.723690 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-vtqkg\" (UID: \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" Oct 31 13:21:35.740326 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:35.740289 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-vtqkg\" (UID: \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" Oct 31 13:21:35.740720 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:35.740338 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-vtqkg\" (UID: \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" Oct 31 13:21:35.740720 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:35.740468 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-vtqkg\" (UID: \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" Oct 31 13:21:35.740720 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:35.740571 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-vtqkg\" (UID: \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" Oct 31 13:21:35.740720 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:35.740598 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-vtqkg\" (UID: \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" Oct 31 13:21:35.740720 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:35.740632 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-vtqkg\" (UID: \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" Oct 31 13:21:35.741287 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:35.740730 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-vtqkg\" (UID: \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" Oct 31 13:21:35.741287 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:35.740864 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dtb2\" (UniqueName: \"kubernetes.io/projected/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-kube-api-access-8dtb2\") pod \"maas-default-gateway-openshift-default-fc8c56547-vtqkg\" (UID: \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" Oct 31 13:21:35.742023 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:35.741994 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-vtqkg\" (UID: \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" Oct 31 13:21:35.742478 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:35.742453 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-vtqkg\" (UID: \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" Oct 31 13:21:35.742872 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:35.742799 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-vtqkg\" (UID: \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" Oct 31 13:21:35.743933 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:35.743911 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-vtqkg\" (UID: \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" Oct 31 13:21:35.750948 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:35.750924 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-vtqkg\" (UID: \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" Oct 31 13:21:35.752286 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:35.752243 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-vtqkg\" (UID: \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" Oct 31 13:21:35.755596 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:35.755570 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-vtqkg\" (UID: \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" Oct 31 13:21:35.763203 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:35.763178 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-vtqkg\" (UID: \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" Oct 31 13:21:35.764171 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:35.764045 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dtb2\" (UniqueName: \"kubernetes.io/projected/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-kube-api-access-8dtb2\") pod \"maas-default-gateway-openshift-default-fc8c56547-vtqkg\" (UID: \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" Oct 31 13:21:35.874755 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:35.874672 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" Oct 31 13:21:36.199529 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:36.199500 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg"] Oct 31 13:21:36.200618 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:21:36.200595 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbfd993e_8ff7_4646_a482_e71ab45f2c1a.slice/crio-c7d5d023a22be6941d3e5be2d36890bf13eb109d9a717b023541f42828586a44 WatchSource:0}: Error finding container c7d5d023a22be6941d3e5be2d36890bf13eb109d9a717b023541f42828586a44: Status 404 returned error can't find the container with id c7d5d023a22be6941d3e5be2d36890bf13eb109d9a717b023541f42828586a44 Oct 31 13:21:36.204342 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:36.204298 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:21:36.204437 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:36.204386 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:21:36.204437 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:36.204417 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:21:37.118384 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:37.118348 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" event={"ID":"fbfd993e-8ff7-4646-a482-e71ab45f2c1a","Type":"ContainerStarted","Data":"086aac0f7add357ef9c84c7770f9f5b8429e00b748d272219bd3d42ec132cce2"} Oct 31 13:21:37.118867 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:37.118407 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" event={"ID":"fbfd993e-8ff7-4646-a482-e71ab45f2c1a","Type":"ContainerStarted","Data":"c7d5d023a22be6941d3e5be2d36890bf13eb109d9a717b023541f42828586a44"} Oct 31 13:21:37.164919 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:37.164869 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" podStartSLOduration=2.164854595 podStartE2EDuration="2.164854595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:21:37.162762218 +0000 UTC m=+2947.870348567" watchObservedRunningTime="2026-10-31 13:21:37.164854595 +0000 UTC m=+2947.872440891" Oct 31 13:21:37.896667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:37.879709 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" Oct 31 13:21:37.900839 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:37.900796 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-vtqkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Oct 31 13:21:37.901038 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:37.900880 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" podUID="fbfd993e-8ff7-4646-a482-e71ab45f2c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Oct 31 13:21:38.875075 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:38.875039 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-vtqkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Oct 31 13:21:38.875500 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:38.875119 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" podUID="fbfd993e-8ff7-4646-a482-e71ab45f2c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Oct 31 13:21:39.032883 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:39.032855 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7" Oct 31 13:21:39.242669 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:39.228714 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g" podUID="a250e7c9-851e-4caf-8d6f-de4a4014a301" containerName="payload-processing" containerID="cri-o://e1d9b071db408dbccbb0afa601f352fe9b5ba231f28f0b42d1fec3796ddd354b" gracePeriod=25 Oct 31 13:21:39.619723 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:39.619698 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g" Oct 31 13:21:39.708897 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:39.708859 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a250e7c9-851e-4caf-8d6f-de4a4014a301-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a250e7c9-851e-4caf-8d6f-de4a4014a301-routing-overlay-volume\") pod \"a250e7c9-851e-4caf-8d6f-de4a4014a301\" (UID: \"a250e7c9-851e-4caf-8d6f-de4a4014a301\") " Oct 31 13:21:39.709109 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:39.709081 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a250e7c9-851e-4caf-8d6f-de4a4014a301-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a250e7c9-851e-4caf-8d6f-de4a4014a301-plugins-config-volume\") pod \"a250e7c9-851e-4caf-8d6f-de4a4014a301\" (UID: \"a250e7c9-851e-4caf-8d6f-de4a4014a301\") " Oct 31 13:21:39.709234 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:39.709216 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a250e7c9-851e-4caf-8d6f-de4a4014a301-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a250e7c9-851e-4caf-8d6f-de4a4014a301-provider-credentials\") pod \"a250e7c9-851e-4caf-8d6f-de4a4014a301\" (UID: \"a250e7c9-851e-4caf-8d6f-de4a4014a301\") " Oct 31 13:21:39.709327 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:39.709310 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a250e7c9-851e-4caf-8d6f-de4a4014a301-kube-api-access-5956f\" (UniqueName: \"kubernetes.io/projected/a250e7c9-851e-4caf-8d6f-de4a4014a301-kube-api-access-5956f\") pod \"a250e7c9-851e-4caf-8d6f-de4a4014a301\" (UID: \"a250e7c9-851e-4caf-8d6f-de4a4014a301\") " Oct 31 13:21:39.709445 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:39.709309 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a250e7c9-851e-4caf-8d6f-de4a4014a301-routing-overlay-volume" pod "a250e7c9-851e-4caf-8d6f-de4a4014a301" (UID: "a250e7c9-851e-4caf-8d6f-de4a4014a301"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:21:39.709566 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:39.709543 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a250e7c9-851e-4caf-8d6f-de4a4014a301-plugins-config-volume" pod "a250e7c9-851e-4caf-8d6f-de4a4014a301" (UID: "a250e7c9-851e-4caf-8d6f-de4a4014a301"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:21:39.710499 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:39.710465 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a250e7c9-851e-4caf-8d6f-de4a4014a301-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:39.710499 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:39.710495 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a250e7c9-851e-4caf-8d6f-de4a4014a301-routing-overlay-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:39.712770 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:39.712746 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a250e7c9-851e-4caf-8d6f-de4a4014a301-kube-api-access-5956f" pod "a250e7c9-851e-4caf-8d6f-de4a4014a301" (UID: "a250e7c9-851e-4caf-8d6f-de4a4014a301"). InnerVolumeSpecName "kube-api-access-5956f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:21:39.713167 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:39.713143 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a250e7c9-851e-4caf-8d6f-de4a4014a301-provider-credentials" pod "a250e7c9-851e-4caf-8d6f-de4a4014a301" (UID: "a250e7c9-851e-4caf-8d6f-de4a4014a301"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:21:39.812726 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:39.812368 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a250e7c9-851e-4caf-8d6f-de4a4014a301-provider-credentials\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:39.812726 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:39.812396 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5956f\" (UniqueName: \"kubernetes.io/projected/a250e7c9-851e-4caf-8d6f-de4a4014a301-kube-api-access-5956f\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:39.836998 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:39.836965 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg"] Oct 31 13:21:39.837194 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:39.837173 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" podUID="69c36f54-5f25-45ea-b887-c0d2e0f53cfa" containerName="istio-proxy" containerID="cri-o://9bf4e45d90ab2d1b7557d892269a9fde0dac279fb391ae18035bc27f3743bb2c" gracePeriod=30 Oct 31 13:21:39.875474 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:39.875436 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-vtqkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Oct 31 13:21:39.876047 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:39.875510 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" podUID="fbfd993e-8ff7-4646-a482-e71ab45f2c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Oct 31 13:21:40.169759 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:40.167545 5693 generic.go:356] "Generic (PLEG): container finished" podID="a250e7c9-851e-4caf-8d6f-de4a4014a301" containerID="e1d9b071db408dbccbb0afa601f352fe9b5ba231f28f0b42d1fec3796ddd354b" exitCode=0 Oct 31 13:21:40.169759 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:40.167816 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g" event={"ID":"a250e7c9-851e-4caf-8d6f-de4a4014a301","Type":"ContainerDied","Data":"e1d9b071db408dbccbb0afa601f352fe9b5ba231f28f0b42d1fec3796ddd354b"} Oct 31 13:21:40.169759 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:40.167936 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g" event={"ID":"a250e7c9-851e-4caf-8d6f-de4a4014a301","Type":"ContainerDied","Data":"62cedcc436a2f9bf905f01cb199584e344dec15160f812a00417215855f44196"} Oct 31 13:21:40.169759 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:40.167962 5693 scope.go:122] "RemoveContainer" containerID="e1d9b071db408dbccbb0afa601f352fe9b5ba231f28f0b42d1fec3796ddd354b" Oct 31 13:21:40.169759 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:40.168111 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g" Oct 31 13:21:40.211356 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:40.211335 5693 scope.go:122] "RemoveContainer" containerID="e1d9b071db408dbccbb0afa601f352fe9b5ba231f28f0b42d1fec3796ddd354b" Oct 31 13:21:40.211768 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:21:40.211739 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1d9b071db408dbccbb0afa601f352fe9b5ba231f28f0b42d1fec3796ddd354b\": container with ID starting with e1d9b071db408dbccbb0afa601f352fe9b5ba231f28f0b42d1fec3796ddd354b not found: ID does not exist" containerID="e1d9b071db408dbccbb0afa601f352fe9b5ba231f28f0b42d1fec3796ddd354b" Oct 31 13:21:40.211848 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:40.211790 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1d9b071db408dbccbb0afa601f352fe9b5ba231f28f0b42d1fec3796ddd354b"} err="failed to get container status \"e1d9b071db408dbccbb0afa601f352fe9b5ba231f28f0b42d1fec3796ddd354b\": rpc error: code = NotFound desc = could not find container \"e1d9b071db408dbccbb0afa601f352fe9b5ba231f28f0b42d1fec3796ddd354b\": container with ID starting with e1d9b071db408dbccbb0afa601f352fe9b5ba231f28f0b42d1fec3796ddd354b not found: ID does not exist" Oct 31 13:21:40.229387 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:40.229353 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g"] Oct 31 13:21:40.234262 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:40.234238 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-8587w2w6g"] Oct 31 13:21:40.903668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:40.894100 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-vtqkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Oct 31 13:21:40.903668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:40.894156 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" podUID="fbfd993e-8ff7-4646-a482-e71ab45f2c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Oct 31 13:21:41.413498 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:41.413468 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a250e7c9-851e-4caf-8d6f-de4a4014a301" path="/var/lib/kubelet/pods/a250e7c9-851e-4caf-8d6f-de4a4014a301/volumes" Oct 31 13:21:41.875250 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:41.875163 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-vtqkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Oct 31 13:21:41.875250 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:41.875232 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" podUID="fbfd993e-8ff7-4646-a482-e71ab45f2c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Oct 31 13:21:42.875819 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:42.875787 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-vtqkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Oct 31 13:21:42.876227 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:42.875839 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" podUID="fbfd993e-8ff7-4646-a482-e71ab45f2c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Oct 31 13:21:43.669384 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:43.669348 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5"] Oct 31 13:21:43.670281 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:43.670264 5693 cpu_manager.go:414] "containerMap: removing container" podUID="a250e7c9-851e-4caf-8d6f-de4a4014a301" containerName="payload-processing" Oct 31 13:21:43.670281 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:43.670281 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a250e7c9-851e-4caf-8d6f-de4a4014a301" containerName="payload-processing" Oct 31 13:21:43.670476 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:43.670455 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="a250e7c9-851e-4caf-8d6f-de4a4014a301" containerName="payload-processing" Oct 31 13:21:43.675594 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:43.675570 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5" Oct 31 13:21:43.685150 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:43.685129 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-54a16b\"/\"agc-cred-93157c2e27cd\"" Oct 31 13:21:43.687196 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:43.687175 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-54a16b\"/\"payload-processing-external-model-e2e--427d574a-dockercfg-xc26m\"" Oct 31 13:21:43.688599 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:43.688570 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-54a16b\"/\"payload-processing-external-model-plugins-e2e-extmod-a-54a16b\"" Oct 31 13:21:43.688851 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:43.688829 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-54a16b\"/\"kube-root-ca.crt\"" Oct 31 13:21:43.688851 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:43.688845 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-54a16b\"/\"openshift-service-ca.crt\"" Oct 31 13:21:43.688985 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:43.688831 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-54a16b\"/\"routing-overlay\"" Oct 31 13:21:43.699828 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:43.699805 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5"] Oct 31 13:21:43.803994 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:43.803951 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5\" (UID: \"dbd6bd5f-f7b4-46f2-9894-89616d8448ff\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5" Oct 31 13:21:43.804232 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:43.804215 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5\" (UID: \"dbd6bd5f-f7b4-46f2-9894-89616d8448ff\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5" Oct 31 13:21:43.804284 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:43.804271 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5\" (UID: \"dbd6bd5f-f7b4-46f2-9894-89616d8448ff\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5" Oct 31 13:21:43.804362 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:43.804338 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4rcv\" (UniqueName: \"kubernetes.io/projected/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-kube-api-access-k4rcv\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5\" (UID: \"dbd6bd5f-f7b4-46f2-9894-89616d8448ff\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5" Oct 31 13:21:43.875478 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:43.875445 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-vtqkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Oct 31 13:21:43.875688 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:43.875496 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" podUID="fbfd993e-8ff7-4646-a482-e71ab45f2c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Oct 31 13:21:43.906287 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:43.906248 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5\" (UID: \"dbd6bd5f-f7b4-46f2-9894-89616d8448ff\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5" Oct 31 13:21:43.906784 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:43.906464 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5\" (UID: \"dbd6bd5f-f7b4-46f2-9894-89616d8448ff\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5" Oct 31 13:21:43.906784 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:43.906518 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5\" (UID: \"dbd6bd5f-f7b4-46f2-9894-89616d8448ff\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5" Oct 31 13:21:43.906784 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:43.906552 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4rcv\" (UniqueName: \"kubernetes.io/projected/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-kube-api-access-k4rcv\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5\" (UID: \"dbd6bd5f-f7b4-46f2-9894-89616d8448ff\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5" Oct 31 13:21:43.907242 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:43.907220 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5\" (UID: \"dbd6bd5f-f7b4-46f2-9894-89616d8448ff\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5" Oct 31 13:21:43.907345 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:43.907216 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5\" (UID: \"dbd6bd5f-f7b4-46f2-9894-89616d8448ff\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5" Oct 31 13:21:43.909406 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:43.909380 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5\" (UID: \"dbd6bd5f-f7b4-46f2-9894-89616d8448ff\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5" Oct 31 13:21:43.919478 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:43.919422 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4rcv\" (UniqueName: \"kubernetes.io/projected/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-kube-api-access-k4rcv\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5\" (UID: \"dbd6bd5f-f7b4-46f2-9894-89616d8448ff\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5" Oct 31 13:21:43.992031 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:43.991993 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5" Oct 31 13:21:44.298725 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:44.298694 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5"] Oct 31 13:21:44.298885 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:21:44.298854 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbd6bd5f_f7b4_46f2_9894_89616d8448ff.slice/crio-3a578a58cb62f5f4e2154c4bcc14cf5abbb5a34fab98c64d0d6cfca925ab76d5 WatchSource:0}: Error finding container 3a578a58cb62f5f4e2154c4bcc14cf5abbb5a34fab98c64d0d6cfca925ab76d5: Status 404 returned error can't find the container with id 3a578a58cb62f5f4e2154c4bcc14cf5abbb5a34fab98c64d0d6cfca925ab76d5 Oct 31 13:21:44.452708 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:44.452386 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5"] Oct 31 13:21:44.827663 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:44.822843 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b"] Oct 31 13:21:44.827663 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:44.823136 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" podUID="ffe4ec4f-e426-4764-9347-34369d72854e" containerName="istio-proxy" containerID="cri-o://68af4afba643056662ab77775a87487c3890f6af466fcda8bd85e494c856e714" gracePeriod=30 Oct 31 13:21:45.243188 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:45.243150 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5" event={"ID":"dbd6bd5f-f7b4-46f2-9894-89616d8448ff","Type":"ContainerStarted","Data":"e48b7a28bf323039060418b949b3af64c3f376ffe047096edb44ab0a22d92152"} Oct 31 13:21:45.243852 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:45.243214 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5" event={"ID":"dbd6bd5f-f7b4-46f2-9894-89616d8448ff","Type":"ContainerStarted","Data":"3a578a58cb62f5f4e2154c4bcc14cf5abbb5a34fab98c64d0d6cfca925ab76d5"} Oct 31 13:21:45.243852 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:45.243241 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5" Oct 31 13:21:45.243852 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:45.243386 5693 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5" secret="" err="secret \"payload-processing-external-model-e2e--427d574a-dockercfg-xc26m\" not found" Oct 31 13:21:45.245264 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:45.245124 5693 generic.go:356] "Generic (PLEG): container finished" podID="69c36f54-5f25-45ea-b887-c0d2e0f53cfa" containerID="9bf4e45d90ab2d1b7557d892269a9fde0dac279fb391ae18035bc27f3743bb2c" exitCode=0 Oct 31 13:21:45.245264 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:45.245208 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" event={"ID":"69c36f54-5f25-45ea-b887-c0d2e0f53cfa","Type":"ContainerDied","Data":"9bf4e45d90ab2d1b7557d892269a9fde0dac279fb391ae18035bc27f3743bb2c"} Oct 31 13:21:45.245264 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:45.245261 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" event={"ID":"69c36f54-5f25-45ea-b887-c0d2e0f53cfa","Type":"ContainerDied","Data":"7b6a91b171c5591863e17993b19ae50df2f286eb6d4f8815a28168e2a4e3b223"} Oct 31 13:21:45.245500 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:45.245280 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b6a91b171c5591863e17993b19ae50df2f286eb6d4f8815a28168e2a4e3b223" Oct 31 13:21:45.270671 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:45.270630 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" Oct 31 13:21:45.289705 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:45.289614 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5" podStartSLOduration=2.289595457 podStartE2EDuration="2.289595457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:21:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:21:45.284428341 +0000 UTC m=+2955.992014636" watchObservedRunningTime="2026-10-31 13:21:45.289595457 +0000 UTC m=+2955.997181756" Oct 31 13:21:45.368839 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:45.368802 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-istio-data\") pod \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\" (UID: \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\") " Oct 31 13:21:45.369030 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:45.368889 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-workload-certs\") pod \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\" (UID: \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\") " Oct 31 13:21:45.369030 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:45.368981 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-credential-socket\") pod \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\" (UID: \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\") " Oct 31 13:21:45.369030 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:45.369026 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-istio-envoy\") pod \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\" (UID: \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\") " Oct 31 13:21:45.369246 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:45.369067 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-workload-socket\") pod \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\" (UID: \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\") " Oct 31 13:21:45.369246 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:45.369134 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-kube-api-access-hh2kd\" (UniqueName: \"kubernetes.io/projected/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-kube-api-access-hh2kd\") pod \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\" (UID: \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\") " Oct 31 13:21:45.369368 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:45.369290 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-istio-token\" (UniqueName: \"kubernetes.io/projected/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-istio-token\") pod \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\" (UID: \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\") " Oct 31 13:21:45.369368 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:45.369328 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-istio-podinfo\") pod \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\" (UID: \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\") " Oct 31 13:21:45.370916 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:45.369550 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-istiod-ca-cert\") pod \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\" (UID: \"69c36f54-5f25-45ea-b887-c0d2e0f53cfa\") " Oct 31 13:21:45.370916 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:45.370359 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-credential-socket" pod "69c36f54-5f25-45ea-b887-c0d2e0f53cfa" (UID: "69c36f54-5f25-45ea-b887-c0d2e0f53cfa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:21:45.370916 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:21:45.370422 5693 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-plugins-e2e-extmod-a-54a16b: configmap "payload-processing-external-model-plugins-e2e-extmod-a-54a16b" not found Oct 31 13:21:45.370916 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:21:45.370507 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-54a16b/agc-cred-93157c2e27cd: secret "agc-cred-93157c2e27cd" not found Oct 31 13:21:45.370916 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:21:45.370508 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-plugins-config-volume podName:dbd6bd5f-f7b4-46f2-9894-89616d8448ff nodeName:}" failed. No retries permitted until 2026-10-31 13:21:45.87047259 +0000 UTC m=+2956.578058882 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5" (UID: "dbd6bd5f-f7b4-46f2-9894-89616d8448ff") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-54a16b" not found Oct 31 13:21:45.370916 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:21:45.370523 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5: secret "agc-cred-93157c2e27cd" not found Oct 31 13:21:45.370916 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:21:45.370717 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-provider-credentials podName:dbd6bd5f-f7b4-46f2-9894-89616d8448ff nodeName:}" failed. No retries permitted until 2026-10-31 13:21:45.870670002 +0000 UTC m=+2956.578256309 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5" (UID: "dbd6bd5f-f7b4-46f2-9894-89616d8448ff") : secret "agc-cred-93157c2e27cd" not found Oct 31 13:21:45.370916 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:45.370889 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-istiod-ca-cert" pod "69c36f54-5f25-45ea-b887-c0d2e0f53cfa" (UID: "69c36f54-5f25-45ea-b887-c0d2e0f53cfa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:21:45.371475 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:45.371143 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-istio-data" pod "69c36f54-5f25-45ea-b887-c0d2e0f53cfa" (UID: "69c36f54-5f25-45ea-b887-c0d2e0f53cfa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:21:45.371475 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:45.371326 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-workload-socket" pod "69c36f54-5f25-45ea-b887-c0d2e0f53cfa" (UID: "69c36f54-5f25-45ea-b887-c0d2e0f53cfa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:21:45.372050 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:45.372022 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-workload-certs" pod "69c36f54-5f25-45ea-b887-c0d2e0f53cfa" (UID: "69c36f54-5f25-45ea-b887-c0d2e0f53cfa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:21:45.373789 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:45.373762 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-istio-podinfo" pod "69c36f54-5f25-45ea-b887-c0d2e0f53cfa" (UID: "69c36f54-5f25-45ea-b887-c0d2e0f53cfa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:21:45.373912 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:45.373823 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-istio-token" pod "69c36f54-5f25-45ea-b887-c0d2e0f53cfa" (UID: "69c36f54-5f25-45ea-b887-c0d2e0f53cfa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:21:45.373989 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:45.373921 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-istio-envoy" pod "69c36f54-5f25-45ea-b887-c0d2e0f53cfa" (UID: "69c36f54-5f25-45ea-b887-c0d2e0f53cfa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:21:45.374279 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:45.374258 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-kube-api-access-hh2kd" pod "69c36f54-5f25-45ea-b887-c0d2e0f53cfa" (UID: "69c36f54-5f25-45ea-b887-c0d2e0f53cfa"). InnerVolumeSpecName "kube-api-access-hh2kd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:21:45.472835 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:45.472798 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:45.472835 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:45.472838 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:45.473069 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:45.472856 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:45.473069 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:45.472871 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:45.473069 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:45.472886 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:45.473069 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:45.472901 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:45.473069 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:45.472915 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh2kd\" (UniqueName: \"kubernetes.io/projected/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-kube-api-access-hh2kd\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:45.473069 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:45.472931 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:45.473069 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:45.472945 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69c36f54-5f25-45ea-b887-c0d2e0f53cfa-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:45.827129 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:45.827053 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" Oct 31 13:21:45.827601 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:45.827575 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" Oct 31 13:21:45.828788 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:45.828765 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" Oct 31 13:21:45.892010 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:21:45.891972 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-54a16b/agc-cred-93157c2e27cd: secret "agc-cred-93157c2e27cd" not found Oct 31 13:21:45.892211 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:21:45.892006 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5: secret "agc-cred-93157c2e27cd" not found Oct 31 13:21:45.892211 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:21:45.892107 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-provider-credentials podName:dbd6bd5f-f7b4-46f2-9894-89616d8448ff nodeName:}" failed. No retries permitted until 2026-10-31 13:21:46.892069625 +0000 UTC m=+2957.599655920 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5" (UID: "dbd6bd5f-f7b4-46f2-9894-89616d8448ff") : secret "agc-cred-93157c2e27cd" not found Oct 31 13:21:45.892374 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:21:45.892353 5693 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-plugins-e2e-extmod-a-54a16b: configmap "payload-processing-external-model-plugins-e2e-extmod-a-54a16b" not found Oct 31 13:21:45.892433 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:21:45.892406 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-plugins-config-volume podName:dbd6bd5f-f7b4-46f2-9894-89616d8448ff nodeName:}" failed. No retries permitted until 2026-10-31 13:21:46.892394145 +0000 UTC m=+2957.599980418 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5" (UID: "dbd6bd5f-f7b4-46f2-9894-89616d8448ff") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-54a16b" not found Oct 31 13:21:46.252610 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:46.252581 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg" Oct 31 13:21:46.292110 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:46.292070 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg"] Oct 31 13:21:46.296976 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:46.296936 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-5nwhg"] Oct 31 13:21:46.907748 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:21:46.907705 5693 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-plugins-e2e-extmod-a-54a16b: configmap "payload-processing-external-model-plugins-e2e-extmod-a-54a16b" not found Oct 31 13:21:46.907972 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:21:46.907933 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-plugins-config-volume podName:dbd6bd5f-f7b4-46f2-9894-89616d8448ff nodeName:}" failed. No retries permitted until 2026-10-31 13:21:48.907903565 +0000 UTC m=+2959.615489840 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5" (UID: "dbd6bd5f-f7b4-46f2-9894-89616d8448ff") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-54a16b" not found Oct 31 13:21:46.907972 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:21:46.907949 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-54a16b/agc-cred-93157c2e27cd: secret "agc-cred-93157c2e27cd" not found Oct 31 13:21:46.908105 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:21:46.907970 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5: secret "agc-cred-93157c2e27cd" not found Oct 31 13:21:46.908105 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:21:46.908033 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-provider-credentials podName:dbd6bd5f-f7b4-46f2-9894-89616d8448ff nodeName:}" failed. No retries permitted until 2026-10-31 13:21:48.908017613 +0000 UTC m=+2959.615603903 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5" (UID: "dbd6bd5f-f7b4-46f2-9894-89616d8448ff") : secret "agc-cred-93157c2e27cd" not found Oct 31 13:21:47.410451 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:47.410418 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69c36f54-5f25-45ea-b887-c0d2e0f53cfa" path="/var/lib/kubelet/pods/69c36f54-5f25-45ea-b887-c0d2e0f53cfa/volumes" Oct 31 13:21:48.693994 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:21:48.693956 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:21:48.942425 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:21:48.942384 5693 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-plugins-e2e-extmod-a-54a16b: configmap "payload-processing-external-model-plugins-e2e-extmod-a-54a16b" not found Oct 31 13:21:48.942962 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:21:48.942739 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-54a16b/agc-cred-93157c2e27cd: secret "agc-cred-93157c2e27cd" not found Oct 31 13:21:48.942962 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:21:48.942885 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5: secret "agc-cred-93157c2e27cd" not found Oct 31 13:21:48.942962 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:21:48.942942 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-provider-credentials podName:dbd6bd5f-f7b4-46f2-9894-89616d8448ff nodeName:}" failed. No retries permitted until 2026-10-31 13:21:52.942920513 +0000 UTC m=+2963.650506824 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5" (UID: "dbd6bd5f-f7b4-46f2-9894-89616d8448ff") : secret "agc-cred-93157c2e27cd" not found Oct 31 13:21:48.943819 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:21:48.943762 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-plugins-config-volume podName:dbd6bd5f-f7b4-46f2-9894-89616d8448ff nodeName:}" failed. No retries permitted until 2026-10-31 13:21:52.943629295 +0000 UTC m=+2963.651215581 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5" (UID: "dbd6bd5f-f7b4-46f2-9894-89616d8448ff") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-54a16b" not found Oct 31 13:21:50.289206 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:50.289176 5693 generic.go:356] "Generic (PLEG): container finished" podID="ffe4ec4f-e426-4764-9347-34369d72854e" containerID="68af4afba643056662ab77775a87487c3890f6af466fcda8bd85e494c856e714" exitCode=0 Oct 31 13:21:50.289581 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:50.289246 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" event={"ID":"ffe4ec4f-e426-4764-9347-34369d72854e","Type":"ContainerDied","Data":"68af4afba643056662ab77775a87487c3890f6af466fcda8bd85e494c856e714"} Oct 31 13:21:50.342866 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:50.342839 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" Oct 31 13:21:50.366396 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:50.366369 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffe4ec4f-e426-4764-9347-34369d72854e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffe4ec4f-e426-4764-9347-34369d72854e-istiod-ca-cert\") pod \"ffe4ec4f-e426-4764-9347-34369d72854e\" (UID: \"ffe4ec4f-e426-4764-9347-34369d72854e\") " Oct 31 13:21:50.366525 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:50.366437 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffe4ec4f-e426-4764-9347-34369d72854e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffe4ec4f-e426-4764-9347-34369d72854e-istio-envoy\") pod \"ffe4ec4f-e426-4764-9347-34369d72854e\" (UID: \"ffe4ec4f-e426-4764-9347-34369d72854e\") " Oct 31 13:21:50.366525 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:50.366515 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ffe4ec4f-e426-4764-9347-34369d72854e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffe4ec4f-e426-4764-9347-34369d72854e-istio-podinfo\") pod \"ffe4ec4f-e426-4764-9347-34369d72854e\" (UID: \"ffe4ec4f-e426-4764-9347-34369d72854e\") " Oct 31 13:21:50.366608 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:50.366575 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffe4ec4f-e426-4764-9347-34369d72854e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe4ec4f-e426-4764-9347-34369d72854e-credential-socket\") pod \"ffe4ec4f-e426-4764-9347-34369d72854e\" (UID: \"ffe4ec4f-e426-4764-9347-34369d72854e\") " Oct 31 13:21:50.366714 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:50.366698 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffe4ec4f-e426-4764-9347-34369d72854e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffe4ec4f-e426-4764-9347-34369d72854e-workload-certs\") pod \"ffe4ec4f-e426-4764-9347-34369d72854e\" (UID: \"ffe4ec4f-e426-4764-9347-34369d72854e\") " Oct 31 13:21:50.366787 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:50.366764 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffe4ec4f-e426-4764-9347-34369d72854e-istiod-ca-cert" pod "ffe4ec4f-e426-4764-9347-34369d72854e" (UID: "ffe4ec4f-e426-4764-9347-34369d72854e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:21:50.366912 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:50.366863 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe4ec4f-e426-4764-9347-34369d72854e-credential-socket" pod "ffe4ec4f-e426-4764-9347-34369d72854e" (UID: "ffe4ec4f-e426-4764-9347-34369d72854e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:21:50.366995 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:50.366770 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffe4ec4f-e426-4764-9347-34369d72854e-kube-api-access-bfm9f\" (UniqueName: \"kubernetes.io/projected/ffe4ec4f-e426-4764-9347-34369d72854e-kube-api-access-bfm9f\") pod \"ffe4ec4f-e426-4764-9347-34369d72854e\" (UID: \"ffe4ec4f-e426-4764-9347-34369d72854e\") " Oct 31 13:21:50.366995 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:50.366964 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe4ec4f-e426-4764-9347-34369d72854e-workload-certs" pod "ffe4ec4f-e426-4764-9347-34369d72854e" (UID: "ffe4ec4f-e426-4764-9347-34369d72854e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:21:50.366995 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:50.366989 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffe4ec4f-e426-4764-9347-34369d72854e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffe4ec4f-e426-4764-9347-34369d72854e-istio-data\") pod \"ffe4ec4f-e426-4764-9347-34369d72854e\" (UID: \"ffe4ec4f-e426-4764-9347-34369d72854e\") " Oct 31 13:21:50.367173 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:50.367102 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffe4ec4f-e426-4764-9347-34369d72854e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe4ec4f-e426-4764-9347-34369d72854e-workload-socket\") pod \"ffe4ec4f-e426-4764-9347-34369d72854e\" (UID: \"ffe4ec4f-e426-4764-9347-34369d72854e\") " Oct 31 13:21:50.367232 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:50.367177 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffe4ec4f-e426-4764-9347-34369d72854e-istio-token\" (UniqueName: \"kubernetes.io/projected/ffe4ec4f-e426-4764-9347-34369d72854e-istio-token\") pod \"ffe4ec4f-e426-4764-9347-34369d72854e\" (UID: \"ffe4ec4f-e426-4764-9347-34369d72854e\") " Oct 31 13:21:50.367287 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:50.367227 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe4ec4f-e426-4764-9347-34369d72854e-istio-data" pod "ffe4ec4f-e426-4764-9347-34369d72854e" (UID: "ffe4ec4f-e426-4764-9347-34369d72854e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:21:50.367432 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:50.367398 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe4ec4f-e426-4764-9347-34369d72854e-workload-socket" pod "ffe4ec4f-e426-4764-9347-34369d72854e" (UID: "ffe4ec4f-e426-4764-9347-34369d72854e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:21:50.367972 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:50.367953 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffe4ec4f-e426-4764-9347-34369d72854e-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:50.367972 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:50.367971 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffe4ec4f-e426-4764-9347-34369d72854e-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:50.368340 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:50.367980 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe4ec4f-e426-4764-9347-34369d72854e-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:50.368340 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:50.367991 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffe4ec4f-e426-4764-9347-34369d72854e-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:50.368340 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:50.368002 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe4ec4f-e426-4764-9347-34369d72854e-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:50.368998 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:50.368975 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ffe4ec4f-e426-4764-9347-34369d72854e-istio-podinfo" pod "ffe4ec4f-e426-4764-9347-34369d72854e" (UID: "ffe4ec4f-e426-4764-9347-34369d72854e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:21:50.369563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:50.369535 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe4ec4f-e426-4764-9347-34369d72854e-istio-envoy" pod "ffe4ec4f-e426-4764-9347-34369d72854e" (UID: "ffe4ec4f-e426-4764-9347-34369d72854e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:21:50.369563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:50.369555 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffe4ec4f-e426-4764-9347-34369d72854e-kube-api-access-bfm9f" pod "ffe4ec4f-e426-4764-9347-34369d72854e" (UID: "ffe4ec4f-e426-4764-9347-34369d72854e"). InnerVolumeSpecName "kube-api-access-bfm9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:21:50.370036 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:50.370003 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffe4ec4f-e426-4764-9347-34369d72854e-istio-token" pod "ffe4ec4f-e426-4764-9347-34369d72854e" (UID: "ffe4ec4f-e426-4764-9347-34369d72854e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:21:50.470115 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:50.470075 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffe4ec4f-e426-4764-9347-34369d72854e-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:50.470115 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:50.470108 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffe4ec4f-e426-4764-9347-34369d72854e-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:50.470367 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:50.470124 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfm9f\" (UniqueName: \"kubernetes.io/projected/ffe4ec4f-e426-4764-9347-34369d72854e-kube-api-access-bfm9f\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:50.470367 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:50.470141 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffe4ec4f-e426-4764-9347-34369d72854e-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:51.253615 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:51.253576 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5" podUID="dbd6bd5f-f7b4-46f2-9894-89616d8448ff" containerName="payload-processing" containerID="cri-o://e48b7a28bf323039060418b949b3af64c3f376ffe047096edb44ab0a22d92152" gracePeriod=25 Oct 31 13:21:51.298855 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:51.298826 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" Oct 31 13:21:51.299278 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:51.298858 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b" event={"ID":"ffe4ec4f-e426-4764-9347-34369d72854e","Type":"ContainerDied","Data":"c8529e835b4f79d828c31bc297631c23564e76668bca5830dd3e9c575f91cdce"} Oct 31 13:21:51.299278 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:51.298928 5693 scope.go:122] "RemoveContainer" containerID="68af4afba643056662ab77775a87487c3890f6af466fcda8bd85e494c856e714" Oct 31 13:21:51.368940 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:51.368898 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b"] Oct 31 13:21:51.370448 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:51.370418 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-vg99b"] Oct 31 13:21:51.450303 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:51.450276 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffe4ec4f-e426-4764-9347-34369d72854e" path="/var/lib/kubelet/pods/ffe4ec4f-e426-4764-9347-34369d72854e/volumes" Oct 31 13:21:51.621130 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:51.621098 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5" Oct 31 13:21:51.717838 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:51.717806 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-kube-api-access-k4rcv\" (UniqueName: \"kubernetes.io/projected/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-kube-api-access-k4rcv\") pod \"dbd6bd5f-f7b4-46f2-9894-89616d8448ff\" (UID: \"dbd6bd5f-f7b4-46f2-9894-89616d8448ff\") " Oct 31 13:21:51.718088 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:51.718024 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-plugins-config-volume\") pod \"dbd6bd5f-f7b4-46f2-9894-89616d8448ff\" (UID: \"dbd6bd5f-f7b4-46f2-9894-89616d8448ff\") " Oct 31 13:21:51.718088 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:51.718078 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-provider-credentials\" (UniqueName: \"kubernetes.io/projected/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-provider-credentials\") pod \"dbd6bd5f-f7b4-46f2-9894-89616d8448ff\" (UID: \"dbd6bd5f-f7b4-46f2-9894-89616d8448ff\") " Oct 31 13:21:51.718221 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:51.718117 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-routing-overlay-volume\") pod \"dbd6bd5f-f7b4-46f2-9894-89616d8448ff\" (UID: \"dbd6bd5f-f7b4-46f2-9894-89616d8448ff\") " Oct 31 13:21:51.718458 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:51.718432 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-plugins-config-volume" pod "dbd6bd5f-f7b4-46f2-9894-89616d8448ff" (UID: "dbd6bd5f-f7b4-46f2-9894-89616d8448ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:21:51.718523 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:51.718497 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-routing-overlay-volume" pod "dbd6bd5f-f7b4-46f2-9894-89616d8448ff" (UID: "dbd6bd5f-f7b4-46f2-9894-89616d8448ff"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:21:51.718850 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:51.718831 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-routing-overlay-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:51.718916 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:51.718853 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:51.720286 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:51.720260 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-kube-api-access-k4rcv" pod "dbd6bd5f-f7b4-46f2-9894-89616d8448ff" (UID: "dbd6bd5f-f7b4-46f2-9894-89616d8448ff"). InnerVolumeSpecName "kube-api-access-k4rcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:21:51.720541 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:51.720527 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-provider-credentials" pod "dbd6bd5f-f7b4-46f2-9894-89616d8448ff" (UID: "dbd6bd5f-f7b4-46f2-9894-89616d8448ff"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:21:51.819980 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:51.819894 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-provider-credentials\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:51.819980 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:51.819922 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4rcv\" (UniqueName: \"kubernetes.io/projected/dbd6bd5f-f7b4-46f2-9894-89616d8448ff-kube-api-access-k4rcv\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:52.307851 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:52.307815 5693 generic.go:356] "Generic (PLEG): container finished" podID="dbd6bd5f-f7b4-46f2-9894-89616d8448ff" containerID="e48b7a28bf323039060418b949b3af64c3f376ffe047096edb44ab0a22d92152" exitCode=0 Oct 31 13:21:52.308327 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:52.307863 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5" Oct 31 13:21:52.308327 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:52.307896 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5" event={"ID":"dbd6bd5f-f7b4-46f2-9894-89616d8448ff","Type":"ContainerDied","Data":"e48b7a28bf323039060418b949b3af64c3f376ffe047096edb44ab0a22d92152"} Oct 31 13:21:52.308327 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:52.307946 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5" event={"ID":"dbd6bd5f-f7b4-46f2-9894-89616d8448ff","Type":"ContainerDied","Data":"3a578a58cb62f5f4e2154c4bcc14cf5abbb5a34fab98c64d0d6cfca925ab76d5"} Oct 31 13:21:52.308327 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:52.307970 5693 scope.go:122] "RemoveContainer" containerID="e48b7a28bf323039060418b949b3af64c3f376ffe047096edb44ab0a22d92152" Oct 31 13:21:52.320461 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:52.320441 5693 scope.go:122] "RemoveContainer" containerID="e48b7a28bf323039060418b949b3af64c3f376ffe047096edb44ab0a22d92152" Oct 31 13:21:52.320773 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:21:52.320741 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e48b7a28bf323039060418b949b3af64c3f376ffe047096edb44ab0a22d92152\": container with ID starting with e48b7a28bf323039060418b949b3af64c3f376ffe047096edb44ab0a22d92152 not found: ID does not exist" containerID="e48b7a28bf323039060418b949b3af64c3f376ffe047096edb44ab0a22d92152" Oct 31 13:21:52.320842 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:52.320767 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e48b7a28bf323039060418b949b3af64c3f376ffe047096edb44ab0a22d92152"} err="failed to get container status \"e48b7a28bf323039060418b949b3af64c3f376ffe047096edb44ab0a22d92152\": rpc error: code = NotFound desc = could not find container \"e48b7a28bf323039060418b949b3af64c3f376ffe047096edb44ab0a22d92152\": container with ID starting with e48b7a28bf323039060418b949b3af64c3f376ffe047096edb44ab0a22d92152 not found: ID does not exist" Oct 31 13:21:52.342740 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:52.342702 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5"] Oct 31 13:21:52.346796 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:52.346774 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-6d4f8bsc5"] Oct 31 13:21:53.411054 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:21:53.411020 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbd6bd5f-f7b4-46f2-9894-89616d8448ff" path="/var/lib/kubelet/pods/dbd6bd5f-f7b4-46f2-9894-89616d8448ff/volumes" Oct 31 13:22:00.532691 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:00.532632 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-76758ccf48-lxlmg"] Oct 31 13:22:00.533503 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:00.533477 5693 cpu_manager.go:414] "containerMap: removing container" podUID="69c36f54-5f25-45ea-b887-c0d2e0f53cfa" containerName="istio-proxy" Oct 31 13:22:00.533503 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:00.533495 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69c36f54-5f25-45ea-b887-c0d2e0f53cfa" containerName="istio-proxy" Oct 31 13:22:00.533671 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:00.533543 5693 cpu_manager.go:414] "containerMap: removing container" podUID="ffe4ec4f-e426-4764-9347-34369d72854e" containerName="istio-proxy" Oct 31 13:22:00.533671 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:00.533548 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffe4ec4f-e426-4764-9347-34369d72854e" containerName="istio-proxy" Oct 31 13:22:00.533671 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:00.533560 5693 cpu_manager.go:414] "containerMap: removing container" podUID="dbd6bd5f-f7b4-46f2-9894-89616d8448ff" containerName="payload-processing" Oct 31 13:22:00.533671 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:00.533565 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbd6bd5f-f7b4-46f2-9894-89616d8448ff" containerName="payload-processing" Oct 31 13:22:00.533816 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:00.533700 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffe4ec4f-e426-4764-9347-34369d72854e" containerName="istio-proxy" Oct 31 13:22:00.533816 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:00.533711 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbd6bd5f-f7b4-46f2-9894-89616d8448ff" containerName="payload-processing" Oct 31 13:22:00.533816 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:00.533721 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="69c36f54-5f25-45ea-b887-c0d2e0f53cfa" containerName="istio-proxy" Oct 31 13:22:00.537493 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:00.537476 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-76758ccf48-lxlmg" Oct 31 13:22:00.545688 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:00.545653 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-76758ccf48-lxlmg"] Oct 31 13:22:00.547725 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:00.547706 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-6ac4bb21ecb0\"" Oct 31 13:22:00.580709 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:00.580669 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31021a03-2b9d-4db5-8934-78bf39bd308f-provider-credentials\") pod \"payload-processing-external-model-76758ccf48-lxlmg\" (UID: \"31021a03-2b9d-4db5-8934-78bf39bd308f\") " pod="models-as-a-service/payload-processing-external-model-76758ccf48-lxlmg" Oct 31 13:22:00.580884 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:00.580864 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31021a03-2b9d-4db5-8934-78bf39bd308f-routing-overlay-volume\") pod \"payload-processing-external-model-76758ccf48-lxlmg\" (UID: \"31021a03-2b9d-4db5-8934-78bf39bd308f\") " pod="models-as-a-service/payload-processing-external-model-76758ccf48-lxlmg" Oct 31 13:22:00.580933 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:00.580919 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31021a03-2b9d-4db5-8934-78bf39bd308f-plugins-config-volume\") pod \"payload-processing-external-model-76758ccf48-lxlmg\" (UID: \"31021a03-2b9d-4db5-8934-78bf39bd308f\") " pod="models-as-a-service/payload-processing-external-model-76758ccf48-lxlmg" Oct 31 13:22:00.581033 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:00.581022 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9b7v\" (UniqueName: \"kubernetes.io/projected/31021a03-2b9d-4db5-8934-78bf39bd308f-kube-api-access-b9b7v\") pod \"payload-processing-external-model-76758ccf48-lxlmg\" (UID: \"31021a03-2b9d-4db5-8934-78bf39bd308f\") " pod="models-as-a-service/payload-processing-external-model-76758ccf48-lxlmg" Oct 31 13:22:00.682920 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:00.682886 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31021a03-2b9d-4db5-8934-78bf39bd308f-provider-credentials\") pod \"payload-processing-external-model-76758ccf48-lxlmg\" (UID: \"31021a03-2b9d-4db5-8934-78bf39bd308f\") " pod="models-as-a-service/payload-processing-external-model-76758ccf48-lxlmg" Oct 31 13:22:00.683109 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:00.683052 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31021a03-2b9d-4db5-8934-78bf39bd308f-routing-overlay-volume\") pod \"payload-processing-external-model-76758ccf48-lxlmg\" (UID: \"31021a03-2b9d-4db5-8934-78bf39bd308f\") " pod="models-as-a-service/payload-processing-external-model-76758ccf48-lxlmg" Oct 31 13:22:00.683109 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:00.683100 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31021a03-2b9d-4db5-8934-78bf39bd308f-plugins-config-volume\") pod \"payload-processing-external-model-76758ccf48-lxlmg\" (UID: \"31021a03-2b9d-4db5-8934-78bf39bd308f\") " pod="models-as-a-service/payload-processing-external-model-76758ccf48-lxlmg" Oct 31 13:22:00.683258 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:00.683159 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9b7v\" (UniqueName: \"kubernetes.io/projected/31021a03-2b9d-4db5-8934-78bf39bd308f-kube-api-access-b9b7v\") pod \"payload-processing-external-model-76758ccf48-lxlmg\" (UID: \"31021a03-2b9d-4db5-8934-78bf39bd308f\") " pod="models-as-a-service/payload-processing-external-model-76758ccf48-lxlmg" Oct 31 13:22:00.683887 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:00.683862 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31021a03-2b9d-4db5-8934-78bf39bd308f-routing-overlay-volume\") pod \"payload-processing-external-model-76758ccf48-lxlmg\" (UID: \"31021a03-2b9d-4db5-8934-78bf39bd308f\") " pod="models-as-a-service/payload-processing-external-model-76758ccf48-lxlmg" Oct 31 13:22:00.684017 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:00.683968 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31021a03-2b9d-4db5-8934-78bf39bd308f-plugins-config-volume\") pod \"payload-processing-external-model-76758ccf48-lxlmg\" (UID: \"31021a03-2b9d-4db5-8934-78bf39bd308f\") " pod="models-as-a-service/payload-processing-external-model-76758ccf48-lxlmg" Oct 31 13:22:00.685694 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:00.685671 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31021a03-2b9d-4db5-8934-78bf39bd308f-provider-credentials\") pod \"payload-processing-external-model-76758ccf48-lxlmg\" (UID: \"31021a03-2b9d-4db5-8934-78bf39bd308f\") " pod="models-as-a-service/payload-processing-external-model-76758ccf48-lxlmg" Oct 31 13:22:00.696014 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:00.695989 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9b7v\" (UniqueName: \"kubernetes.io/projected/31021a03-2b9d-4db5-8934-78bf39bd308f-kube-api-access-b9b7v\") pod \"payload-processing-external-model-76758ccf48-lxlmg\" (UID: \"31021a03-2b9d-4db5-8934-78bf39bd308f\") " pod="models-as-a-service/payload-processing-external-model-76758ccf48-lxlmg" Oct 31 13:22:00.855109 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:00.855023 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-76758ccf48-lxlmg" Oct 31 13:22:01.183233 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:01.183202 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-76758ccf48-lxlmg"] Oct 31 13:22:01.183546 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:22:01.183525 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31021a03_2b9d_4db5_8934_78bf39bd308f.slice/crio-efa298b0db10c6a088aabd62877ab460905754a3e4d9a1c2ec389e65b0ece1cf WatchSource:0}: Error finding container efa298b0db10c6a088aabd62877ab460905754a3e4d9a1c2ec389e65b0ece1cf: Status 404 returned error can't find the container with id efa298b0db10c6a088aabd62877ab460905754a3e4d9a1c2ec389e65b0ece1cf Oct 31 13:22:01.417130 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:01.417092 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-76758ccf48-lxlmg" event={"ID":"31021a03-2b9d-4db5-8934-78bf39bd308f","Type":"ContainerStarted","Data":"26cdf73a08f1cc8994f92a97a09560716c674103fa0b93f27535d88031643dc6"} Oct 31 13:22:01.417329 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:01.417143 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-76758ccf48-lxlmg" event={"ID":"31021a03-2b9d-4db5-8934-78bf39bd308f","Type":"ContainerStarted","Data":"efa298b0db10c6a088aabd62877ab460905754a3e4d9a1c2ec389e65b0ece1cf"} Oct 31 13:22:01.417329 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:01.417169 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-76758ccf48-lxlmg" Oct 31 13:22:01.443430 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:01.443332 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-76758ccf48-lxlmg" podStartSLOduration=1.443317349 podStartE2EDuration="1.443317349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:22:01.439022085 +0000 UTC m=+2972.146608384" watchObservedRunningTime="2026-10-31 13:22:01.443317349 +0000 UTC m=+2972.150903643" Oct 31 13:22:12.428659 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:12.428573 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-76758ccf48-lxlmg" Oct 31 13:22:12.486589 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:12.486552 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d47957896-7c8qs"] Oct 31 13:22:17.487649 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:17.487607 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5d47957896-7c8qs" podUID="9e64beca-4cd9-4507-a577-349e369ae4b8" containerName="payload-processing" containerID="cri-o://ba6a898d87137b11698cbfdeb235125a186892c773c201c9fc3ba5730ae101a7" gracePeriod=25 Oct 31 13:22:17.755922 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:17.755420 5693 generic.go:356] "Generic (PLEG): container finished" podID="9e64beca-4cd9-4507-a577-349e369ae4b8" containerID="ba6a898d87137b11698cbfdeb235125a186892c773c201c9fc3ba5730ae101a7" exitCode=0 Oct 31 13:22:17.755922 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:17.755785 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d47957896-7c8qs" event={"ID":"9e64beca-4cd9-4507-a577-349e369ae4b8","Type":"ContainerDied","Data":"ba6a898d87137b11698cbfdeb235125a186892c773c201c9fc3ba5730ae101a7"} Oct 31 13:22:17.860140 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:17.860106 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d47957896-7c8qs" Oct 31 13:22:17.963857 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:17.963827 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e64beca-4cd9-4507-a577-349e369ae4b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e64beca-4cd9-4507-a577-349e369ae4b8-plugins-config-volume\") pod \"9e64beca-4cd9-4507-a577-349e369ae4b8\" (UID: \"9e64beca-4cd9-4507-a577-349e369ae4b8\") " Oct 31 13:22:17.964054 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:17.963934 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e64beca-4cd9-4507-a577-349e369ae4b8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9e64beca-4cd9-4507-a577-349e369ae4b8-provider-credentials\") pod \"9e64beca-4cd9-4507-a577-349e369ae4b8\" (UID: \"9e64beca-4cd9-4507-a577-349e369ae4b8\") " Oct 31 13:22:17.964106 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:17.964054 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e64beca-4cd9-4507-a577-349e369ae4b8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9e64beca-4cd9-4507-a577-349e369ae4b8-routing-overlay-volume\") pod \"9e64beca-4cd9-4507-a577-349e369ae4b8\" (UID: \"9e64beca-4cd9-4507-a577-349e369ae4b8\") " Oct 31 13:22:17.964149 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:17.964132 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e64beca-4cd9-4507-a577-349e369ae4b8-kube-api-access-kcwzf\" (UniqueName: \"kubernetes.io/projected/9e64beca-4cd9-4507-a577-349e369ae4b8-kube-api-access-kcwzf\") pod \"9e64beca-4cd9-4507-a577-349e369ae4b8\" (UID: \"9e64beca-4cd9-4507-a577-349e369ae4b8\") " Oct 31 13:22:17.964419 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:17.964394 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e64beca-4cd9-4507-a577-349e369ae4b8-plugins-config-volume" pod "9e64beca-4cd9-4507-a577-349e369ae4b8" (UID: "9e64beca-4cd9-4507-a577-349e369ae4b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:22:17.964517 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:17.964497 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e64beca-4cd9-4507-a577-349e369ae4b8-routing-overlay-volume" pod "9e64beca-4cd9-4507-a577-349e369ae4b8" (UID: "9e64beca-4cd9-4507-a577-349e369ae4b8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:22:17.964892 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:17.964875 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e64beca-4cd9-4507-a577-349e369ae4b8-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:22:17.964963 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:17.964894 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9e64beca-4cd9-4507-a577-349e369ae4b8-routing-overlay-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:22:17.966292 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:17.966274 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e64beca-4cd9-4507-a577-349e369ae4b8-provider-credentials" pod "9e64beca-4cd9-4507-a577-349e369ae4b8" (UID: "9e64beca-4cd9-4507-a577-349e369ae4b8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:22:17.966504 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:17.966487 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e64beca-4cd9-4507-a577-349e369ae4b8-kube-api-access-kcwzf" pod "9e64beca-4cd9-4507-a577-349e369ae4b8" (UID: "9e64beca-4cd9-4507-a577-349e369ae4b8"). InnerVolumeSpecName "kube-api-access-kcwzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:22:18.066094 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:18.066005 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9e64beca-4cd9-4507-a577-349e369ae4b8-provider-credentials\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:22:18.066094 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:18.066037 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcwzf\" (UniqueName: \"kubernetes.io/projected/9e64beca-4cd9-4507-a577-349e369ae4b8-kube-api-access-kcwzf\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:22:18.689576 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:22:18.689540 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:22:18.764968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:18.764937 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d47957896-7c8qs" Oct 31 13:22:18.765162 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:18.764970 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d47957896-7c8qs" event={"ID":"9e64beca-4cd9-4507-a577-349e369ae4b8","Type":"ContainerDied","Data":"18402c3368bec60dbf7b23ed103c8820624e9bb2524d67c9fac07c0784bf8880"} Oct 31 13:22:18.765162 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:18.765043 5693 scope.go:122] "RemoveContainer" containerID="ba6a898d87137b11698cbfdeb235125a186892c773c201c9fc3ba5730ae101a7" Oct 31 13:22:18.820199 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:18.820167 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d47957896-7c8qs"] Oct 31 13:22:18.834353 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:18.834323 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d47957896-7c8qs"] Oct 31 13:22:19.410841 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:19.410807 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e64beca-4cd9-4507-a577-349e369ae4b8" path="/var/lib/kubelet/pods/9e64beca-4cd9-4507-a577-349e369ae4b8/volumes" Oct 31 13:22:26.973835 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:26.973784 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fv482 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:22:26.974369 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:26.973876 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" podUID="c0bdd69c-691b-47b2-ba17-2e0ac7185643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:22:30.827031 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:30.826996 5693 scope.go:122] "RemoveContainer" containerID="9bf4e45d90ab2d1b7557d892269a9fde0dac279fb391ae18035bc27f3743bb2c" Oct 31 13:22:32.331532 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:32.331497 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx"] Oct 31 13:22:32.332555 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:32.332535 5693 cpu_manager.go:414] "containerMap: removing container" podUID="9e64beca-4cd9-4507-a577-349e369ae4b8" containerName="payload-processing" Oct 31 13:22:32.332555 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:32.332556 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e64beca-4cd9-4507-a577-349e369ae4b8" containerName="payload-processing" Oct 31 13:22:32.332760 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:32.332713 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e64beca-4cd9-4507-a577-349e369ae4b8" containerName="payload-processing" Oct 31 13:22:32.355701 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:32.353610 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx" Oct 31 13:22:32.359096 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:32.359046 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9ef744\"/\"payload-processing-external-model-plugins-e2e-extmod-a-9ef744\"" Oct 31 13:22:32.371460 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:32.371414 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9ef744\"/\"payload-processing-external-model-e2e--4724aaf0-dockercfg-fj5fx\"" Oct 31 13:22:32.377055 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:32.375449 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9ef744\"/\"openshift-service-ca.crt\"" Oct 31 13:22:32.377055 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:32.375529 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9ef744\"/\"agc-cred-225017692890\"" Oct 31 13:22:32.377055 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:32.375615 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9ef744\"/\"kube-root-ca.crt\"" Oct 31 13:22:32.377055 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:32.375695 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9ef744\"/\"routing-overlay\"" Oct 31 13:22:32.433065 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:32.433028 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx"] Oct 31 13:22:32.470295 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:32.464603 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzpkh\" (UniqueName: \"kubernetes.io/projected/30e9e170-b40a-4f10-b43c-984e3d9083e3-kube-api-access-pzpkh\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx\" (UID: \"30e9e170-b40a-4f10-b43c-984e3d9083e3\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx" Oct 31 13:22:32.470295 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:32.464740 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/30e9e170-b40a-4f10-b43c-984e3d9083e3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx\" (UID: \"30e9e170-b40a-4f10-b43c-984e3d9083e3\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx" Oct 31 13:22:32.470295 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:32.464862 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/30e9e170-b40a-4f10-b43c-984e3d9083e3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx\" (UID: \"30e9e170-b40a-4f10-b43c-984e3d9083e3\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx" Oct 31 13:22:32.470295 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:32.464957 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30e9e170-b40a-4f10-b43c-984e3d9083e3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx\" (UID: \"30e9e170-b40a-4f10-b43c-984e3d9083e3\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx" Oct 31 13:22:32.566386 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:32.566357 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzpkh\" (UniqueName: \"kubernetes.io/projected/30e9e170-b40a-4f10-b43c-984e3d9083e3-kube-api-access-pzpkh\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx\" (UID: \"30e9e170-b40a-4f10-b43c-984e3d9083e3\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx" Oct 31 13:22:32.566592 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:32.566437 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/30e9e170-b40a-4f10-b43c-984e3d9083e3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx\" (UID: \"30e9e170-b40a-4f10-b43c-984e3d9083e3\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx" Oct 31 13:22:32.566592 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:32.566542 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/30e9e170-b40a-4f10-b43c-984e3d9083e3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx\" (UID: \"30e9e170-b40a-4f10-b43c-984e3d9083e3\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx" Oct 31 13:22:32.566852 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:32.566622 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30e9e170-b40a-4f10-b43c-984e3d9083e3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx\" (UID: \"30e9e170-b40a-4f10-b43c-984e3d9083e3\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx" Oct 31 13:22:32.567165 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:32.567146 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/30e9e170-b40a-4f10-b43c-984e3d9083e3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx\" (UID: \"30e9e170-b40a-4f10-b43c-984e3d9083e3\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx" Oct 31 13:22:32.567376 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:32.567355 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30e9e170-b40a-4f10-b43c-984e3d9083e3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx\" (UID: \"30e9e170-b40a-4f10-b43c-984e3d9083e3\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx" Oct 31 13:22:32.569258 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:32.569235 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/30e9e170-b40a-4f10-b43c-984e3d9083e3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx\" (UID: \"30e9e170-b40a-4f10-b43c-984e3d9083e3\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx" Oct 31 13:22:32.579094 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:32.579070 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzpkh\" (UniqueName: \"kubernetes.io/projected/30e9e170-b40a-4f10-b43c-984e3d9083e3-kube-api-access-pzpkh\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx\" (UID: \"30e9e170-b40a-4f10-b43c-984e3d9083e3\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx" Oct 31 13:22:32.752361 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:32.752327 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx" Oct 31 13:22:33.044781 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:33.044748 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx"] Oct 31 13:22:33.045278 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:22:33.045252 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30e9e170_b40a_4f10_b43c_984e3d9083e3.slice/crio-7796c8bda920e14cb2d03b41dbdc3a6290ca84a9f4e981286f266601f7a8a336 WatchSource:0}: Error finding container 7796c8bda920e14cb2d03b41dbdc3a6290ca84a9f4e981286f266601f7a8a336: Status 404 returned error can't find the container with id 7796c8bda920e14cb2d03b41dbdc3a6290ca84a9f4e981286f266601f7a8a336 Oct 31 13:22:33.332365 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:33.332290 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx"] Oct 31 13:22:33.927282 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:33.927243 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx" event={"ID":"30e9e170-b40a-4f10-b43c-984e3d9083e3","Type":"ContainerStarted","Data":"6889cdacd33a85e00dea789b3155a47c8f33f99206d367b0cb22b80eb0a39c1a"} Oct 31 13:22:33.927481 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:33.927301 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx" event={"ID":"30e9e170-b40a-4f10-b43c-984e3d9083e3","Type":"ContainerStarted","Data":"7796c8bda920e14cb2d03b41dbdc3a6290ca84a9f4e981286f266601f7a8a336"} Oct 31 13:22:33.927481 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:33.927325 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx" Oct 31 13:22:33.927584 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:33.927482 5693 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx" secret="" err="secret \"payload-processing-external-model-e2e--4724aaf0-dockercfg-fj5fx\" not found" Oct 31 13:22:33.957062 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:33.957012 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx" podStartSLOduration=1.956997498 podStartE2EDuration="1.956997498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:22:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:22:33.953203209 +0000 UTC m=+3004.660789530" watchObservedRunningTime="2026-10-31 13:22:33.956997498 +0000 UTC m=+3004.664583793" Oct 31 13:22:33.996401 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:22:33.996370 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-9ef744/agc-cred-225017692890: secret "agc-cred-225017692890" not found Oct 31 13:22:33.996401 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:22:33.996399 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx: secret "agc-cred-225017692890" not found Oct 31 13:22:33.996690 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:22:33.996405 5693 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-plugins-e2e-extmod-a-9ef744: configmap "payload-processing-external-model-plugins-e2e-extmod-a-9ef744" not found Oct 31 13:22:33.996690 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:22:33.996465 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/30e9e170-b40a-4f10-b43c-984e3d9083e3-provider-credentials podName:30e9e170-b40a-4f10-b43c-984e3d9083e3 nodeName:}" failed. No retries permitted until 2026-10-31 13:22:34.496445329 +0000 UTC m=+3005.204031620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/30e9e170-b40a-4f10-b43c-984e3d9083e3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx" (UID: "30e9e170-b40a-4f10-b43c-984e3d9083e3") : secret "agc-cred-225017692890" not found Oct 31 13:22:33.996690 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:22:33.996491 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/30e9e170-b40a-4f10-b43c-984e3d9083e3-plugins-config-volume podName:30e9e170-b40a-4f10-b43c-984e3d9083e3 nodeName:}" failed. No retries permitted until 2026-10-31 13:22:34.496472034 +0000 UTC m=+3005.204058311 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/30e9e170-b40a-4f10-b43c-984e3d9083e3-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx" (UID: "30e9e170-b40a-4f10-b43c-984e3d9083e3") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-9ef744" not found Oct 31 13:22:34.502535 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:22:34.502503 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-9ef744/agc-cred-225017692890: secret "agc-cred-225017692890" not found Oct 31 13:22:34.502535 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:22:34.502531 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx: secret "agc-cred-225017692890" not found Oct 31 13:22:34.502535 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:22:34.502507 5693 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-plugins-e2e-extmod-a-9ef744: configmap "payload-processing-external-model-plugins-e2e-extmod-a-9ef744" not found Oct 31 13:22:34.503043 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:22:34.502592 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/30e9e170-b40a-4f10-b43c-984e3d9083e3-plugins-config-volume podName:30e9e170-b40a-4f10-b43c-984e3d9083e3 nodeName:}" failed. No retries permitted until 2026-10-31 13:22:35.502577193 +0000 UTC m=+3006.210163479 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/30e9e170-b40a-4f10-b43c-984e3d9083e3-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx" (UID: "30e9e170-b40a-4f10-b43c-984e3d9083e3") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-9ef744" not found Oct 31 13:22:34.503043 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:22:34.502606 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/30e9e170-b40a-4f10-b43c-984e3d9083e3-provider-credentials podName:30e9e170-b40a-4f10-b43c-984e3d9083e3 nodeName:}" failed. No retries permitted until 2026-10-31 13:22:35.502599839 +0000 UTC m=+3006.210186113 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/30e9e170-b40a-4f10-b43c-984e3d9083e3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx" (UID: "30e9e170-b40a-4f10-b43c-984e3d9083e3") : secret "agc-cred-225017692890" not found Oct 31 13:22:35.519472 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:22:35.519433 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-9ef744/agc-cred-225017692890: secret "agc-cred-225017692890" not found Oct 31 13:22:35.519472 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:22:35.519467 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx: secret "agc-cred-225017692890" not found Oct 31 13:22:35.519956 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:22:35.519525 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/30e9e170-b40a-4f10-b43c-984e3d9083e3-provider-credentials podName:30e9e170-b40a-4f10-b43c-984e3d9083e3 nodeName:}" failed. No retries permitted until 2026-10-31 13:22:37.519507234 +0000 UTC m=+3008.227093506 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/30e9e170-b40a-4f10-b43c-984e3d9083e3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx" (UID: "30e9e170-b40a-4f10-b43c-984e3d9083e3") : secret "agc-cred-225017692890" not found Oct 31 13:22:35.519956 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:22:35.519450 5693 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-plugins-e2e-extmod-a-9ef744: configmap "payload-processing-external-model-plugins-e2e-extmod-a-9ef744" not found Oct 31 13:22:35.519956 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:22:35.519575 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/30e9e170-b40a-4f10-b43c-984e3d9083e3-plugins-config-volume podName:30e9e170-b40a-4f10-b43c-984e3d9083e3 nodeName:}" failed. No retries permitted until 2026-10-31 13:22:37.519564518 +0000 UTC m=+3008.227150805 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/30e9e170-b40a-4f10-b43c-984e3d9083e3-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx" (UID: "30e9e170-b40a-4f10-b43c-984e3d9083e3") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-9ef744" not found Oct 31 13:22:37.591558 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:22:37.591522 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-9ef744/agc-cred-225017692890: secret "agc-cred-225017692890" not found Oct 31 13:22:37.591558 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:22:37.591549 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx: secret "agc-cred-225017692890" not found Oct 31 13:22:37.592053 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:22:37.591603 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/30e9e170-b40a-4f10-b43c-984e3d9083e3-provider-credentials podName:30e9e170-b40a-4f10-b43c-984e3d9083e3 nodeName:}" failed. No retries permitted until 2026-10-31 13:22:41.591584702 +0000 UTC m=+3012.299170995 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/30e9e170-b40a-4f10-b43c-984e3d9083e3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx" (UID: "30e9e170-b40a-4f10-b43c-984e3d9083e3") : secret "agc-cred-225017692890" not found Oct 31 13:22:37.592053 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:22:37.591534 5693 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-plugins-e2e-extmod-a-9ef744: configmap "payload-processing-external-model-plugins-e2e-extmod-a-9ef744" not found Oct 31 13:22:37.592053 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:22:37.591683 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/30e9e170-b40a-4f10-b43c-984e3d9083e3-plugins-config-volume podName:30e9e170-b40a-4f10-b43c-984e3d9083e3 nodeName:}" failed. No retries permitted until 2026-10-31 13:22:41.591670397 +0000 UTC m=+3012.299256670 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/30e9e170-b40a-4f10-b43c-984e3d9083e3-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx" (UID: "30e9e170-b40a-4f10-b43c-984e3d9083e3") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-9ef744" not found Oct 31 13:22:39.936713 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:39.936676 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx" podUID="30e9e170-b40a-4f10-b43c-984e3d9083e3" containerName="payload-processing" containerID="cri-o://6889cdacd33a85e00dea789b3155a47c8f33f99206d367b0cb22b80eb0a39c1a" gracePeriod=25 Oct 31 13:22:40.214507 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:40.214480 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx" Oct 31 13:22:40.337879 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:40.337846 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30e9e170-b40a-4f10-b43c-984e3d9083e3-kube-api-access-pzpkh\" (UniqueName: \"kubernetes.io/projected/30e9e170-b40a-4f10-b43c-984e3d9083e3-kube-api-access-pzpkh\") pod \"30e9e170-b40a-4f10-b43c-984e3d9083e3\" (UID: \"30e9e170-b40a-4f10-b43c-984e3d9083e3\") " Oct 31 13:22:40.338080 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:40.337961 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30e9e170-b40a-4f10-b43c-984e3d9083e3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/30e9e170-b40a-4f10-b43c-984e3d9083e3-provider-credentials\") pod \"30e9e170-b40a-4f10-b43c-984e3d9083e3\" (UID: \"30e9e170-b40a-4f10-b43c-984e3d9083e3\") " Oct 31 13:22:40.338145 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:40.338109 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30e9e170-b40a-4f10-b43c-984e3d9083e3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/30e9e170-b40a-4f10-b43c-984e3d9083e3-routing-overlay-volume\") pod \"30e9e170-b40a-4f10-b43c-984e3d9083e3\" (UID: \"30e9e170-b40a-4f10-b43c-984e3d9083e3\") " Oct 31 13:22:40.338256 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:40.338236 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30e9e170-b40a-4f10-b43c-984e3d9083e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30e9e170-b40a-4f10-b43c-984e3d9083e3-plugins-config-volume\") pod \"30e9e170-b40a-4f10-b43c-984e3d9083e3\" (UID: \"30e9e170-b40a-4f10-b43c-984e3d9083e3\") " Oct 31 13:22:40.338568 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:40.338538 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30e9e170-b40a-4f10-b43c-984e3d9083e3-routing-overlay-volume" pod "30e9e170-b40a-4f10-b43c-984e3d9083e3" (UID: "30e9e170-b40a-4f10-b43c-984e3d9083e3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:22:40.338679 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:40.338599 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30e9e170-b40a-4f10-b43c-984e3d9083e3-plugins-config-volume" pod "30e9e170-b40a-4f10-b43c-984e3d9083e3" (UID: "30e9e170-b40a-4f10-b43c-984e3d9083e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:22:40.339020 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:40.339001 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/30e9e170-b40a-4f10-b43c-984e3d9083e3-routing-overlay-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:22:40.339080 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:40.339026 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30e9e170-b40a-4f10-b43c-984e3d9083e3-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:22:40.340318 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:40.340293 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30e9e170-b40a-4f10-b43c-984e3d9083e3-provider-credentials" pod "30e9e170-b40a-4f10-b43c-984e3d9083e3" (UID: "30e9e170-b40a-4f10-b43c-984e3d9083e3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:22:40.340419 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:40.340344 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30e9e170-b40a-4f10-b43c-984e3d9083e3-kube-api-access-pzpkh" pod "30e9e170-b40a-4f10-b43c-984e3d9083e3" (UID: "30e9e170-b40a-4f10-b43c-984e3d9083e3"). InnerVolumeSpecName "kube-api-access-pzpkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:22:40.441059 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:40.440980 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/30e9e170-b40a-4f10-b43c-984e3d9083e3-provider-credentials\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:22:40.441059 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:40.441008 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzpkh\" (UniqueName: \"kubernetes.io/projected/30e9e170-b40a-4f10-b43c-984e3d9083e3-kube-api-access-pzpkh\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:22:41.030015 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:41.029980 5693 generic.go:356] "Generic (PLEG): container finished" podID="30e9e170-b40a-4f10-b43c-984e3d9083e3" containerID="6889cdacd33a85e00dea789b3155a47c8f33f99206d367b0cb22b80eb0a39c1a" exitCode=0 Oct 31 13:22:41.030467 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:41.030023 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx" Oct 31 13:22:41.030467 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:41.030021 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx" event={"ID":"30e9e170-b40a-4f10-b43c-984e3d9083e3","Type":"ContainerDied","Data":"6889cdacd33a85e00dea789b3155a47c8f33f99206d367b0cb22b80eb0a39c1a"} Oct 31 13:22:41.030467 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:41.030152 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx" event={"ID":"30e9e170-b40a-4f10-b43c-984e3d9083e3","Type":"ContainerDied","Data":"7796c8bda920e14cb2d03b41dbdc3a6290ca84a9f4e981286f266601f7a8a336"} Oct 31 13:22:41.030467 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:41.030182 5693 scope.go:122] "RemoveContainer" containerID="6889cdacd33a85e00dea789b3155a47c8f33f99206d367b0cb22b80eb0a39c1a" Oct 31 13:22:41.041602 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:41.041580 5693 scope.go:122] "RemoveContainer" containerID="6889cdacd33a85e00dea789b3155a47c8f33f99206d367b0cb22b80eb0a39c1a" Oct 31 13:22:41.041982 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:22:41.041960 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6889cdacd33a85e00dea789b3155a47c8f33f99206d367b0cb22b80eb0a39c1a\": container with ID starting with 6889cdacd33a85e00dea789b3155a47c8f33f99206d367b0cb22b80eb0a39c1a not found: ID does not exist" containerID="6889cdacd33a85e00dea789b3155a47c8f33f99206d367b0cb22b80eb0a39c1a" Oct 31 13:22:41.042053 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:41.041989 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6889cdacd33a85e00dea789b3155a47c8f33f99206d367b0cb22b80eb0a39c1a"} err="failed to get container status \"6889cdacd33a85e00dea789b3155a47c8f33f99206d367b0cb22b80eb0a39c1a\": rpc error: code = NotFound desc = could not find container \"6889cdacd33a85e00dea789b3155a47c8f33f99206d367b0cb22b80eb0a39c1a\": container with ID starting with 6889cdacd33a85e00dea789b3155a47c8f33f99206d367b0cb22b80eb0a39c1a not found: ID does not exist" Oct 31 13:22:41.055903 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:41.055877 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx"] Oct 31 13:22:41.061526 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:41.061504 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f77btpkx"] Oct 31 13:22:41.412328 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:41.412296 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30e9e170-b40a-4f10-b43c-984e3d9083e3" path="/var/lib/kubelet/pods/30e9e170-b40a-4f10-b43c-984e3d9083e3/volumes" Oct 31 13:22:48.689314 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:22:48.689243 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:22:52.169661 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:52.164761 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg"] Oct 31 13:22:52.169661 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:52.166269 5693 cpu_manager.go:414] "containerMap: removing container" podUID="30e9e170-b40a-4f10-b43c-984e3d9083e3" containerName="payload-processing" Oct 31 13:22:52.169661 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:52.166292 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30e9e170-b40a-4f10-b43c-984e3d9083e3" containerName="payload-processing" Oct 31 13:22:52.169661 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:52.166620 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="30e9e170-b40a-4f10-b43c-984e3d9083e3" containerName="payload-processing" Oct 31 13:22:52.244261 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:52.242548 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg" Oct 31 13:22:52.254309 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:52.251946 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-29e3a0\"/\"agc-cred-29aa17c26ce2\"" Oct 31 13:22:52.258818 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:52.256850 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-29e3a0\"/\"payload-processing-external-model-e2e--e205175e-dockercfg-4dwt6\"" Oct 31 13:22:52.258818 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:52.257323 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg"] Oct 31 13:22:52.258818 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:52.258602 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-29e3a0\"/\"routing-overlay\"" Oct 31 13:22:52.258818 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:52.258732 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-29e3a0\"/\"payload-processing-external-model-plugins-e2e-extmod-b-29e3a0\"" Oct 31 13:22:52.258818 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:52.258780 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-29e3a0\"/\"kube-root-ca.crt\"" Oct 31 13:22:52.259346 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:52.258838 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-29e3a0\"/\"openshift-service-ca.crt\"" Oct 31 13:22:52.305718 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:52.305684 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bbed289-bc91-4d34-b4fa-dffc1dc45338-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg\" (UID: \"1bbed289-bc91-4d34-b4fa-dffc1dc45338\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg" Oct 31 13:22:52.306070 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:52.306046 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1bbed289-bc91-4d34-b4fa-dffc1dc45338-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg\" (UID: \"1bbed289-bc91-4d34-b4fa-dffc1dc45338\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg" Oct 31 13:22:52.306422 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:52.306400 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq4w2\" (UniqueName: \"kubernetes.io/projected/1bbed289-bc91-4d34-b4fa-dffc1dc45338-kube-api-access-dq4w2\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg\" (UID: \"1bbed289-bc91-4d34-b4fa-dffc1dc45338\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg" Oct 31 13:22:52.307220 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:52.307194 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1bbed289-bc91-4d34-b4fa-dffc1dc45338-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg\" (UID: \"1bbed289-bc91-4d34-b4fa-dffc1dc45338\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg" Oct 31 13:22:52.426712 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:52.413733 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bbed289-bc91-4d34-b4fa-dffc1dc45338-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg\" (UID: \"1bbed289-bc91-4d34-b4fa-dffc1dc45338\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg" Oct 31 13:22:52.426712 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:52.413802 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1bbed289-bc91-4d34-b4fa-dffc1dc45338-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg\" (UID: \"1bbed289-bc91-4d34-b4fa-dffc1dc45338\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg" Oct 31 13:22:52.426712 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:52.413852 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq4w2\" (UniqueName: \"kubernetes.io/projected/1bbed289-bc91-4d34-b4fa-dffc1dc45338-kube-api-access-dq4w2\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg\" (UID: \"1bbed289-bc91-4d34-b4fa-dffc1dc45338\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg" Oct 31 13:22:52.426712 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:52.413972 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1bbed289-bc91-4d34-b4fa-dffc1dc45338-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg\" (UID: \"1bbed289-bc91-4d34-b4fa-dffc1dc45338\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg" Oct 31 13:22:52.426712 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:52.416335 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1bbed289-bc91-4d34-b4fa-dffc1dc45338-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg\" (UID: \"1bbed289-bc91-4d34-b4fa-dffc1dc45338\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg" Oct 31 13:22:52.426712 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:52.417070 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bbed289-bc91-4d34-b4fa-dffc1dc45338-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg\" (UID: \"1bbed289-bc91-4d34-b4fa-dffc1dc45338\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg" Oct 31 13:22:52.426712 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:52.418513 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1bbed289-bc91-4d34-b4fa-dffc1dc45338-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg\" (UID: \"1bbed289-bc91-4d34-b4fa-dffc1dc45338\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg" Oct 31 13:22:52.452522 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:52.452467 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq4w2\" (UniqueName: \"kubernetes.io/projected/1bbed289-bc91-4d34-b4fa-dffc1dc45338-kube-api-access-dq4w2\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg\" (UID: \"1bbed289-bc91-4d34-b4fa-dffc1dc45338\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg" Oct 31 13:22:52.572761 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:52.572497 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg" Oct 31 13:22:52.885109 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:22:52.884256 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bbed289_bc91_4d34_b4fa_dffc1dc45338.slice/crio-60eba20c514fbbc03d4215804c3e4b750f1c40412b9a665ca98784b58842932c WatchSource:0}: Error finding container 60eba20c514fbbc03d4215804c3e4b750f1c40412b9a665ca98784b58842932c: Status 404 returned error can't find the container with id 60eba20c514fbbc03d4215804c3e4b750f1c40412b9a665ca98784b58842932c Oct 31 13:22:52.885838 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:52.885809 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg"] Oct 31 13:22:53.286927 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:53.286888 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg" event={"ID":"1bbed289-bc91-4d34-b4fa-dffc1dc45338","Type":"ContainerStarted","Data":"aa402b2c23270a5449e5a0bff33f158e4b1bb24e6644c8546d8b72b7010a77e4"} Oct 31 13:22:53.287508 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:53.286954 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg" event={"ID":"1bbed289-bc91-4d34-b4fa-dffc1dc45338","Type":"ContainerStarted","Data":"60eba20c514fbbc03d4215804c3e4b750f1c40412b9a665ca98784b58842932c"} Oct 31 13:22:53.287508 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:53.287122 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg" Oct 31 13:22:53.309095 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:22:53.309038 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg" podStartSLOduration=1.30901847 podStartE2EDuration="1.30901847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:22:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:22:53.308849314 +0000 UTC m=+3024.016435611" watchObservedRunningTime="2026-10-31 13:22:53.30901847 +0000 UTC m=+3024.016604769" Oct 31 13:23:04.301935 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:04.301909 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg" Oct 31 13:23:04.551600 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:04.551562 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7"] Oct 31 13:23:04.556842 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:04.556791 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7" Oct 31 13:23:04.561866 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:04.561152 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9ef744\"/\"routing-overlay\"" Oct 31 13:23:04.561866 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:04.561502 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9ef744\"/\"agc-cred-d0bf873b9006\"" Oct 31 13:23:04.561866 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:04.561546 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9ef744\"/\"payload-processing-external-model-plugins-e2e-extmod-a-9ef744\"" Oct 31 13:23:04.561866 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:04.561728 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9ef744\"/\"payload-processing-external-model-e2e--4724aaf0-dockercfg-rtn5f\"" Oct 31 13:23:04.561866 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:04.561768 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9ef744\"/\"agc-cred-a12737161cbe\"" Oct 31 13:23:04.561866 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:04.561837 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9ef744\"/\"kube-root-ca.crt\"" Oct 31 13:23:04.562267 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:04.561975 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9ef744\"/\"openshift-service-ca.crt\"" Oct 31 13:23:04.566494 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:04.566466 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7"] Oct 31 13:23:04.663122 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:04.663080 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c115210-830a-4c8c-b8d3-b661941f535c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7\" (UID: \"9c115210-830a-4c8c-b8d3-b661941f535c\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7" Oct 31 13:23:04.663313 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:04.663274 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9c115210-830a-4c8c-b8d3-b661941f535c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7\" (UID: \"9c115210-830a-4c8c-b8d3-b661941f535c\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7" Oct 31 13:23:04.663471 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:04.663440 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9c115210-830a-4c8c-b8d3-b661941f535c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7\" (UID: \"9c115210-830a-4c8c-b8d3-b661941f535c\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7" Oct 31 13:23:04.663530 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:04.663515 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwh2w\" (UniqueName: \"kubernetes.io/projected/9c115210-830a-4c8c-b8d3-b661941f535c-kube-api-access-gwh2w\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7\" (UID: \"9c115210-830a-4c8c-b8d3-b661941f535c\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7" Oct 31 13:23:04.765360 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:04.765328 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9c115210-830a-4c8c-b8d3-b661941f535c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7\" (UID: \"9c115210-830a-4c8c-b8d3-b661941f535c\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7" Oct 31 13:23:04.765550 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:04.765479 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9c115210-830a-4c8c-b8d3-b661941f535c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7\" (UID: \"9c115210-830a-4c8c-b8d3-b661941f535c\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7" Oct 31 13:23:04.765550 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:04.765540 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwh2w\" (UniqueName: \"kubernetes.io/projected/9c115210-830a-4c8c-b8d3-b661941f535c-kube-api-access-gwh2w\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7\" (UID: \"9c115210-830a-4c8c-b8d3-b661941f535c\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7" Oct 31 13:23:04.765767 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:04.765749 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c115210-830a-4c8c-b8d3-b661941f535c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7\" (UID: \"9c115210-830a-4c8c-b8d3-b661941f535c\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7" Oct 31 13:23:04.766358 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:04.766327 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9c115210-830a-4c8c-b8d3-b661941f535c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7\" (UID: \"9c115210-830a-4c8c-b8d3-b661941f535c\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7" Oct 31 13:23:04.766554 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:04.766537 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c115210-830a-4c8c-b8d3-b661941f535c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7\" (UID: \"9c115210-830a-4c8c-b8d3-b661941f535c\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7" Oct 31 13:23:04.768265 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:04.768245 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9c115210-830a-4c8c-b8d3-b661941f535c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7\" (UID: \"9c115210-830a-4c8c-b8d3-b661941f535c\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7" Oct 31 13:23:04.778838 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:04.778810 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwh2w\" (UniqueName: \"kubernetes.io/projected/9c115210-830a-4c8c-b8d3-b661941f535c-kube-api-access-gwh2w\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7\" (UID: \"9c115210-830a-4c8c-b8d3-b661941f535c\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7" Oct 31 13:23:04.876712 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:04.876608 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7" Oct 31 13:23:04.978499 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:04.976137 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm"] Oct 31 13:23:05.000966 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:05.000914 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm" Oct 31 13:23:05.001340 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:05.001312 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm"] Oct 31 13:23:05.016381 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:05.016354 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-29e3a0\"/\"agc-cred-3c34f7db80f8\"" Oct 31 13:23:05.098219 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:05.098188 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/76fa58b1-d342-45ee-9b00-5aca023ac806-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm\" (UID: \"76fa58b1-d342-45ee-9b00-5aca023ac806\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm" Oct 31 13:23:05.098467 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:05.098390 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/76fa58b1-d342-45ee-9b00-5aca023ac806-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm\" (UID: \"76fa58b1-d342-45ee-9b00-5aca023ac806\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm" Oct 31 13:23:05.098467 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:05.098425 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxs27\" (UniqueName: \"kubernetes.io/projected/76fa58b1-d342-45ee-9b00-5aca023ac806-kube-api-access-rxs27\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm\" (UID: \"76fa58b1-d342-45ee-9b00-5aca023ac806\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm" Oct 31 13:23:05.098580 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:05.098502 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76fa58b1-d342-45ee-9b00-5aca023ac806-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm\" (UID: \"76fa58b1-d342-45ee-9b00-5aca023ac806\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm" Oct 31 13:23:05.200475 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:05.200434 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/76fa58b1-d342-45ee-9b00-5aca023ac806-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm\" (UID: \"76fa58b1-d342-45ee-9b00-5aca023ac806\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm" Oct 31 13:23:05.200743 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:05.200629 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/76fa58b1-d342-45ee-9b00-5aca023ac806-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm\" (UID: \"76fa58b1-d342-45ee-9b00-5aca023ac806\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm" Oct 31 13:23:05.200743 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:05.200686 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxs27\" (UniqueName: \"kubernetes.io/projected/76fa58b1-d342-45ee-9b00-5aca023ac806-kube-api-access-rxs27\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm\" (UID: \"76fa58b1-d342-45ee-9b00-5aca023ac806\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm" Oct 31 13:23:05.200895 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:05.200760 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76fa58b1-d342-45ee-9b00-5aca023ac806-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm\" (UID: \"76fa58b1-d342-45ee-9b00-5aca023ac806\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm" Oct 31 13:23:05.201526 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:05.201467 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/76fa58b1-d342-45ee-9b00-5aca023ac806-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm\" (UID: \"76fa58b1-d342-45ee-9b00-5aca023ac806\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm" Oct 31 13:23:05.201782 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:05.201600 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76fa58b1-d342-45ee-9b00-5aca023ac806-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm\" (UID: \"76fa58b1-d342-45ee-9b00-5aca023ac806\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm" Oct 31 13:23:05.203353 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:05.203330 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/76fa58b1-d342-45ee-9b00-5aca023ac806-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm\" (UID: \"76fa58b1-d342-45ee-9b00-5aca023ac806\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm" Oct 31 13:23:05.215204 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:05.215181 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxs27\" (UniqueName: \"kubernetes.io/projected/76fa58b1-d342-45ee-9b00-5aca023ac806-kube-api-access-rxs27\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm\" (UID: \"76fa58b1-d342-45ee-9b00-5aca023ac806\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm" Oct 31 13:23:05.247992 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:05.247964 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7"] Oct 31 13:23:05.249849 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:23:05.249824 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c115210_830a_4c8c_b8d3_b661941f535c.slice/crio-b705c0454293b1a5a715e1c519850093c132c77a99ab6108d12f58dac7c98dcc WatchSource:0}: Error finding container b705c0454293b1a5a715e1c519850093c132c77a99ab6108d12f58dac7c98dcc: Status 404 returned error can't find the container with id b705c0454293b1a5a715e1c519850093c132c77a99ab6108d12f58dac7c98dcc Oct 31 13:23:05.325903 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:05.325874 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm" Oct 31 13:23:05.397695 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:05.397325 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7" event={"ID":"9c115210-830a-4c8c-b8d3-b661941f535c","Type":"ContainerStarted","Data":"362cb8a23c8d9094beded9e873c2af5c5472f8c92af8c6a1f578c405d2032d6c"} Oct 31 13:23:05.397695 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:05.397379 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7" event={"ID":"9c115210-830a-4c8c-b8d3-b661941f535c","Type":"ContainerStarted","Data":"b705c0454293b1a5a715e1c519850093c132c77a99ab6108d12f58dac7c98dcc"} Oct 31 13:23:05.398504 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:05.398481 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7" Oct 31 13:23:05.424505 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:05.424445 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7" podStartSLOduration=1.424424787 podStartE2EDuration="1.424424787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:23:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:23:05.420898134 +0000 UTC m=+3036.128484431" watchObservedRunningTime="2026-10-31 13:23:05.424424787 +0000 UTC m=+3036.132011084" Oct 31 13:23:05.618305 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:05.618271 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm"] Oct 31 13:23:05.618469 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:23:05.618277 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76fa58b1_d342_45ee_9b00_5aca023ac806.slice/crio-018b24304e7bea02f4dfc266488b29d1c65691baaf8a2d617bc64742405b358b WatchSource:0}: Error finding container 018b24304e7bea02f4dfc266488b29d1c65691baaf8a2d617bc64742405b358b: Status 404 returned error can't find the container with id 018b24304e7bea02f4dfc266488b29d1c65691baaf8a2d617bc64742405b358b Oct 31 13:23:06.442709 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:06.442625 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm" event={"ID":"76fa58b1-d342-45ee-9b00-5aca023ac806","Type":"ContainerStarted","Data":"68efc67710ef2187efc90c12a93cf219e2020b9f85e28dbc35bf80ab5b452869"} Oct 31 13:23:06.443334 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:06.443301 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm" event={"ID":"76fa58b1-d342-45ee-9b00-5aca023ac806","Type":"ContainerStarted","Data":"018b24304e7bea02f4dfc266488b29d1c65691baaf8a2d617bc64742405b358b"} Oct 31 13:23:06.443557 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:06.443356 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm" Oct 31 13:23:06.494132 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:06.494083 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm" podStartSLOduration=2.494069656 podStartE2EDuration="2.494069656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:23:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:23:06.491341571 +0000 UTC m=+3037.198927878" watchObservedRunningTime="2026-10-31 13:23:06.494069656 +0000 UTC m=+3037.201655951" Oct 31 13:23:17.452943 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:17.452917 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm" Oct 31 13:23:17.453342 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:17.452980 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7" Oct 31 13:23:17.528900 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:17.528865 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg"] Oct 31 13:23:18.688747 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:18.688713 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:23:22.536662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:22.532702 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg" podUID="1bbed289-bc91-4d34-b4fa-dffc1dc45338" containerName="payload-processing" containerID="cri-o://aa402b2c23270a5449e5a0bff33f158e4b1bb24e6644c8546d8b72b7010a77e4" gracePeriod=25 Oct 31 13:23:23.174707 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:23.170391 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg" Oct 31 13:23:23.282662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:23.282610 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bbed289-bc91-4d34-b4fa-dffc1dc45338-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1bbed289-bc91-4d34-b4fa-dffc1dc45338-routing-overlay-volume\") pod \"1bbed289-bc91-4d34-b4fa-dffc1dc45338\" (UID: \"1bbed289-bc91-4d34-b4fa-dffc1dc45338\") " Oct 31 13:23:23.282867 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:23.282707 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bbed289-bc91-4d34-b4fa-dffc1dc45338-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1bbed289-bc91-4d34-b4fa-dffc1dc45338-provider-credentials\") pod \"1bbed289-bc91-4d34-b4fa-dffc1dc45338\" (UID: \"1bbed289-bc91-4d34-b4fa-dffc1dc45338\") " Oct 31 13:23:23.282927 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:23.282911 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bbed289-bc91-4d34-b4fa-dffc1dc45338-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bbed289-bc91-4d34-b4fa-dffc1dc45338-plugins-config-volume\") pod \"1bbed289-bc91-4d34-b4fa-dffc1dc45338\" (UID: \"1bbed289-bc91-4d34-b4fa-dffc1dc45338\") " Oct 31 13:23:23.282977 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:23.282957 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bbed289-bc91-4d34-b4fa-dffc1dc45338-kube-api-access-dq4w2\" (UniqueName: \"kubernetes.io/projected/1bbed289-bc91-4d34-b4fa-dffc1dc45338-kube-api-access-dq4w2\") pod \"1bbed289-bc91-4d34-b4fa-dffc1dc45338\" (UID: \"1bbed289-bc91-4d34-b4fa-dffc1dc45338\") " Oct 31 13:23:23.287623 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:23.284763 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bbed289-bc91-4d34-b4fa-dffc1dc45338-routing-overlay-volume" pod "1bbed289-bc91-4d34-b4fa-dffc1dc45338" (UID: "1bbed289-bc91-4d34-b4fa-dffc1dc45338"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:23:23.287623 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:23.285162 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bbed289-bc91-4d34-b4fa-dffc1dc45338-plugins-config-volume" pod "1bbed289-bc91-4d34-b4fa-dffc1dc45338" (UID: "1bbed289-bc91-4d34-b4fa-dffc1dc45338"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:23:23.294935 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:23.294792 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bbed289-bc91-4d34-b4fa-dffc1dc45338-provider-credentials" pod "1bbed289-bc91-4d34-b4fa-dffc1dc45338" (UID: "1bbed289-bc91-4d34-b4fa-dffc1dc45338"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:23:23.295662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:23.295468 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bbed289-bc91-4d34-b4fa-dffc1dc45338-kube-api-access-dq4w2" pod "1bbed289-bc91-4d34-b4fa-dffc1dc45338" (UID: "1bbed289-bc91-4d34-b4fa-dffc1dc45338"). InnerVolumeSpecName "kube-api-access-dq4w2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:23:23.390722 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:23.385770 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bbed289-bc91-4d34-b4fa-dffc1dc45338-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:23.390722 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:23.385806 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq4w2\" (UniqueName: \"kubernetes.io/projected/1bbed289-bc91-4d34-b4fa-dffc1dc45338-kube-api-access-dq4w2\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:23.390722 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:23.385849 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1bbed289-bc91-4d34-b4fa-dffc1dc45338-routing-overlay-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:23.390722 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:23.385864 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1bbed289-bc91-4d34-b4fa-dffc1dc45338-provider-credentials\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:23.600788 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:23.600749 5693 generic.go:356] "Generic (PLEG): container finished" podID="1bbed289-bc91-4d34-b4fa-dffc1dc45338" containerID="aa402b2c23270a5449e5a0bff33f158e4b1bb24e6644c8546d8b72b7010a77e4" exitCode=0 Oct 31 13:23:23.601325 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:23.601008 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg" event={"ID":"1bbed289-bc91-4d34-b4fa-dffc1dc45338","Type":"ContainerDied","Data":"aa402b2c23270a5449e5a0bff33f158e4b1bb24e6644c8546d8b72b7010a77e4"} Oct 31 13:23:23.601325 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:23.601064 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg" event={"ID":"1bbed289-bc91-4d34-b4fa-dffc1dc45338","Type":"ContainerDied","Data":"60eba20c514fbbc03d4215804c3e4b750f1c40412b9a665ca98784b58842932c"} Oct 31 13:23:23.601325 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:23.601094 5693 scope.go:122] "RemoveContainer" containerID="aa402b2c23270a5449e5a0bff33f158e4b1bb24e6644c8546d8b72b7010a77e4" Oct 31 13:23:23.601325 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:23.601267 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg" Oct 31 13:23:23.619470 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:23.619414 5693 scope.go:122] "RemoveContainer" containerID="aa402b2c23270a5449e5a0bff33f158e4b1bb24e6644c8546d8b72b7010a77e4" Oct 31 13:23:23.620196 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:23.620055 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa402b2c23270a5449e5a0bff33f158e4b1bb24e6644c8546d8b72b7010a77e4\": container with ID starting with aa402b2c23270a5449e5a0bff33f158e4b1bb24e6644c8546d8b72b7010a77e4 not found: ID does not exist" containerID="aa402b2c23270a5449e5a0bff33f158e4b1bb24e6644c8546d8b72b7010a77e4" Oct 31 13:23:23.620196 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:23.620093 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa402b2c23270a5449e5a0bff33f158e4b1bb24e6644c8546d8b72b7010a77e4"} err="failed to get container status \"aa402b2c23270a5449e5a0bff33f158e4b1bb24e6644c8546d8b72b7010a77e4\": rpc error: code = NotFound desc = could not find container \"aa402b2c23270a5449e5a0bff33f158e4b1bb24e6644c8546d8b72b7010a77e4\": container with ID starting with aa402b2c23270a5449e5a0bff33f158e4b1bb24e6644c8546d8b72b7010a77e4 not found: ID does not exist" Oct 31 13:23:23.632025 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:23.631989 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg"] Oct 31 13:23:23.637740 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:23.637710 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97qc5xg"] Oct 31 13:23:25.493663 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:25.492815 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bbed289-bc91-4d34-b4fa-dffc1dc45338" path="/var/lib/kubelet/pods/1bbed289-bc91-4d34-b4fa-dffc1dc45338/volumes" Oct 31 13:23:39.292905 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:39.292697 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7b9f9576bb-vch9g"] Oct 31 13:23:39.343858 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:39.294933 5693 cpu_manager.go:414] "containerMap: removing container" podUID="1bbed289-bc91-4d34-b4fa-dffc1dc45338" containerName="payload-processing" Oct 31 13:23:39.344123 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:39.344093 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bbed289-bc91-4d34-b4fa-dffc1dc45338" containerName="payload-processing" Oct 31 13:23:39.344619 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:39.344595 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bbed289-bc91-4d34-b4fa-dffc1dc45338" containerName="payload-processing" Oct 31 13:23:39.392570 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:39.392540 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b9f9576bb-vch9g" Oct 31 13:23:39.486627 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:39.486594 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b9f9576bb-vch9g"] Oct 31 13:23:39.499947 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:39.499884 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eda7993-42b2-4c04-85d9-39dd2c0f8caa-plugins-config-volume\") pod \"payload-processing-external-model-7b9f9576bb-vch9g\" (UID: \"0eda7993-42b2-4c04-85d9-39dd2c0f8caa\") " pod="models-as-a-service/payload-processing-external-model-7b9f9576bb-vch9g" Oct 31 13:23:39.500116 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:39.500103 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0eda7993-42b2-4c04-85d9-39dd2c0f8caa-provider-credentials\") pod \"payload-processing-external-model-7b9f9576bb-vch9g\" (UID: \"0eda7993-42b2-4c04-85d9-39dd2c0f8caa\") " pod="models-as-a-service/payload-processing-external-model-7b9f9576bb-vch9g" Oct 31 13:23:39.500284 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:39.500263 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cghs8\" (UniqueName: \"kubernetes.io/projected/0eda7993-42b2-4c04-85d9-39dd2c0f8caa-kube-api-access-cghs8\") pod \"payload-processing-external-model-7b9f9576bb-vch9g\" (UID: \"0eda7993-42b2-4c04-85d9-39dd2c0f8caa\") " pod="models-as-a-service/payload-processing-external-model-7b9f9576bb-vch9g" Oct 31 13:23:39.501165 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:39.501122 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0eda7993-42b2-4c04-85d9-39dd2c0f8caa-routing-overlay-volume\") pod \"payload-processing-external-model-7b9f9576bb-vch9g\" (UID: \"0eda7993-42b2-4c04-85d9-39dd2c0f8caa\") " pod="models-as-a-service/payload-processing-external-model-7b9f9576bb-vch9g" Oct 31 13:23:39.604715 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:39.603737 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0eda7993-42b2-4c04-85d9-39dd2c0f8caa-provider-credentials\") pod \"payload-processing-external-model-7b9f9576bb-vch9g\" (UID: \"0eda7993-42b2-4c04-85d9-39dd2c0f8caa\") " pod="models-as-a-service/payload-processing-external-model-7b9f9576bb-vch9g" Oct 31 13:23:39.604715 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:39.603890 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cghs8\" (UniqueName: \"kubernetes.io/projected/0eda7993-42b2-4c04-85d9-39dd2c0f8caa-kube-api-access-cghs8\") pod \"payload-processing-external-model-7b9f9576bb-vch9g\" (UID: \"0eda7993-42b2-4c04-85d9-39dd2c0f8caa\") " pod="models-as-a-service/payload-processing-external-model-7b9f9576bb-vch9g" Oct 31 13:23:39.604715 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:39.603931 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0eda7993-42b2-4c04-85d9-39dd2c0f8caa-routing-overlay-volume\") pod \"payload-processing-external-model-7b9f9576bb-vch9g\" (UID: \"0eda7993-42b2-4c04-85d9-39dd2c0f8caa\") " pod="models-as-a-service/payload-processing-external-model-7b9f9576bb-vch9g" Oct 31 13:23:39.604715 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:39.604141 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eda7993-42b2-4c04-85d9-39dd2c0f8caa-plugins-config-volume\") pod \"payload-processing-external-model-7b9f9576bb-vch9g\" (UID: \"0eda7993-42b2-4c04-85d9-39dd2c0f8caa\") " pod="models-as-a-service/payload-processing-external-model-7b9f9576bb-vch9g" Oct 31 13:23:39.609662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:39.605271 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eda7993-42b2-4c04-85d9-39dd2c0f8caa-plugins-config-volume\") pod \"payload-processing-external-model-7b9f9576bb-vch9g\" (UID: \"0eda7993-42b2-4c04-85d9-39dd2c0f8caa\") " pod="models-as-a-service/payload-processing-external-model-7b9f9576bb-vch9g" Oct 31 13:23:39.609662 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:39.605395 5693 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6ac4bb21ecb0: secret "agc-cred-6ac4bb21ecb0" not found Oct 31 13:23:39.609662 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:39.605412 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7b9f9576bb-vch9g: secret "agc-cred-6ac4bb21ecb0" not found Oct 31 13:23:39.609662 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:39.605469 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0eda7993-42b2-4c04-85d9-39dd2c0f8caa-provider-credentials podName:0eda7993-42b2-4c04-85d9-39dd2c0f8caa nodeName:}" failed. No retries permitted until 2026-10-31 13:23:40.105450587 +0000 UTC m=+3070.813036867 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0eda7993-42b2-4c04-85d9-39dd2c0f8caa-provider-credentials") pod "payload-processing-external-model-7b9f9576bb-vch9g" (UID: "0eda7993-42b2-4c04-85d9-39dd2c0f8caa") : secret "agc-cred-6ac4bb21ecb0" not found Oct 31 13:23:39.609662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:39.606493 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0eda7993-42b2-4c04-85d9-39dd2c0f8caa-routing-overlay-volume\") pod \"payload-processing-external-model-7b9f9576bb-vch9g\" (UID: \"0eda7993-42b2-4c04-85d9-39dd2c0f8caa\") " pod="models-as-a-service/payload-processing-external-model-7b9f9576bb-vch9g" Oct 31 13:23:39.633621 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:39.633587 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cghs8\" (UniqueName: \"kubernetes.io/projected/0eda7993-42b2-4c04-85d9-39dd2c0f8caa-kube-api-access-cghs8\") pod \"payload-processing-external-model-7b9f9576bb-vch9g\" (UID: \"0eda7993-42b2-4c04-85d9-39dd2c0f8caa\") " pod="models-as-a-service/payload-processing-external-model-7b9f9576bb-vch9g" Oct 31 13:23:39.660668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:39.658598 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-76758ccf48-lxlmg"] Oct 31 13:23:39.675669 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:39.673879 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b9f9576bb-vch9g"] Oct 31 13:23:39.675980 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:39.675948 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="models-as-a-service/payload-processing-external-model-7b9f9576bb-vch9g" podUID="0eda7993-42b2-4c04-85d9-39dd2c0f8caa" Oct 31 13:23:39.892017 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:39.891938 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b9f9576bb-vch9g" Oct 31 13:23:39.902010 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:39.901984 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b9f9576bb-vch9g" Oct 31 13:23:40.029672 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:40.021050 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eda7993-42b2-4c04-85d9-39dd2c0f8caa-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0eda7993-42b2-4c04-85d9-39dd2c0f8caa-routing-overlay-volume\") pod \"0eda7993-42b2-4c04-85d9-39dd2c0f8caa\" (UID: \"0eda7993-42b2-4c04-85d9-39dd2c0f8caa\") " Oct 31 13:23:40.029672 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:40.021119 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eda7993-42b2-4c04-85d9-39dd2c0f8caa-kube-api-access-cghs8\" (UniqueName: \"kubernetes.io/projected/0eda7993-42b2-4c04-85d9-39dd2c0f8caa-kube-api-access-cghs8\") pod \"0eda7993-42b2-4c04-85d9-39dd2c0f8caa\" (UID: \"0eda7993-42b2-4c04-85d9-39dd2c0f8caa\") " Oct 31 13:23:40.029672 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:40.021158 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eda7993-42b2-4c04-85d9-39dd2c0f8caa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eda7993-42b2-4c04-85d9-39dd2c0f8caa-plugins-config-volume\") pod \"0eda7993-42b2-4c04-85d9-39dd2c0f8caa\" (UID: \"0eda7993-42b2-4c04-85d9-39dd2c0f8caa\") " Oct 31 13:23:40.029672 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:40.025014 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eda7993-42b2-4c04-85d9-39dd2c0f8caa-routing-overlay-volume" pod "0eda7993-42b2-4c04-85d9-39dd2c0f8caa" (UID: "0eda7993-42b2-4c04-85d9-39dd2c0f8caa"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:23:40.029672 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:40.025533 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eda7993-42b2-4c04-85d9-39dd2c0f8caa-plugins-config-volume" pod "0eda7993-42b2-4c04-85d9-39dd2c0f8caa" (UID: "0eda7993-42b2-4c04-85d9-39dd2c0f8caa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:23:40.030282 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:40.030256 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eda7993-42b2-4c04-85d9-39dd2c0f8caa-kube-api-access-cghs8" pod "0eda7993-42b2-4c04-85d9-39dd2c0f8caa" (UID: "0eda7993-42b2-4c04-85d9-39dd2c0f8caa"). InnerVolumeSpecName "kube-api-access-cghs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:23:40.131197 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:40.131145 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0eda7993-42b2-4c04-85d9-39dd2c0f8caa-provider-credentials\") pod \"payload-processing-external-model-7b9f9576bb-vch9g\" (UID: \"0eda7993-42b2-4c04-85d9-39dd2c0f8caa\") " pod="models-as-a-service/payload-processing-external-model-7b9f9576bb-vch9g" Oct 31 13:23:40.131457 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:40.131424 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0eda7993-42b2-4c04-85d9-39dd2c0f8caa-routing-overlay-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:40.131457 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:40.131447 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cghs8\" (UniqueName: \"kubernetes.io/projected/0eda7993-42b2-4c04-85d9-39dd2c0f8caa-kube-api-access-cghs8\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:40.131623 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:40.131464 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eda7993-42b2-4c04-85d9-39dd2c0f8caa-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:40.131623 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:40.131578 5693 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6ac4bb21ecb0: secret "agc-cred-6ac4bb21ecb0" not found Oct 31 13:23:40.131623 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:40.131593 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7b9f9576bb-vch9g: secret "agc-cred-6ac4bb21ecb0" not found Oct 31 13:23:40.131860 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:40.131685 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0eda7993-42b2-4c04-85d9-39dd2c0f8caa-provider-credentials podName:0eda7993-42b2-4c04-85d9-39dd2c0f8caa nodeName:}" failed. No retries permitted until 2026-10-31 13:23:41.131663473 +0000 UTC m=+3071.839249762 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0eda7993-42b2-4c04-85d9-39dd2c0f8caa-provider-credentials") pod "payload-processing-external-model-7b9f9576bb-vch9g" (UID: "0eda7993-42b2-4c04-85d9-39dd2c0f8caa") : secret "agc-cred-6ac4bb21ecb0" not found Oct 31 13:23:40.899297 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:40.899265 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b9f9576bb-vch9g" Oct 31 13:23:40.939958 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:40.938974 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b9f9576bb-vch9g"] Oct 31 13:23:40.945004 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:40.944975 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b9f9576bb-vch9g"] Oct 31 13:23:41.101725 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:41.089290 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0eda7993-42b2-4c04-85d9-39dd2c0f8caa-provider-credentials\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:41.411587 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:41.411558 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eda7993-42b2-4c04-85d9-39dd2c0f8caa" path="/var/lib/kubelet/pods/0eda7993-42b2-4c04-85d9-39dd2c0f8caa/volumes" Oct 31 13:23:41.679194 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:41.679118 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h"] Oct 31 13:23:41.685736 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:41.685711 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" Oct 31 13:23:41.695421 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:41.695395 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4023ad-openshift-default-dockercfg-n4px5\"" Oct 31 13:23:41.697274 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:41.697248 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h"] Oct 31 13:23:41.731719 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:41.731683 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c08b4ba-3066-4b09-ba62-de15be6ccea2-workload-certs\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h\" (UID: \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" Oct 31 13:23:41.731932 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:41.731736 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c08b4ba-3066-4b09-ba62-de15be6ccea2-istio-envoy\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h\" (UID: \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" Oct 31 13:23:41.732020 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:41.731996 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c08b4ba-3066-4b09-ba62-de15be6ccea2-workload-socket\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h\" (UID: \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" Oct 31 13:23:41.732092 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:41.732058 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c08b4ba-3066-4b09-ba62-de15be6ccea2-istio-podinfo\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h\" (UID: \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" Oct 31 13:23:41.732152 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:41.732099 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c08b4ba-3066-4b09-ba62-de15be6ccea2-istiod-ca-cert\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h\" (UID: \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" Oct 31 13:23:41.732152 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:41.732124 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjjp6\" (UniqueName: \"kubernetes.io/projected/8c08b4ba-3066-4b09-ba62-de15be6ccea2-kube-api-access-zjjp6\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h\" (UID: \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" Oct 31 13:23:41.732259 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:41.732170 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c08b4ba-3066-4b09-ba62-de15be6ccea2-istio-data\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h\" (UID: \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" Oct 31 13:23:41.732367 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:41.732349 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c08b4ba-3066-4b09-ba62-de15be6ccea2-credential-socket\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h\" (UID: \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" Oct 31 13:23:41.732526 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:41.732503 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c08b4ba-3066-4b09-ba62-de15be6ccea2-istio-token\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h\" (UID: \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" Oct 31 13:23:41.838666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:41.838136 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c08b4ba-3066-4b09-ba62-de15be6ccea2-istio-token\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h\" (UID: \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" Oct 31 13:23:41.838666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:41.838353 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c08b4ba-3066-4b09-ba62-de15be6ccea2-workload-certs\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h\" (UID: \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" Oct 31 13:23:41.838666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:41.838393 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c08b4ba-3066-4b09-ba62-de15be6ccea2-istio-envoy\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h\" (UID: \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" Oct 31 13:23:41.838666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:41.838597 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c08b4ba-3066-4b09-ba62-de15be6ccea2-workload-socket\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h\" (UID: \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" Oct 31 13:23:41.839070 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:41.838749 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c08b4ba-3066-4b09-ba62-de15be6ccea2-istio-podinfo\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h\" (UID: \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" Oct 31 13:23:41.839070 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:41.838792 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c08b4ba-3066-4b09-ba62-de15be6ccea2-istiod-ca-cert\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h\" (UID: \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" Oct 31 13:23:41.839070 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:41.838831 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjjp6\" (UniqueName: \"kubernetes.io/projected/8c08b4ba-3066-4b09-ba62-de15be6ccea2-kube-api-access-zjjp6\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h\" (UID: \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" Oct 31 13:23:41.839070 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:41.838879 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c08b4ba-3066-4b09-ba62-de15be6ccea2-istio-data\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h\" (UID: \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" Oct 31 13:23:41.839296 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:41.839071 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c08b4ba-3066-4b09-ba62-de15be6ccea2-credential-socket\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h\" (UID: \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" Oct 31 13:23:41.868665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:41.839553 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c08b4ba-3066-4b09-ba62-de15be6ccea2-credential-socket\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h\" (UID: \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" Oct 31 13:23:41.868665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:41.840161 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c08b4ba-3066-4b09-ba62-de15be6ccea2-workload-certs\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h\" (UID: \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" Oct 31 13:23:41.868665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:41.863332 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c08b4ba-3066-4b09-ba62-de15be6ccea2-istio-data\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h\" (UID: \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" Oct 31 13:23:41.868665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:41.863444 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c08b4ba-3066-4b09-ba62-de15be6ccea2-istiod-ca-cert\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h\" (UID: \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" Oct 31 13:23:41.868665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:41.863549 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c08b4ba-3066-4b09-ba62-de15be6ccea2-istio-envoy\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h\" (UID: \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" Oct 31 13:23:41.868665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:41.863752 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c08b4ba-3066-4b09-ba62-de15be6ccea2-workload-socket\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h\" (UID: \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" Oct 31 13:23:41.881723 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:41.879345 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c08b4ba-3066-4b09-ba62-de15be6ccea2-istio-podinfo\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h\" (UID: \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" Oct 31 13:23:41.885785 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:41.884953 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c08b4ba-3066-4b09-ba62-de15be6ccea2-istio-token\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h\" (UID: \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" Oct 31 13:23:41.897374 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:41.897340 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjjp6\" (UniqueName: \"kubernetes.io/projected/8c08b4ba-3066-4b09-ba62-de15be6ccea2-kube-api-access-zjjp6\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h\" (UID: \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" Oct 31 13:23:42.006427 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:42.006346 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" Oct 31 13:23:42.359872 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:42.359842 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h"] Oct 31 13:23:42.360233 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:23:42.360206 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c08b4ba_3066_4b09_ba62_de15be6ccea2.slice/crio-7e002553c568441811742f917e89619999698b55d4e91201473b592e0a8623b1 WatchSource:0}: Error finding container 7e002553c568441811742f917e89619999698b55d4e91201473b592e0a8623b1: Status 404 returned error can't find the container with id 7e002553c568441811742f917e89619999698b55d4e91201473b592e0a8623b1 Oct 31 13:23:42.363039 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:42.363004 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:23:42.363184 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:42.363080 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:23:42.363184 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:42.363115 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:23:43.009546 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:43.009506 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" event={"ID":"8c08b4ba-3066-4b09-ba62-de15be6ccea2","Type":"ContainerStarted","Data":"37714a9885d5013797267376d3e7f865f72904ea538f01a35e946d9fbb52bb90"} Oct 31 13:23:43.010089 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:43.009591 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" event={"ID":"8c08b4ba-3066-4b09-ba62-de15be6ccea2","Type":"ContainerStarted","Data":"7e002553c568441811742f917e89619999698b55d4e91201473b592e0a8623b1"} Oct 31 13:23:43.046088 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:43.046022 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" podStartSLOduration=2.046003481 podStartE2EDuration="2.046003481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:23:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:23:43.043762813 +0000 UTC m=+3073.751349107" watchObservedRunningTime="2026-10-31 13:23:43.046003481 +0000 UTC m=+3073.753589777" Oct 31 13:23:43.496939 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:43.496906 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9"] Oct 31 13:23:43.506549 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:43.506510 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9"] Oct 31 13:23:43.506794 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:43.506691 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9" Oct 31 13:23:43.514533 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:43.514496 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0c1fbb\"/\"agc-cred-ac74976b3ec1\"" Oct 31 13:23:43.515205 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:43.515185 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0c1fbb\"/\"kube-root-ca.crt\"" Oct 31 13:23:43.515330 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:43.515260 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0c1fbb\"/\"payload-processing-external-model-e2e--9527f114-dockercfg-6ds6k\"" Oct 31 13:23:43.515400 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:43.515346 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0c1fbb\"/\"routing-overlay\"" Oct 31 13:23:43.515455 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:43.515406 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0c1fbb\"/\"payload-processing-external-model-plugins-e2e-extmod-b-0c1fbb\"" Oct 31 13:23:43.515455 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:43.515446 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0c1fbb\"/\"openshift-service-ca.crt\"" Oct 31 13:23:43.680635 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:43.680593 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9\" (UID: \"0e06e892-03d5-4cf7-8858-0e7a2d9adfbc\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9" Oct 31 13:23:43.681130 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:43.681081 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq6mh\" (UniqueName: \"kubernetes.io/projected/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-kube-api-access-nq6mh\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9\" (UID: \"0e06e892-03d5-4cf7-8858-0e7a2d9adfbc\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9" Oct 31 13:23:43.681473 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:43.681455 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9\" (UID: \"0e06e892-03d5-4cf7-8858-0e7a2d9adfbc\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9" Oct 31 13:23:43.681785 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:43.681762 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9\" (UID: \"0e06e892-03d5-4cf7-8858-0e7a2d9adfbc\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9" Oct 31 13:23:43.787961 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:43.787855 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq6mh\" (UniqueName: \"kubernetes.io/projected/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-kube-api-access-nq6mh\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9\" (UID: \"0e06e892-03d5-4cf7-8858-0e7a2d9adfbc\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9" Oct 31 13:23:43.788602 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:43.788577 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9\" (UID: \"0e06e892-03d5-4cf7-8858-0e7a2d9adfbc\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9" Oct 31 13:23:43.789781 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:43.789675 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9\" (UID: \"0e06e892-03d5-4cf7-8858-0e7a2d9adfbc\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9" Oct 31 13:23:43.804429 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:43.802827 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9\" (UID: \"0e06e892-03d5-4cf7-8858-0e7a2d9adfbc\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9" Oct 31 13:23:43.804429 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:43.803001 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9\" (UID: \"0e06e892-03d5-4cf7-8858-0e7a2d9adfbc\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9" Oct 31 13:23:43.804429 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:43.804010 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9\" (UID: \"0e06e892-03d5-4cf7-8858-0e7a2d9adfbc\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9" Oct 31 13:23:43.838693 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:43.836405 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9\" (UID: \"0e06e892-03d5-4cf7-8858-0e7a2d9adfbc\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9" Oct 31 13:23:43.843460 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:43.843357 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq6mh\" (UniqueName: \"kubernetes.io/projected/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-kube-api-access-nq6mh\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9\" (UID: \"0e06e892-03d5-4cf7-8858-0e7a2d9adfbc\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9" Oct 31 13:23:44.007456 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.007424 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" Oct 31 13:23:44.014239 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.014212 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" Oct 31 13:23:44.017957 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.017931 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" Oct 31 13:23:44.018727 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.018709 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" Oct 31 13:23:44.130586 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.130486 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9" Oct 31 13:23:44.161620 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.161589 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h"] Oct 31 13:23:44.168708 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.168682 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h" Oct 31 13:23:44.172695 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.172629 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h"] Oct 31 13:23:44.178862 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.178806 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-54a16b\"/\"payload-processing-external-model-plugins-e2e-extmod-a-54a16b\"" Oct 31 13:23:44.178862 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.178840 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-54a16b\"/\"routing-overlay\"" Oct 31 13:23:44.178862 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.178865 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-54a16b\"/\"openshift-service-ca.crt\"" Oct 31 13:23:44.180790 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.178948 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-54a16b\"/\"kube-root-ca.crt\"" Oct 31 13:23:44.210474 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.210435 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf08107d-640e-4a43-8994-10e51fa506ec-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h\" (UID: \"bf08107d-640e-4a43-8994-10e51fa506ec\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h" Oct 31 13:23:44.210844 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.210815 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf08107d-640e-4a43-8994-10e51fa506ec-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h\" (UID: \"bf08107d-640e-4a43-8994-10e51fa506ec\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h" Oct 31 13:23:44.211048 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.211029 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf8w4\" (UniqueName: \"kubernetes.io/projected/bf08107d-640e-4a43-8994-10e51fa506ec-kube-api-access-tf8w4\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h\" (UID: \"bf08107d-640e-4a43-8994-10e51fa506ec\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h" Oct 31 13:23:44.215439 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.215395 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf08107d-640e-4a43-8994-10e51fa506ec-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h\" (UID: \"bf08107d-640e-4a43-8994-10e51fa506ec\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h" Oct 31 13:23:44.302264 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.301609 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-54a16b\"/\"payload-processing-external-model-e2e--427d574a-dockercfg-8gs8z\"" Oct 31 13:23:44.302264 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.301714 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-54a16b\"/\"agc-cred-bac9ea917c02\"" Oct 31 13:23:44.335313 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.335272 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf08107d-640e-4a43-8994-10e51fa506ec-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h\" (UID: \"bf08107d-640e-4a43-8994-10e51fa506ec\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h" Oct 31 13:23:44.354065 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.354035 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf08107d-640e-4a43-8994-10e51fa506ec-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h\" (UID: \"bf08107d-640e-4a43-8994-10e51fa506ec\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h" Oct 31 13:23:44.354408 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.354372 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf8w4\" (UniqueName: \"kubernetes.io/projected/bf08107d-640e-4a43-8994-10e51fa506ec-kube-api-access-tf8w4\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h\" (UID: \"bf08107d-640e-4a43-8994-10e51fa506ec\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h" Oct 31 13:23:44.396719 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.336549 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf08107d-640e-4a43-8994-10e51fa506ec-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h\" (UID: \"bf08107d-640e-4a43-8994-10e51fa506ec\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h" Oct 31 13:23:44.396719 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.355421 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf08107d-640e-4a43-8994-10e51fa506ec-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h\" (UID: \"bf08107d-640e-4a43-8994-10e51fa506ec\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h" Oct 31 13:23:44.437670 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.435466 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf8w4\" (UniqueName: \"kubernetes.io/projected/bf08107d-640e-4a43-8994-10e51fa506ec-kube-api-access-tf8w4\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h\" (UID: \"bf08107d-640e-4a43-8994-10e51fa506ec\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h" Oct 31 13:23:44.437670 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.436826 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf08107d-640e-4a43-8994-10e51fa506ec-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h\" (UID: \"bf08107d-640e-4a43-8994-10e51fa506ec\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h" Oct 31 13:23:44.441311 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.441281 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf08107d-640e-4a43-8994-10e51fa506ec-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h\" (UID: \"bf08107d-640e-4a43-8994-10e51fa506ec\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h" Oct 31 13:23:44.565239 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.565206 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx"] Oct 31 13:23:44.572507 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.572479 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" Oct 31 13:23:44.615861 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.583385 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-5a9e45-openshift-default-dockercfg-szwmr\"" Oct 31 13:23:44.625997 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.625809 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9"] Oct 31 13:23:44.636409 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.636379 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx"] Oct 31 13:23:44.641014 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.640208 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f368dfe0-177d-4e5f-9d68-f0f440717fe2-istio-envoy\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx\" (UID: \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" Oct 31 13:23:44.641014 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.640263 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f368dfe0-177d-4e5f-9d68-f0f440717fe2-istio-token\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx\" (UID: \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" Oct 31 13:23:44.641014 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.640291 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f368dfe0-177d-4e5f-9d68-f0f440717fe2-credential-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx\" (UID: \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" Oct 31 13:23:44.641014 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.640342 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f368dfe0-177d-4e5f-9d68-f0f440717fe2-istiod-ca-cert\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx\" (UID: \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" Oct 31 13:23:44.641014 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.640416 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f368dfe0-177d-4e5f-9d68-f0f440717fe2-workload-certs\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx\" (UID: \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" Oct 31 13:23:44.641014 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.640456 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f368dfe0-177d-4e5f-9d68-f0f440717fe2-istio-podinfo\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx\" (UID: \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" Oct 31 13:23:44.641014 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.640577 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdz9r\" (UniqueName: \"kubernetes.io/projected/f368dfe0-177d-4e5f-9d68-f0f440717fe2-kube-api-access-bdz9r\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx\" (UID: \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" Oct 31 13:23:44.641014 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.640630 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f368dfe0-177d-4e5f-9d68-f0f440717fe2-istio-data\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx\" (UID: \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" Oct 31 13:23:44.641014 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.640744 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f368dfe0-177d-4e5f-9d68-f0f440717fe2-workload-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx\" (UID: \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" Oct 31 13:23:44.659616 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.659580 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-76758ccf48-lxlmg" podUID="31021a03-2b9d-4db5-8934-78bf39bd308f" containerName="payload-processing" containerID="cri-o://26cdf73a08f1cc8994f92a97a09560716c674103fa0b93f27535d88031643dc6" gracePeriod=25 Oct 31 13:23:44.736734 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.736695 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h" Oct 31 13:23:44.743081 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.742525 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f368dfe0-177d-4e5f-9d68-f0f440717fe2-istio-envoy\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx\" (UID: \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" Oct 31 13:23:44.743081 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.742581 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f368dfe0-177d-4e5f-9d68-f0f440717fe2-istio-token\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx\" (UID: \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" Oct 31 13:23:44.743081 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.742611 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f368dfe0-177d-4e5f-9d68-f0f440717fe2-credential-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx\" (UID: \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" Oct 31 13:23:44.743081 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.742693 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f368dfe0-177d-4e5f-9d68-f0f440717fe2-istiod-ca-cert\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx\" (UID: \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" Oct 31 13:23:44.743081 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.742786 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f368dfe0-177d-4e5f-9d68-f0f440717fe2-workload-certs\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx\" (UID: \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" Oct 31 13:23:44.743459 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.743391 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f368dfe0-177d-4e5f-9d68-f0f440717fe2-workload-certs\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx\" (UID: \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" Oct 31 13:23:44.743459 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.743443 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f368dfe0-177d-4e5f-9d68-f0f440717fe2-credential-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx\" (UID: \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" Oct 31 13:23:44.743671 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.743632 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f368dfe0-177d-4e5f-9d68-f0f440717fe2-istiod-ca-cert\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx\" (UID: \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" Oct 31 13:23:44.745123 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.745096 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f368dfe0-177d-4e5f-9d68-f0f440717fe2-istio-podinfo\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx\" (UID: \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" Oct 31 13:23:44.745305 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.745255 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdz9r\" (UniqueName: \"kubernetes.io/projected/f368dfe0-177d-4e5f-9d68-f0f440717fe2-kube-api-access-bdz9r\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx\" (UID: \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" Oct 31 13:23:44.745370 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.745355 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f368dfe0-177d-4e5f-9d68-f0f440717fe2-istio-data\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx\" (UID: \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" Oct 31 13:23:44.745473 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.745454 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f368dfe0-177d-4e5f-9d68-f0f440717fe2-workload-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx\" (UID: \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" Oct 31 13:23:44.745794 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.745773 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f368dfe0-177d-4e5f-9d68-f0f440717fe2-istio-data\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx\" (UID: \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" Oct 31 13:23:44.745912 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.745872 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f368dfe0-177d-4e5f-9d68-f0f440717fe2-workload-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx\" (UID: \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" Oct 31 13:23:44.748004 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.747979 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f368dfe0-177d-4e5f-9d68-f0f440717fe2-istio-podinfo\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx\" (UID: \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" Oct 31 13:23:44.750523 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.750493 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f368dfe0-177d-4e5f-9d68-f0f440717fe2-istio-envoy\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx\" (UID: \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" Oct 31 13:23:44.773510 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.773424 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdz9r\" (UniqueName: \"kubernetes.io/projected/f368dfe0-177d-4e5f-9d68-f0f440717fe2-kube-api-access-bdz9r\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx\" (UID: \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" Oct 31 13:23:44.774487 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.774463 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f368dfe0-177d-4e5f-9d68-f0f440717fe2-istio-token\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx\" (UID: \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" Oct 31 13:23:44.905095 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:44.905064 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" Oct 31 13:23:45.169531 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:45.168885 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9"] Oct 31 13:23:45.172316 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:45.170661 5693 generic.go:356] "Generic (PLEG): container finished" podID="31021a03-2b9d-4db5-8934-78bf39bd308f" containerID="26cdf73a08f1cc8994f92a97a09560716c674103fa0b93f27535d88031643dc6" exitCode=0 Oct 31 13:23:45.172316 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:45.170838 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-76758ccf48-lxlmg" event={"ID":"31021a03-2b9d-4db5-8934-78bf39bd308f","Type":"ContainerDied","Data":"26cdf73a08f1cc8994f92a97a09560716c674103fa0b93f27535d88031643dc6"} Oct 31 13:23:45.174593 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:45.174119 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9" event={"ID":"0e06e892-03d5-4cf7-8858-0e7a2d9adfbc","Type":"ContainerStarted","Data":"fdbdb7911f500591a12e037af05e85439141f83b3a66d2db3f1d80ec8c6db12c"} Oct 31 13:23:45.174593 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:45.174171 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9" Oct 31 13:23:45.174593 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:45.174195 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9" event={"ID":"0e06e892-03d5-4cf7-8858-0e7a2d9adfbc","Type":"ContainerStarted","Data":"bd5cfd07ed5badfcbfb0ac421ea0b16613a73b5cf01386dd27c9b26fec0a7a73"} Oct 31 13:23:45.184843 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:45.183164 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h"] Oct 31 13:23:45.186626 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:23:45.186599 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf08107d_640e_4a43_8994_10e51fa506ec.slice/crio-38ae2c034abc08e872a010dc55f86a7875364b74cb259a45694f4375ac26ac64 WatchSource:0}: Error finding container 38ae2c034abc08e872a010dc55f86a7875364b74cb259a45694f4375ac26ac64: Status 404 returned error can't find the container with id 38ae2c034abc08e872a010dc55f86a7875364b74cb259a45694f4375ac26ac64 Oct 31 13:23:45.204095 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:45.204039 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9" podStartSLOduration=2.204021124 podStartE2EDuration="2.204021124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:23:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:23:45.195517282 +0000 UTC m=+3075.903103578" watchObservedRunningTime="2026-10-31 13:23:45.204021124 +0000 UTC m=+3075.911607420" Oct 31 13:23:45.240402 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:45.240375 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-76758ccf48-lxlmg" Oct 31 13:23:45.250351 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:45.250324 5693 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-plugins-e2e-extmod-b-0c1fbb: configmap "payload-processing-external-model-plugins-e2e-extmod-b-0c1fbb" not found Oct 31 13:23:45.250524 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:45.250397 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-plugins-config-volume podName:0e06e892-03d5-4cf7-8858-0e7a2d9adfbc nodeName:}" failed. No retries permitted until 2026-10-31 13:23:45.750376373 +0000 UTC m=+3076.457962668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9" (UID: "0e06e892-03d5-4cf7-8858-0e7a2d9adfbc") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-0c1fbb" not found Oct 31 13:23:45.250524 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:45.250441 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-0c1fbb/agc-cred-ac74976b3ec1: secret "agc-cred-ac74976b3ec1" not found Oct 31 13:23:45.250524 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:45.250463 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9: secret "agc-cred-ac74976b3ec1" not found Oct 31 13:23:45.250794 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:45.250552 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-provider-credentials podName:0e06e892-03d5-4cf7-8858-0e7a2d9adfbc nodeName:}" failed. No retries permitted until 2026-10-31 13:23:45.750532698 +0000 UTC m=+3076.458118975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9" (UID: "0e06e892-03d5-4cf7-8858-0e7a2d9adfbc") : secret "agc-cred-ac74976b3ec1" not found Oct 31 13:23:45.351490 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:45.351461 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31021a03-2b9d-4db5-8934-78bf39bd308f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31021a03-2b9d-4db5-8934-78bf39bd308f-routing-overlay-volume\") pod \"31021a03-2b9d-4db5-8934-78bf39bd308f\" (UID: \"31021a03-2b9d-4db5-8934-78bf39bd308f\") " Oct 31 13:23:45.351713 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:45.351514 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31021a03-2b9d-4db5-8934-78bf39bd308f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/31021a03-2b9d-4db5-8934-78bf39bd308f-provider-credentials\") pod \"31021a03-2b9d-4db5-8934-78bf39bd308f\" (UID: \"31021a03-2b9d-4db5-8934-78bf39bd308f\") " Oct 31 13:23:45.351713 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:45.351617 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31021a03-2b9d-4db5-8934-78bf39bd308f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31021a03-2b9d-4db5-8934-78bf39bd308f-plugins-config-volume\") pod \"31021a03-2b9d-4db5-8934-78bf39bd308f\" (UID: \"31021a03-2b9d-4db5-8934-78bf39bd308f\") " Oct 31 13:23:45.351898 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:45.351880 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31021a03-2b9d-4db5-8934-78bf39bd308f-kube-api-access-b9b7v\" (UniqueName: \"kubernetes.io/projected/31021a03-2b9d-4db5-8934-78bf39bd308f-kube-api-access-b9b7v\") pod \"31021a03-2b9d-4db5-8934-78bf39bd308f\" (UID: \"31021a03-2b9d-4db5-8934-78bf39bd308f\") " Oct 31 13:23:45.351988 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:45.351841 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31021a03-2b9d-4db5-8934-78bf39bd308f-routing-overlay-volume" pod "31021a03-2b9d-4db5-8934-78bf39bd308f" (UID: "31021a03-2b9d-4db5-8934-78bf39bd308f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:23:45.352092 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:45.352071 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31021a03-2b9d-4db5-8934-78bf39bd308f-plugins-config-volume" pod "31021a03-2b9d-4db5-8934-78bf39bd308f" (UID: "31021a03-2b9d-4db5-8934-78bf39bd308f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:23:45.352553 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:45.352537 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31021a03-2b9d-4db5-8934-78bf39bd308f-routing-overlay-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:45.352595 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:45.352557 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31021a03-2b9d-4db5-8934-78bf39bd308f-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:45.354078 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:45.354057 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31021a03-2b9d-4db5-8934-78bf39bd308f-provider-credentials" pod "31021a03-2b9d-4db5-8934-78bf39bd308f" (UID: "31021a03-2b9d-4db5-8934-78bf39bd308f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:23:45.354125 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:45.354076 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31021a03-2b9d-4db5-8934-78bf39bd308f-kube-api-access-b9b7v" pod "31021a03-2b9d-4db5-8934-78bf39bd308f" (UID: "31021a03-2b9d-4db5-8934-78bf39bd308f"). InnerVolumeSpecName "kube-api-access-b9b7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:23:45.364240 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:45.364218 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx"] Oct 31 13:23:45.364550 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:23:45.364531 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf368dfe0_177d_4e5f_9d68_f0f440717fe2.slice/crio-74430db3282c9b59a1ed9925855e82e1e45174ef20d522084c82cb9d0fab71a5 WatchSource:0}: Error finding container 74430db3282c9b59a1ed9925855e82e1e45174ef20d522084c82cb9d0fab71a5: Status 404 returned error can't find the container with id 74430db3282c9b59a1ed9925855e82e1e45174ef20d522084c82cb9d0fab71a5 Oct 31 13:23:45.367517 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:45.367485 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:23:45.367584 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:45.367552 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:23:45.367622 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:45.367581 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:23:45.457486 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:45.457348 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9b7v\" (UniqueName: \"kubernetes.io/projected/31021a03-2b9d-4db5-8934-78bf39bd308f-kube-api-access-b9b7v\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:45.457486 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:45.457394 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31021a03-2b9d-4db5-8934-78bf39bd308f-provider-credentials\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:45.705610 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:45.705571 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h"] Oct 31 13:23:45.786855 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:45.786769 5693 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-plugins-e2e-extmod-b-0c1fbb: configmap "payload-processing-external-model-plugins-e2e-extmod-b-0c1fbb" not found Oct 31 13:23:45.787143 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:45.787129 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-plugins-config-volume podName:0e06e892-03d5-4cf7-8858-0e7a2d9adfbc nodeName:}" failed. No retries permitted until 2026-10-31 13:23:46.787104377 +0000 UTC m=+3077.494690654 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9" (UID: "0e06e892-03d5-4cf7-8858-0e7a2d9adfbc") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-0c1fbb" not found Oct 31 13:23:45.787774 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:45.787749 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-0c1fbb/agc-cred-ac74976b3ec1: secret "agc-cred-ac74976b3ec1" not found Oct 31 13:23:45.787927 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:45.787916 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9: secret "agc-cred-ac74976b3ec1" not found Oct 31 13:23:45.788063 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:45.788054 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-provider-credentials podName:0e06e892-03d5-4cf7-8858-0e7a2d9adfbc nodeName:}" failed. No retries permitted until 2026-10-31 13:23:46.788038703 +0000 UTC m=+3077.495624995 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9" (UID: "0e06e892-03d5-4cf7-8858-0e7a2d9adfbc") : secret "agc-cred-ac74976b3ec1" not found Oct 31 13:23:46.184110 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:46.184075 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h" event={"ID":"bf08107d-640e-4a43-8994-10e51fa506ec","Type":"ContainerStarted","Data":"feeadd17b8e77eee7c3a65c1db256e3a68c1e2e9078aff202cb69a2a72559b58"} Oct 31 13:23:46.184564 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:46.184134 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h" event={"ID":"bf08107d-640e-4a43-8994-10e51fa506ec","Type":"ContainerStarted","Data":"38ae2c034abc08e872a010dc55f86a7875364b74cb259a45694f4375ac26ac64"} Oct 31 13:23:46.184564 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:46.184165 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h" Oct 31 13:23:46.184564 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:46.184329 5693 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h" secret="" err="secret \"payload-processing-external-model-e2e--427d574a-dockercfg-8gs8z\" not found" Oct 31 13:23:46.198757 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:46.198722 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" event={"ID":"f368dfe0-177d-4e5f-9d68-f0f440717fe2","Type":"ContainerStarted","Data":"f4ae3f8c7bcbeb87b58a8a8db24a1c50661483091f435f72d16ab0b17d318bef"} Oct 31 13:23:46.199032 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:46.199012 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" event={"ID":"f368dfe0-177d-4e5f-9d68-f0f440717fe2","Type":"ContainerStarted","Data":"74430db3282c9b59a1ed9925855e82e1e45174ef20d522084c82cb9d0fab71a5"} Oct 31 13:23:46.203844 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:46.203814 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-76758ccf48-lxlmg" event={"ID":"31021a03-2b9d-4db5-8934-78bf39bd308f","Type":"ContainerDied","Data":"efa298b0db10c6a088aabd62877ab460905754a3e4d9a1c2ec389e65b0ece1cf"} Oct 31 13:23:46.204407 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:46.204382 5693 scope.go:122] "RemoveContainer" containerID="26cdf73a08f1cc8994f92a97a09560716c674103fa0b93f27535d88031643dc6" Oct 31 13:23:46.204761 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:46.203851 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-76758ccf48-lxlmg" Oct 31 13:23:46.235175 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:46.235124 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h" podStartSLOduration=2.2351045 podStartE2EDuration="2.2351045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:23:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:23:46.226657849 +0000 UTC m=+3076.934244137" watchObservedRunningTime="2026-10-31 13:23:46.2351045 +0000 UTC m=+3076.942690798" Oct 31 13:23:46.319599 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:46.319555 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-54a16b/agc-cred-bac9ea917c02: secret "agc-cred-bac9ea917c02" not found Oct 31 13:23:46.323141 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:46.322872 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h: secret "agc-cred-bac9ea917c02" not found Oct 31 13:23:46.323141 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:46.322961 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf08107d-640e-4a43-8994-10e51fa506ec-provider-credentials podName:bf08107d-640e-4a43-8994-10e51fa506ec nodeName:}" failed. No retries permitted until 2026-10-31 13:23:46.822936703 +0000 UTC m=+3077.530522977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bf08107d-640e-4a43-8994-10e51fa506ec-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h" (UID: "bf08107d-640e-4a43-8994-10e51fa506ec") : secret "agc-cred-bac9ea917c02" not found Oct 31 13:23:46.326013 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:46.325989 5693 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-plugins-e2e-extmod-a-54a16b: configmap "payload-processing-external-model-plugins-e2e-extmod-a-54a16b" not found Oct 31 13:23:46.328699 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:46.328623 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bf08107d-640e-4a43-8994-10e51fa506ec-plugins-config-volume podName:bf08107d-640e-4a43-8994-10e51fa506ec nodeName:}" failed. No retries permitted until 2026-10-31 13:23:46.828586499 +0000 UTC m=+3077.536172987 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/bf08107d-640e-4a43-8994-10e51fa506ec-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h" (UID: "bf08107d-640e-4a43-8994-10e51fa506ec") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-54a16b" not found Oct 31 13:23:46.380223 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:46.380169 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" podStartSLOduration=2.380149882 podStartE2EDuration="2.380149882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:23:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:23:46.367329475 +0000 UTC m=+3077.074915771" watchObservedRunningTime="2026-10-31 13:23:46.380149882 +0000 UTC m=+3077.087736179" Oct 31 13:23:46.397073 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:46.397033 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-76758ccf48-lxlmg"] Oct 31 13:23:46.410580 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:46.410547 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-76758ccf48-lxlmg"] Oct 31 13:23:46.826750 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:46.826717 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-54a16b/agc-cred-bac9ea917c02: secret "agc-cred-bac9ea917c02" not found Oct 31 13:23:46.826750 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:46.826742 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h: secret "agc-cred-bac9ea917c02" not found Oct 31 13:23:46.826995 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:46.826779 5693 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-plugins-e2e-extmod-b-0c1fbb: configmap "payload-processing-external-model-plugins-e2e-extmod-b-0c1fbb" not found Oct 31 13:23:46.826995 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:46.826790 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf08107d-640e-4a43-8994-10e51fa506ec-provider-credentials podName:bf08107d-640e-4a43-8994-10e51fa506ec nodeName:}" failed. No retries permitted until 2026-10-31 13:23:47.82677722 +0000 UTC m=+3078.534363494 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bf08107d-640e-4a43-8994-10e51fa506ec-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h" (UID: "bf08107d-640e-4a43-8994-10e51fa506ec") : secret "agc-cred-bac9ea917c02" not found Oct 31 13:23:46.826995 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:46.826838 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-0c1fbb/agc-cred-ac74976b3ec1: secret "agc-cred-ac74976b3ec1" not found Oct 31 13:23:46.826995 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:46.826857 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9: secret "agc-cred-ac74976b3ec1" not found Oct 31 13:23:46.826995 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:46.826858 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-plugins-config-volume podName:0e06e892-03d5-4cf7-8858-0e7a2d9adfbc nodeName:}" failed. No retries permitted until 2026-10-31 13:23:48.826841372 +0000 UTC m=+3079.534427645 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9" (UID: "0e06e892-03d5-4cf7-8858-0e7a2d9adfbc") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-0c1fbb" not found Oct 31 13:23:46.826995 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:46.826900 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-provider-credentials podName:0e06e892-03d5-4cf7-8858-0e7a2d9adfbc nodeName:}" failed. No retries permitted until 2026-10-31 13:23:48.826888435 +0000 UTC m=+3079.534474725 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9" (UID: "0e06e892-03d5-4cf7-8858-0e7a2d9adfbc") : secret "agc-cred-ac74976b3ec1" not found Oct 31 13:23:46.906016 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:46.905982 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" Oct 31 13:23:46.907855 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:46.907820 5693 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 31 13:23:46.908009 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:46.907877 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" podUID="f368dfe0-177d-4e5f-9d68-f0f440717fe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 31 13:23:46.928709 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:46.928680 5693 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-plugins-e2e-extmod-a-54a16b: configmap "payload-processing-external-model-plugins-e2e-extmod-a-54a16b" not found Oct 31 13:23:46.928879 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:46.928764 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bf08107d-640e-4a43-8994-10e51fa506ec-plugins-config-volume podName:bf08107d-640e-4a43-8994-10e51fa506ec nodeName:}" failed. No retries permitted until 2026-10-31 13:23:47.928742949 +0000 UTC m=+3078.636329239 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/bf08107d-640e-4a43-8994-10e51fa506ec-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h" (UID: "bf08107d-640e-4a43-8994-10e51fa506ec") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-54a16b" not found Oct 31 13:23:47.327390 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:47.327349 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4023ad-7994b685f7-9m2jh"] Oct 31 13:23:47.328550 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:47.328516 5693 cpu_manager.go:414] "containerMap: removing container" podUID="31021a03-2b9d-4db5-8934-78bf39bd308f" containerName="payload-processing" Oct 31 13:23:47.328550 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:47.328541 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31021a03-2b9d-4db5-8934-78bf39bd308f" containerName="payload-processing" Oct 31 13:23:47.328802 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:47.328770 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="31021a03-2b9d-4db5-8934-78bf39bd308f" containerName="payload-processing" Oct 31 13:23:47.334892 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:47.334843 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4023ad-7994b685f7-9m2jh" Oct 31 13:23:47.344399 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:47.344076 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-4023ad-dockercfg-m44m7\"" Oct 31 13:23:47.344399 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:47.344214 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-4023ad\"" Oct 31 13:23:47.368386 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:47.368352 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4023ad-7994b685f7-9m2jh"] Oct 31 13:23:47.417281 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:47.417247 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31021a03-2b9d-4db5-8934-78bf39bd308f" path="/var/lib/kubelet/pods/31021a03-2b9d-4db5-8934-78bf39bd308f/volumes" Oct 31 13:23:47.438461 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:47.438431 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc39dc34-d1e4-466d-bc1d-5e77bac8dc39-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4023ad-7994b685f7-9m2jh\" (UID: \"bc39dc34-d1e4-466d-bc1d-5e77bac8dc39\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4023ad-7994b685f7-9m2jh" Oct 31 13:23:47.438627 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:47.438594 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxfcg\" (UniqueName: \"kubernetes.io/projected/bc39dc34-d1e4-466d-bc1d-5e77bac8dc39-kube-api-access-gxfcg\") pod \"payload-processing-e2e-inf-a-4023ad-7994b685f7-9m2jh\" (UID: \"bc39dc34-d1e4-466d-bc1d-5e77bac8dc39\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4023ad-7994b685f7-9m2jh" Oct 31 13:23:47.567829 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:47.547189 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc39dc34-d1e4-466d-bc1d-5e77bac8dc39-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4023ad-7994b685f7-9m2jh\" (UID: \"bc39dc34-d1e4-466d-bc1d-5e77bac8dc39\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4023ad-7994b685f7-9m2jh" Oct 31 13:23:47.567829 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:47.547345 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxfcg\" (UniqueName: \"kubernetes.io/projected/bc39dc34-d1e4-466d-bc1d-5e77bac8dc39-kube-api-access-gxfcg\") pod \"payload-processing-e2e-inf-a-4023ad-7994b685f7-9m2jh\" (UID: \"bc39dc34-d1e4-466d-bc1d-5e77bac8dc39\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4023ad-7994b685f7-9m2jh" Oct 31 13:23:47.567829 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:47.565317 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc39dc34-d1e4-466d-bc1d-5e77bac8dc39-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4023ad-7994b685f7-9m2jh\" (UID: \"bc39dc34-d1e4-466d-bc1d-5e77bac8dc39\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4023ad-7994b685f7-9m2jh" Oct 31 13:23:47.622520 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:47.622408 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxfcg\" (UniqueName: \"kubernetes.io/projected/bc39dc34-d1e4-466d-bc1d-5e77bac8dc39-kube-api-access-gxfcg\") pod \"payload-processing-e2e-inf-a-4023ad-7994b685f7-9m2jh\" (UID: \"bc39dc34-d1e4-466d-bc1d-5e77bac8dc39\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4023ad-7994b685f7-9m2jh" Oct 31 13:23:47.654986 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:47.654711 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4023ad-7994b685f7-9m2jh" Oct 31 13:23:47.880338 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:47.879877 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-54a16b/agc-cred-bac9ea917c02: secret "agc-cred-bac9ea917c02" not found Oct 31 13:23:47.880338 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:47.879908 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h: secret "agc-cred-bac9ea917c02" not found Oct 31 13:23:47.880338 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:47.879974 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf08107d-640e-4a43-8994-10e51fa506ec-provider-credentials podName:bf08107d-640e-4a43-8994-10e51fa506ec nodeName:}" failed. No retries permitted until 2026-10-31 13:23:49.879953937 +0000 UTC m=+3080.587540213 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bf08107d-640e-4a43-8994-10e51fa506ec-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h" (UID: "bf08107d-640e-4a43-8994-10e51fa506ec") : secret "agc-cred-bac9ea917c02" not found Oct 31 13:23:47.924205 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:47.909140 5693 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 31 13:23:47.924205 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:47.909200 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" podUID="f368dfe0-177d-4e5f-9d68-f0f440717fe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 31 13:23:48.001369 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:48.001057 5693 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-plugins-e2e-extmod-a-54a16b: configmap "payload-processing-external-model-plugins-e2e-extmod-a-54a16b" not found Oct 31 13:23:48.001369 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:48.001127 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bf08107d-640e-4a43-8994-10e51fa506ec-plugins-config-volume podName:bf08107d-640e-4a43-8994-10e51fa506ec nodeName:}" failed. No retries permitted until 2026-10-31 13:23:50.001112662 +0000 UTC m=+3080.708698959 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/bf08107d-640e-4a43-8994-10e51fa506ec-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h" (UID: "bf08107d-640e-4a43-8994-10e51fa506ec") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-54a16b" not found Oct 31 13:23:48.110212 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:48.110180 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4023ad-7994b685f7-9m2jh"] Oct 31 13:23:48.110725 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:23:48.110695 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc39dc34_d1e4_466d_bc1d_5e77bac8dc39.slice/crio-0c29280248f3f683c5b2e0feb0e4c1d18c472b66fb3cf6aa46a7fd93248e23a4 WatchSource:0}: Error finding container 0c29280248f3f683c5b2e0feb0e4c1d18c472b66fb3cf6aa46a7fd93248e23a4: Status 404 returned error can't find the container with id 0c29280248f3f683c5b2e0feb0e4c1d18c472b66fb3cf6aa46a7fd93248e23a4 Oct 31 13:23:48.229191 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:48.229158 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4023ad-7994b685f7-9m2jh" event={"ID":"bc39dc34-d1e4-466d-bc1d-5e77bac8dc39","Type":"ContainerStarted","Data":"14d06494c6a1c4e9297ea00ecd34d28a7928cd274fbe50098db8b70b9853bd05"} Oct 31 13:23:48.229349 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:48.229218 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4023ad-7994b685f7-9m2jh" event={"ID":"bc39dc34-d1e4-466d-bc1d-5e77bac8dc39","Type":"ContainerStarted","Data":"0c29280248f3f683c5b2e0feb0e4c1d18c472b66fb3cf6aa46a7fd93248e23a4"} Oct 31 13:23:48.229349 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:48.229249 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4023ad-7994b685f7-9m2jh" Oct 31 13:23:48.256314 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:48.256260 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-4023ad-7994b685f7-9m2jh" podStartSLOduration=1.2562469410000001 podStartE2EDuration="1.256246941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:23:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:23:48.251547763 +0000 UTC m=+3078.959134082" watchObservedRunningTime="2026-10-31 13:23:48.256246941 +0000 UTC m=+3078.963833235" Oct 31 13:23:48.707615 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:48.707576 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:23:48.909669 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:48.908131 5693 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 31 13:23:48.909669 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:48.908218 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" podUID="f368dfe0-177d-4e5f-9d68-f0f440717fe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 31 13:23:48.920937 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:48.920602 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-0c1fbb/agc-cred-ac74976b3ec1: secret "agc-cred-ac74976b3ec1" not found Oct 31 13:23:48.920937 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:48.920628 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9: secret "agc-cred-ac74976b3ec1" not found Oct 31 13:23:48.920937 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:48.920712 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-provider-credentials podName:0e06e892-03d5-4cf7-8858-0e7a2d9adfbc nodeName:}" failed. No retries permitted until 2026-10-31 13:23:52.920694395 +0000 UTC m=+3083.628280669 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9" (UID: "0e06e892-03d5-4cf7-8858-0e7a2d9adfbc") : secret "agc-cred-ac74976b3ec1" not found Oct 31 13:23:48.921178 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:48.920963 5693 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-plugins-e2e-extmod-b-0c1fbb: configmap "payload-processing-external-model-plugins-e2e-extmod-b-0c1fbb" not found Oct 31 13:23:48.921178 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:48.921054 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-plugins-config-volume podName:0e06e892-03d5-4cf7-8858-0e7a2d9adfbc nodeName:}" failed. No retries permitted until 2026-10-31 13:23:52.921036902 +0000 UTC m=+3083.628623189 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9" (UID: "0e06e892-03d5-4cf7-8858-0e7a2d9adfbc") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-0c1fbb" not found Oct 31 13:23:49.905902 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:49.905862 5693 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 31 13:23:49.906422 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:49.905942 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" podUID="f368dfe0-177d-4e5f-9d68-f0f440717fe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 31 13:23:49.945422 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:49.945391 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-54a16b/agc-cred-bac9ea917c02: secret "agc-cred-bac9ea917c02" not found Oct 31 13:23:49.945422 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:49.945418 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h: secret "agc-cred-bac9ea917c02" not found Oct 31 13:23:49.945737 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:49.945480 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf08107d-640e-4a43-8994-10e51fa506ec-provider-credentials podName:bf08107d-640e-4a43-8994-10e51fa506ec nodeName:}" failed. No retries permitted until 2026-10-31 13:23:53.945461638 +0000 UTC m=+3084.653047915 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bf08107d-640e-4a43-8994-10e51fa506ec-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h" (UID: "bf08107d-640e-4a43-8994-10e51fa506ec") : secret "agc-cred-bac9ea917c02" not found Oct 31 13:23:50.047400 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:50.047364 5693 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-plugins-e2e-extmod-a-54a16b: configmap "payload-processing-external-model-plugins-e2e-extmod-a-54a16b" not found Oct 31 13:23:50.047583 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:50.047447 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bf08107d-640e-4a43-8994-10e51fa506ec-plugins-config-volume podName:bf08107d-640e-4a43-8994-10e51fa506ec nodeName:}" failed. No retries permitted until 2026-10-31 13:23:54.047432288 +0000 UTC m=+3084.755018574 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/bf08107d-640e-4a43-8994-10e51fa506ec-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h" (UID: "bf08107d-640e-4a43-8994-10e51fa506ec") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-54a16b" not found Oct 31 13:23:50.906163 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:50.905925 5693 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 31 13:23:50.906163 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:50.906016 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" podUID="f368dfe0-177d-4e5f-9d68-f0f440717fe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 31 13:23:51.216721 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:51.209910 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9" podUID="0e06e892-03d5-4cf7-8858-0e7a2d9adfbc" containerName="payload-processing" containerID="cri-o://fdbdb7911f500591a12e037af05e85439141f83b3a66d2db3f1d80ec8c6db12c" gracePeriod=25 Oct 31 13:23:51.564677 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:51.564632 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9" Oct 31 13:23:51.720967 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:51.720935 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-provider-credentials\") pod \"0e06e892-03d5-4cf7-8858-0e7a2d9adfbc\" (UID: \"0e06e892-03d5-4cf7-8858-0e7a2d9adfbc\") " Oct 31 13:23:51.721162 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:51.721096 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-plugins-config-volume\") pod \"0e06e892-03d5-4cf7-8858-0e7a2d9adfbc\" (UID: \"0e06e892-03d5-4cf7-8858-0e7a2d9adfbc\") " Oct 31 13:23:51.721234 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:51.721209 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-kube-api-access-nq6mh\" (UniqueName: \"kubernetes.io/projected/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-kube-api-access-nq6mh\") pod \"0e06e892-03d5-4cf7-8858-0e7a2d9adfbc\" (UID: \"0e06e892-03d5-4cf7-8858-0e7a2d9adfbc\") " Oct 31 13:23:51.721452 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:51.721426 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-routing-overlay-volume\") pod \"0e06e892-03d5-4cf7-8858-0e7a2d9adfbc\" (UID: \"0e06e892-03d5-4cf7-8858-0e7a2d9adfbc\") " Oct 31 13:23:51.721692 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:51.721554 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-plugins-config-volume" pod "0e06e892-03d5-4cf7-8858-0e7a2d9adfbc" (UID: "0e06e892-03d5-4cf7-8858-0e7a2d9adfbc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:23:51.722016 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:51.721913 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-routing-overlay-volume" pod "0e06e892-03d5-4cf7-8858-0e7a2d9adfbc" (UID: "0e06e892-03d5-4cf7-8858-0e7a2d9adfbc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:23:51.722407 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:51.722386 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:51.722497 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:51.722409 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-routing-overlay-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:51.723733 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:51.723703 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-provider-credentials" pod "0e06e892-03d5-4cf7-8858-0e7a2d9adfbc" (UID: "0e06e892-03d5-4cf7-8858-0e7a2d9adfbc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:23:51.723944 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:51.723922 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-kube-api-access-nq6mh" pod "0e06e892-03d5-4cf7-8858-0e7a2d9adfbc" (UID: "0e06e892-03d5-4cf7-8858-0e7a2d9adfbc"). InnerVolumeSpecName "kube-api-access-nq6mh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:23:51.823700 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:51.823617 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-provider-credentials\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:51.823700 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:51.823661 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq6mh\" (UniqueName: \"kubernetes.io/projected/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc-kube-api-access-nq6mh\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:51.906599 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:51.906565 5693 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 31 13:23:51.907107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:51.906615 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" podUID="f368dfe0-177d-4e5f-9d68-f0f440717fe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 31 13:23:52.219995 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:52.219949 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h" podUID="bf08107d-640e-4a43-8994-10e51fa506ec" containerName="payload-processing" containerID="cri-o://feeadd17b8e77eee7c3a65c1db256e3a68c1e2e9078aff202cb69a2a72559b58" gracePeriod=25 Oct 31 13:23:52.314907 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:52.314876 5693 generic.go:356] "Generic (PLEG): container finished" podID="0e06e892-03d5-4cf7-8858-0e7a2d9adfbc" containerID="fdbdb7911f500591a12e037af05e85439141f83b3a66d2db3f1d80ec8c6db12c" exitCode=0 Oct 31 13:23:52.315082 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:52.314949 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9" Oct 31 13:23:52.315082 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:52.314967 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9" event={"ID":"0e06e892-03d5-4cf7-8858-0e7a2d9adfbc","Type":"ContainerDied","Data":"fdbdb7911f500591a12e037af05e85439141f83b3a66d2db3f1d80ec8c6db12c"} Oct 31 13:23:52.315082 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:52.315024 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9" event={"ID":"0e06e892-03d5-4cf7-8858-0e7a2d9adfbc","Type":"ContainerDied","Data":"bd5cfd07ed5badfcbfb0ac421ea0b16613a73b5cf01386dd27c9b26fec0a7a73"} Oct 31 13:23:52.315082 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:52.315046 5693 scope.go:122] "RemoveContainer" containerID="fdbdb7911f500591a12e037af05e85439141f83b3a66d2db3f1d80ec8c6db12c" Oct 31 13:23:52.344620 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:52.344588 5693 scope.go:122] "RemoveContainer" containerID="fdbdb7911f500591a12e037af05e85439141f83b3a66d2db3f1d80ec8c6db12c" Oct 31 13:23:52.357524 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:52.357270 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdbdb7911f500591a12e037af05e85439141f83b3a66d2db3f1d80ec8c6db12c\": container with ID starting with fdbdb7911f500591a12e037af05e85439141f83b3a66d2db3f1d80ec8c6db12c not found: ID does not exist" containerID="fdbdb7911f500591a12e037af05e85439141f83b3a66d2db3f1d80ec8c6db12c" Oct 31 13:23:52.357524 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:52.357319 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdbdb7911f500591a12e037af05e85439141f83b3a66d2db3f1d80ec8c6db12c"} err="failed to get container status \"fdbdb7911f500591a12e037af05e85439141f83b3a66d2db3f1d80ec8c6db12c\": rpc error: code = NotFound desc = could not find container \"fdbdb7911f500591a12e037af05e85439141f83b3a66d2db3f1d80ec8c6db12c\": container with ID starting with fdbdb7911f500591a12e037af05e85439141f83b3a66d2db3f1d80ec8c6db12c not found: ID does not exist" Oct 31 13:23:52.371045 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:52.371012 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9"] Oct 31 13:23:52.379087 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:52.379053 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6745trqf9"] Oct 31 13:23:52.607700 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:52.607671 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h" Oct 31 13:23:52.754091 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:52.754003 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf08107d-640e-4a43-8994-10e51fa506ec-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf08107d-640e-4a43-8994-10e51fa506ec-routing-overlay-volume\") pod \"bf08107d-640e-4a43-8994-10e51fa506ec\" (UID: \"bf08107d-640e-4a43-8994-10e51fa506ec\") " Oct 31 13:23:52.754091 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:52.754068 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf08107d-640e-4a43-8994-10e51fa506ec-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf08107d-640e-4a43-8994-10e51fa506ec-provider-credentials\") pod \"bf08107d-640e-4a43-8994-10e51fa506ec\" (UID: \"bf08107d-640e-4a43-8994-10e51fa506ec\") " Oct 31 13:23:52.754325 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:52.754232 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf08107d-640e-4a43-8994-10e51fa506ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf08107d-640e-4a43-8994-10e51fa506ec-plugins-config-volume\") pod \"bf08107d-640e-4a43-8994-10e51fa506ec\" (UID: \"bf08107d-640e-4a43-8994-10e51fa506ec\") " Oct 31 13:23:52.754325 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:52.754265 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf08107d-640e-4a43-8994-10e51fa506ec-kube-api-access-tf8w4\" (UniqueName: \"kubernetes.io/projected/bf08107d-640e-4a43-8994-10e51fa506ec-kube-api-access-tf8w4\") pod \"bf08107d-640e-4a43-8994-10e51fa506ec\" (UID: \"bf08107d-640e-4a43-8994-10e51fa506ec\") " Oct 31 13:23:52.754445 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:52.754402 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf08107d-640e-4a43-8994-10e51fa506ec-routing-overlay-volume" pod "bf08107d-640e-4a43-8994-10e51fa506ec" (UID: "bf08107d-640e-4a43-8994-10e51fa506ec"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:23:52.754698 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:52.754670 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf08107d-640e-4a43-8994-10e51fa506ec-plugins-config-volume" pod "bf08107d-640e-4a43-8994-10e51fa506ec" (UID: "bf08107d-640e-4a43-8994-10e51fa506ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:23:52.754923 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:52.754904 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf08107d-640e-4a43-8994-10e51fa506ec-routing-overlay-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:52.754996 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:52.754925 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf08107d-640e-4a43-8994-10e51fa506ec-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:52.756701 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:52.756654 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf08107d-640e-4a43-8994-10e51fa506ec-provider-credentials" pod "bf08107d-640e-4a43-8994-10e51fa506ec" (UID: "bf08107d-640e-4a43-8994-10e51fa506ec"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:23:52.756979 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:52.756959 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf08107d-640e-4a43-8994-10e51fa506ec-kube-api-access-tf8w4" pod "bf08107d-640e-4a43-8994-10e51fa506ec" (UID: "bf08107d-640e-4a43-8994-10e51fa506ec"). InnerVolumeSpecName "kube-api-access-tf8w4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:23:52.857152 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:52.857114 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf8w4\" (UniqueName: \"kubernetes.io/projected/bf08107d-640e-4a43-8994-10e51fa506ec-kube-api-access-tf8w4\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:52.857152 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:52.857152 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf08107d-640e-4a43-8994-10e51fa506ec-provider-credentials\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:52.911856 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:52.911817 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" Oct 31 13:23:52.912349 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:52.912314 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" Oct 31 13:23:52.913008 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:52.912990 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" Oct 31 13:23:53.366666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:53.352080 5693 generic.go:356] "Generic (PLEG): container finished" podID="bf08107d-640e-4a43-8994-10e51fa506ec" containerID="feeadd17b8e77eee7c3a65c1db256e3a68c1e2e9078aff202cb69a2a72559b58" exitCode=0 Oct 31 13:23:53.366666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:53.352330 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h" event={"ID":"bf08107d-640e-4a43-8994-10e51fa506ec","Type":"ContainerDied","Data":"feeadd17b8e77eee7c3a65c1db256e3a68c1e2e9078aff202cb69a2a72559b58"} Oct 31 13:23:53.366666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:53.352385 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h" event={"ID":"bf08107d-640e-4a43-8994-10e51fa506ec","Type":"ContainerDied","Data":"38ae2c034abc08e872a010dc55f86a7875364b74cb259a45694f4375ac26ac64"} Oct 31 13:23:53.366666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:53.352416 5693 scope.go:122] "RemoveContainer" containerID="feeadd17b8e77eee7c3a65c1db256e3a68c1e2e9078aff202cb69a2a72559b58" Oct 31 13:23:53.366666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:53.352574 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h" Oct 31 13:23:53.431002 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:53.416321 5693 scope.go:122] "RemoveContainer" containerID="feeadd17b8e77eee7c3a65c1db256e3a68c1e2e9078aff202cb69a2a72559b58" Oct 31 13:23:53.431002 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:23:53.423157 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feeadd17b8e77eee7c3a65c1db256e3a68c1e2e9078aff202cb69a2a72559b58\": container with ID starting with feeadd17b8e77eee7c3a65c1db256e3a68c1e2e9078aff202cb69a2a72559b58 not found: ID does not exist" containerID="feeadd17b8e77eee7c3a65c1db256e3a68c1e2e9078aff202cb69a2a72559b58" Oct 31 13:23:53.431002 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:53.423196 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feeadd17b8e77eee7c3a65c1db256e3a68c1e2e9078aff202cb69a2a72559b58"} err="failed to get container status \"feeadd17b8e77eee7c3a65c1db256e3a68c1e2e9078aff202cb69a2a72559b58\": rpc error: code = NotFound desc = could not find container \"feeadd17b8e77eee7c3a65c1db256e3a68c1e2e9078aff202cb69a2a72559b58\": container with ID starting with feeadd17b8e77eee7c3a65c1db256e3a68c1e2e9078aff202cb69a2a72559b58 not found: ID does not exist" Oct 31 13:23:53.431002 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:53.423557 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e06e892-03d5-4cf7-8858-0e7a2d9adfbc" path="/var/lib/kubelet/pods/0e06e892-03d5-4cf7-8858-0e7a2d9adfbc/volumes" Oct 31 13:23:53.431002 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:53.424060 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h"] Oct 31 13:23:53.443055 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:53.443018 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-7d98bmk2h"] Oct 31 13:23:55.412027 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:55.411997 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf08107d-640e-4a43-8994-10e51fa506ec" path="/var/lib/kubelet/pods/bf08107d-640e-4a43-8994-10e51fa506ec/volumes" Oct 31 13:23:58.704771 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:58.704736 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5dbd39-5c6f7bcf67-d5m99"] Oct 31 13:23:58.707092 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:58.705630 5693 cpu_manager.go:414] "containerMap: removing container" podUID="0e06e892-03d5-4cf7-8858-0e7a2d9adfbc" containerName="payload-processing" Oct 31 13:23:58.707092 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:58.705657 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e06e892-03d5-4cf7-8858-0e7a2d9adfbc" containerName="payload-processing" Oct 31 13:23:58.707092 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:58.705674 5693 cpu_manager.go:414] "containerMap: removing container" podUID="bf08107d-640e-4a43-8994-10e51fa506ec" containerName="payload-processing" Oct 31 13:23:58.707092 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:58.705679 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf08107d-640e-4a43-8994-10e51fa506ec" containerName="payload-processing" Oct 31 13:23:58.707092 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:58.705805 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf08107d-640e-4a43-8994-10e51fa506ec" containerName="payload-processing" Oct 31 13:23:58.707092 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:58.705825 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e06e892-03d5-4cf7-8858-0e7a2d9adfbc" containerName="payload-processing" Oct 31 13:23:58.709800 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:58.709777 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5dbd39-5c6f7bcf67-d5m99" Oct 31 13:23:58.728460 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:58.728427 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-5dbd39\"" Oct 31 13:23:58.729851 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:58.729833 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-5dbd39-dockercfg-jdf4w\"" Oct 31 13:23:58.730282 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:58.730262 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5dbd39-5c6f7bcf67-d5m99"] Oct 31 13:23:58.746827 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:58.744259 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5dbd39-6b6f6d9559-9tbjq"] Oct 31 13:23:58.750991 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:58.750965 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5dbd39-6b6f6d9559-9tbjq" Oct 31 13:23:58.762003 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:58.761980 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5dbd39-6b6f6d9559-9tbjq"] Oct 31 13:23:58.783663 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:58.781615 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rqs9\" (UniqueName: \"kubernetes.io/projected/5cd6c998-975c-4628-a886-e9eeb20b3a64-kube-api-access-9rqs9\") pod \"payload-pre-processing-e2e-inf-a-5dbd39-5c6f7bcf67-d5m99\" (UID: \"5cd6c998-975c-4628-a886-e9eeb20b3a64\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5dbd39-5c6f7bcf67-d5m99" Oct 31 13:23:58.783663 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:58.781682 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cd6c998-975c-4628-a886-e9eeb20b3a64-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5dbd39-5c6f7bcf67-d5m99\" (UID: \"5cd6c998-975c-4628-a886-e9eeb20b3a64\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5dbd39-5c6f7bcf67-d5m99" Oct 31 13:23:58.900667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:58.900206 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rqs9\" (UniqueName: \"kubernetes.io/projected/5cd6c998-975c-4628-a886-e9eeb20b3a64-kube-api-access-9rqs9\") pod \"payload-pre-processing-e2e-inf-a-5dbd39-5c6f7bcf67-d5m99\" (UID: \"5cd6c998-975c-4628-a886-e9eeb20b3a64\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5dbd39-5c6f7bcf67-d5m99" Oct 31 13:23:58.900667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:58.900264 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cd6c998-975c-4628-a886-e9eeb20b3a64-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5dbd39-5c6f7bcf67-d5m99\" (UID: \"5cd6c998-975c-4628-a886-e9eeb20b3a64\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5dbd39-5c6f7bcf67-d5m99" Oct 31 13:23:58.900667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:58.900480 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g7mw\" (UniqueName: \"kubernetes.io/projected/b8730031-f623-4f9e-ba5f-014340976d85-kube-api-access-5g7mw\") pod \"payload-processing-e2e-inf-a-5dbd39-6b6f6d9559-9tbjq\" (UID: \"b8730031-f623-4f9e-ba5f-014340976d85\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5dbd39-6b6f6d9559-9tbjq" Oct 31 13:23:58.900667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:58.900520 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8730031-f623-4f9e-ba5f-014340976d85-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5dbd39-6b6f6d9559-9tbjq\" (UID: \"b8730031-f623-4f9e-ba5f-014340976d85\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5dbd39-6b6f6d9559-9tbjq" Oct 31 13:23:58.917663 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:58.905931 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cd6c998-975c-4628-a886-e9eeb20b3a64-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5dbd39-5c6f7bcf67-d5m99\" (UID: \"5cd6c998-975c-4628-a886-e9eeb20b3a64\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5dbd39-5c6f7bcf67-d5m99" Oct 31 13:23:58.975716 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:58.974516 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rqs9\" (UniqueName: \"kubernetes.io/projected/5cd6c998-975c-4628-a886-e9eeb20b3a64-kube-api-access-9rqs9\") pod \"payload-pre-processing-e2e-inf-a-5dbd39-5c6f7bcf67-d5m99\" (UID: \"5cd6c998-975c-4628-a886-e9eeb20b3a64\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5dbd39-5c6f7bcf67-d5m99" Oct 31 13:23:59.008670 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:59.008183 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g7mw\" (UniqueName: \"kubernetes.io/projected/b8730031-f623-4f9e-ba5f-014340976d85-kube-api-access-5g7mw\") pod \"payload-processing-e2e-inf-a-5dbd39-6b6f6d9559-9tbjq\" (UID: \"b8730031-f623-4f9e-ba5f-014340976d85\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5dbd39-6b6f6d9559-9tbjq" Oct 31 13:23:59.008670 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:59.008243 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8730031-f623-4f9e-ba5f-014340976d85-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5dbd39-6b6f6d9559-9tbjq\" (UID: \"b8730031-f623-4f9e-ba5f-014340976d85\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5dbd39-6b6f6d9559-9tbjq" Oct 31 13:23:59.032925 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:59.009291 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8730031-f623-4f9e-ba5f-014340976d85-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5dbd39-6b6f6d9559-9tbjq\" (UID: \"b8730031-f623-4f9e-ba5f-014340976d85\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5dbd39-6b6f6d9559-9tbjq" Oct 31 13:23:59.032925 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:59.031012 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5dbd39-5c6f7bcf67-d5m99" Oct 31 13:23:59.070791 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:59.067765 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g7mw\" (UniqueName: \"kubernetes.io/projected/b8730031-f623-4f9e-ba5f-014340976d85-kube-api-access-5g7mw\") pod \"payload-processing-e2e-inf-a-5dbd39-6b6f6d9559-9tbjq\" (UID: \"b8730031-f623-4f9e-ba5f-014340976d85\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5dbd39-6b6f6d9559-9tbjq" Oct 31 13:23:59.084195 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:59.083882 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5dbd39-6b6f6d9559-9tbjq" Oct 31 13:23:59.240337 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:59.240257 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4023ad-7994b685f7-9m2jh" Oct 31 13:23:59.375849 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:59.375816 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5dbd39-5c6f7bcf67-d5m99"] Oct 31 13:23:59.400540 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:59.400512 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5dbd39-6b6f6d9559-9tbjq"] Oct 31 13:23:59.400856 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:23:59.400828 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8730031_f623_4f9e_ba5f_014340976d85.slice/crio-eaa09af39ea26ea011f162f3c3c4949a67b5491047b421e9f096640a43bfcfc6 WatchSource:0}: Error finding container eaa09af39ea26ea011f162f3c3c4949a67b5491047b421e9f096640a43bfcfc6: Status 404 returned error can't find the container with id eaa09af39ea26ea011f162f3c3c4949a67b5491047b421e9f096640a43bfcfc6 Oct 31 13:23:59.434347 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:59.434320 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5dbd39-5c6f7bcf67-d5m99" event={"ID":"5cd6c998-975c-4628-a886-e9eeb20b3a64","Type":"ContainerStarted","Data":"0077d370a4ef0714c9c060d086ad1bc916b39a461591b5f841b1b5f2c76c660d"} Oct 31 13:23:59.435760 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:23:59.435735 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5dbd39-6b6f6d9559-9tbjq" event={"ID":"b8730031-f623-4f9e-ba5f-014340976d85","Type":"ContainerStarted","Data":"eaa09af39ea26ea011f162f3c3c4949a67b5491047b421e9f096640a43bfcfc6"} Oct 31 13:24:00.446293 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:00.446261 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5dbd39-5c6f7bcf67-d5m99" event={"ID":"5cd6c998-975c-4628-a886-e9eeb20b3a64","Type":"ContainerStarted","Data":"49f0dc0eefa54aab9f5ecedd4715d6b9de50e2f38f03c4f1ce27d5f5149129c2"} Oct 31 13:24:00.446765 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:00.446337 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5dbd39-5c6f7bcf67-d5m99" Oct 31 13:24:00.448256 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:00.448228 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5dbd39-6b6f6d9559-9tbjq" event={"ID":"b8730031-f623-4f9e-ba5f-014340976d85","Type":"ContainerStarted","Data":"a16f39cf357902316f9b559f733bf6a67d98e9d8cee3ef73b846cc9b7c2fd486"} Oct 31 13:24:00.448463 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:00.448440 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5dbd39-6b6f6d9559-9tbjq" Oct 31 13:24:00.470841 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:00.470792 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5dbd39-5c6f7bcf67-d5m99" podStartSLOduration=2.47077939 podStartE2EDuration="2.47077939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:23:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:00.466310573 +0000 UTC m=+3091.173896869" watchObservedRunningTime="2026-10-31 13:24:00.47077939 +0000 UTC m=+3091.178365685" Oct 31 13:24:00.490973 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:00.490894 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-5dbd39-6b6f6d9559-9tbjq" podStartSLOduration=2.490874865 podStartE2EDuration="2.490874865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:23:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:00.485909735 +0000 UTC m=+3091.193496033" watchObservedRunningTime="2026-10-31 13:24:00.490874865 +0000 UTC m=+3091.198461161" Oct 31 13:24:11.481441 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:11.481407 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5dbd39-5c6f7bcf67-d5m99" Oct 31 13:24:11.482630 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:11.482601 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5dbd39-6b6f6d9559-9tbjq" Oct 31 13:24:14.605128 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:14.605097 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q"] Oct 31 13:24:14.610593 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:14.610571 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" Oct 31 13:24:14.634153 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:14.634121 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q"] Oct 31 13:24:14.752902 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:14.752823 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e899259-b760-4d69-9576-aca3d72409c8-workload-certs\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q\" (UID: \"9e899259-b760-4d69-9576-aca3d72409c8\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" Oct 31 13:24:14.753110 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:14.752970 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48l74\" (UniqueName: \"kubernetes.io/projected/9e899259-b760-4d69-9576-aca3d72409c8-kube-api-access-48l74\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q\" (UID: \"9e899259-b760-4d69-9576-aca3d72409c8\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" Oct 31 13:24:14.753110 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:14.753077 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e899259-b760-4d69-9576-aca3d72409c8-istio-token\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q\" (UID: \"9e899259-b760-4d69-9576-aca3d72409c8\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" Oct 31 13:24:14.753250 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:14.753207 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e899259-b760-4d69-9576-aca3d72409c8-istio-data\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q\" (UID: \"9e899259-b760-4d69-9576-aca3d72409c8\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" Oct 31 13:24:14.753922 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:14.753335 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e899259-b760-4d69-9576-aca3d72409c8-istio-podinfo\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q\" (UID: \"9e899259-b760-4d69-9576-aca3d72409c8\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" Oct 31 13:24:14.753922 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:14.753445 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e899259-b760-4d69-9576-aca3d72409c8-workload-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q\" (UID: \"9e899259-b760-4d69-9576-aca3d72409c8\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" Oct 31 13:24:14.753922 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:14.753555 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e899259-b760-4d69-9576-aca3d72409c8-istiod-ca-cert\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q\" (UID: \"9e899259-b760-4d69-9576-aca3d72409c8\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" Oct 31 13:24:14.753922 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:14.753591 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e899259-b760-4d69-9576-aca3d72409c8-credential-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q\" (UID: \"9e899259-b760-4d69-9576-aca3d72409c8\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" Oct 31 13:24:14.753922 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:14.753666 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e899259-b760-4d69-9576-aca3d72409c8-istio-envoy\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q\" (UID: \"9e899259-b760-4d69-9576-aca3d72409c8\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" Oct 31 13:24:14.855584 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:14.855506 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e899259-b760-4d69-9576-aca3d72409c8-istiod-ca-cert\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q\" (UID: \"9e899259-b760-4d69-9576-aca3d72409c8\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" Oct 31 13:24:14.855584 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:14.855548 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e899259-b760-4d69-9576-aca3d72409c8-credential-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q\" (UID: \"9e899259-b760-4d69-9576-aca3d72409c8\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" Oct 31 13:24:14.855854 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:14.855585 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e899259-b760-4d69-9576-aca3d72409c8-istio-envoy\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q\" (UID: \"9e899259-b760-4d69-9576-aca3d72409c8\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" Oct 31 13:24:14.855854 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:14.855700 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e899259-b760-4d69-9576-aca3d72409c8-workload-certs\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q\" (UID: \"9e899259-b760-4d69-9576-aca3d72409c8\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" Oct 31 13:24:14.855854 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:14.855727 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48l74\" (UniqueName: \"kubernetes.io/projected/9e899259-b760-4d69-9576-aca3d72409c8-kube-api-access-48l74\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q\" (UID: \"9e899259-b760-4d69-9576-aca3d72409c8\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" Oct 31 13:24:14.855854 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:14.855801 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e899259-b760-4d69-9576-aca3d72409c8-istio-token\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q\" (UID: \"9e899259-b760-4d69-9576-aca3d72409c8\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" Oct 31 13:24:14.856064 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:14.855907 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e899259-b760-4d69-9576-aca3d72409c8-istio-data\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q\" (UID: \"9e899259-b760-4d69-9576-aca3d72409c8\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" Oct 31 13:24:14.856064 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:14.856018 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e899259-b760-4d69-9576-aca3d72409c8-istio-podinfo\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q\" (UID: \"9e899259-b760-4d69-9576-aca3d72409c8\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" Oct 31 13:24:14.856064 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:14.856046 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e899259-b760-4d69-9576-aca3d72409c8-workload-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q\" (UID: \"9e899259-b760-4d69-9576-aca3d72409c8\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" Oct 31 13:24:14.856219 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:14.856055 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e899259-b760-4d69-9576-aca3d72409c8-workload-certs\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q\" (UID: \"9e899259-b760-4d69-9576-aca3d72409c8\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" Oct 31 13:24:14.856219 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:14.856090 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e899259-b760-4d69-9576-aca3d72409c8-credential-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q\" (UID: \"9e899259-b760-4d69-9576-aca3d72409c8\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" Oct 31 13:24:14.856307 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:14.856260 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e899259-b760-4d69-9576-aca3d72409c8-istiod-ca-cert\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q\" (UID: \"9e899259-b760-4d69-9576-aca3d72409c8\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" Oct 31 13:24:14.856355 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:14.856324 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e899259-b760-4d69-9576-aca3d72409c8-istio-data\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q\" (UID: \"9e899259-b760-4d69-9576-aca3d72409c8\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" Oct 31 13:24:14.856406 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:14.856367 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e899259-b760-4d69-9576-aca3d72409c8-workload-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q\" (UID: \"9e899259-b760-4d69-9576-aca3d72409c8\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" Oct 31 13:24:14.858266 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:14.858242 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e899259-b760-4d69-9576-aca3d72409c8-istio-envoy\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q\" (UID: \"9e899259-b760-4d69-9576-aca3d72409c8\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" Oct 31 13:24:14.858694 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:14.858675 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e899259-b760-4d69-9576-aca3d72409c8-istio-podinfo\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q\" (UID: \"9e899259-b760-4d69-9576-aca3d72409c8\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" Oct 31 13:24:14.876850 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:14.876824 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e899259-b760-4d69-9576-aca3d72409c8-istio-token\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q\" (UID: \"9e899259-b760-4d69-9576-aca3d72409c8\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" Oct 31 13:24:14.876958 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:14.876919 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48l74\" (UniqueName: \"kubernetes.io/projected/9e899259-b760-4d69-9576-aca3d72409c8-kube-api-access-48l74\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q\" (UID: \"9e899259-b760-4d69-9576-aca3d72409c8\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" Oct 31 13:24:14.942374 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:14.942339 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" Oct 31 13:24:15.237940 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:15.237912 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q"] Oct 31 13:24:15.238050 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:24:15.237911 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e899259_b760_4d69_9576_aca3d72409c8.slice/crio-5246b83dc927ddd0aa68a2b5139cf12a4460640f5524d964dc312a707d74dcae WatchSource:0}: Error finding container 5246b83dc927ddd0aa68a2b5139cf12a4460640f5524d964dc312a707d74dcae: Status 404 returned error can't find the container with id 5246b83dc927ddd0aa68a2b5139cf12a4460640f5524d964dc312a707d74dcae Oct 31 13:24:15.241079 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:15.241042 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:24:15.241179 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:15.241109 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:24:15.241179 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:15.241140 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:24:15.737876 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:15.737837 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" event={"ID":"9e899259-b760-4d69-9576-aca3d72409c8","Type":"ContainerStarted","Data":"4a598835f59fe61969c5a52abe888aca1a182fb5397dc051c271da43d46bc52f"} Oct 31 13:24:15.738394 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:15.737902 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" event={"ID":"9e899259-b760-4d69-9576-aca3d72409c8","Type":"ContainerStarted","Data":"5246b83dc927ddd0aa68a2b5139cf12a4460640f5524d964dc312a707d74dcae"} Oct 31 13:24:15.942465 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:15.942389 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" Oct 31 13:24:16.945358 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:16.945320 5693 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Oct 31 13:24:16.945814 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:16.945383 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" podUID="9e899259-b760-4d69-9576-aca3d72409c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Oct 31 13:24:17.942819 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:17.942783 5693 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Oct 31 13:24:17.943018 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:17.942851 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" podUID="9e899259-b760-4d69-9576-aca3d72409c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Oct 31 13:24:18.724219 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:24:18.724183 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:24:18.942664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:18.942610 5693 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Oct 31 13:24:18.942826 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:18.942683 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" podUID="9e899259-b760-4d69-9576-aca3d72409c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Oct 31 13:24:19.942825 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:19.942787 5693 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Oct 31 13:24:19.943257 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:19.942853 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" podUID="9e899259-b760-4d69-9576-aca3d72409c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Oct 31 13:24:20.950816 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:20.946114 5693 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Oct 31 13:24:20.950816 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:20.946189 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" podUID="9e899259-b760-4d69-9576-aca3d72409c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Oct 31 13:24:22.047943 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:22.047905 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" Oct 31 13:24:22.048858 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:22.048835 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" Oct 31 13:24:22.049880 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:22.049857 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" Oct 31 13:24:22.077501 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:22.077425 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" podStartSLOduration=8.077407572 podStartE2EDuration="8.077407572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:15.788600824 +0000 UTC m=+3106.496187140" watchObservedRunningTime="2026-10-31 13:24:22.077407572 +0000 UTC m=+3112.784993868" Oct 31 13:24:29.620321 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:29.620286 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc"] Oct 31 13:24:29.625469 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:29.625449 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" Oct 31 13:24:29.644778 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:29.644749 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc"] Oct 31 13:24:29.655801 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:29.655774 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-istio-data\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc\" (UID: \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" Oct 31 13:24:29.656165 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:29.656131 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9w2l\" (UniqueName: \"kubernetes.io/projected/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-kube-api-access-q9w2l\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc\" (UID: \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" Oct 31 13:24:29.656323 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:29.656183 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-istio-token\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc\" (UID: \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" Oct 31 13:24:29.656323 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:29.656267 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-workload-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc\" (UID: \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" Oct 31 13:24:29.656490 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:29.656467 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-istio-podinfo\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc\" (UID: \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" Oct 31 13:24:29.656716 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:29.656693 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-istiod-ca-cert\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc\" (UID: \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" Oct 31 13:24:29.656982 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:29.656956 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-istio-envoy\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc\" (UID: \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" Oct 31 13:24:29.657098 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:29.657004 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-workload-certs\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc\" (UID: \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" Oct 31 13:24:29.657098 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:29.657040 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-credential-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc\" (UID: \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" Oct 31 13:24:29.758628 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:29.758596 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-istio-data\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc\" (UID: \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" Oct 31 13:24:29.758862 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:29.758712 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9w2l\" (UniqueName: \"kubernetes.io/projected/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-kube-api-access-q9w2l\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc\" (UID: \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" Oct 31 13:24:29.758862 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:29.758735 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-istio-token\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc\" (UID: \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" Oct 31 13:24:29.758862 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:29.758790 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-workload-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc\" (UID: \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" Oct 31 13:24:29.759054 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:29.758945 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-istio-podinfo\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc\" (UID: \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" Oct 31 13:24:29.759054 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:29.758999 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-istiod-ca-cert\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc\" (UID: \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" Oct 31 13:24:29.759054 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:29.759028 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-istio-data\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc\" (UID: \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" Oct 31 13:24:29.759222 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:29.759122 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-istio-envoy\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc\" (UID: \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" Oct 31 13:24:29.759222 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:29.759150 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-workload-certs\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc\" (UID: \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" Oct 31 13:24:29.759222 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:29.759175 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-credential-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc\" (UID: \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" Oct 31 13:24:29.759404 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:29.759356 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-workload-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc\" (UID: \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" Oct 31 13:24:29.759677 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:29.759609 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-workload-certs\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc\" (UID: \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" Oct 31 13:24:29.759810 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:29.759746 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-istiod-ca-cert\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc\" (UID: \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" Oct 31 13:24:29.759912 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:29.759881 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-credential-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc\" (UID: \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" Oct 31 13:24:29.761934 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:29.761888 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-istio-podinfo\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc\" (UID: \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" Oct 31 13:24:29.762037 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:29.761951 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-istio-envoy\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc\" (UID: \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" Oct 31 13:24:29.775569 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:29.775549 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9w2l\" (UniqueName: \"kubernetes.io/projected/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-kube-api-access-q9w2l\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc\" (UID: \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" Oct 31 13:24:29.776417 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:29.776397 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-istio-token\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc\" (UID: \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" Oct 31 13:24:29.948715 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:29.948688 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" Oct 31 13:24:30.252658 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:30.252608 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc"] Oct 31 13:24:30.253321 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:24:30.253294 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb054e25_5c0a_4525_a337_1a29d0c7fe0a.slice/crio-68363ae29167356f88c9a86cdf7b4e38a9aacf54be26a3aefb275bd30693f6d9 WatchSource:0}: Error finding container 68363ae29167356f88c9a86cdf7b4e38a9aacf54be26a3aefb275bd30693f6d9: Status 404 returned error can't find the container with id 68363ae29167356f88c9a86cdf7b4e38a9aacf54be26a3aefb275bd30693f6d9 Oct 31 13:24:30.257417 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:30.257376 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:24:30.257520 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:30.257467 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:24:30.257520 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:30.257513 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:24:31.047391 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:31.047355 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" event={"ID":"fb054e25-5c0a-4525-a337-1a29d0c7fe0a","Type":"ContainerStarted","Data":"cc2e3b6f760cafba62d9f44c96c764de06c0ee2e24e15495df493952d98aa2de"} Oct 31 13:24:31.047836 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:31.047411 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" event={"ID":"fb054e25-5c0a-4525-a337-1a29d0c7fe0a","Type":"ContainerStarted","Data":"68363ae29167356f88c9a86cdf7b4e38a9aacf54be26a3aefb275bd30693f6d9"} Oct 31 13:24:31.138625 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:31.138481 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" podStartSLOduration=2.138458104 podStartE2EDuration="2.138458104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:31.088443089 +0000 UTC m=+3121.796029385" watchObservedRunningTime="2026-10-31 13:24:31.138458104 +0000 UTC m=+3121.846044400" Oct 31 13:24:31.951199 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:31.949816 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" Oct 31 13:24:31.952125 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:31.952100 5693 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body= Oct 31 13:24:31.952237 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:31.952145 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" podUID="fb054e25-5c0a-4525-a337-1a29d0c7fe0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" Oct 31 13:24:32.949118 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:32.949079 5693 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body= Oct 31 13:24:32.949611 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:32.949138 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" podUID="fb054e25-5c0a-4525-a337-1a29d0c7fe0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" Oct 31 13:24:33.949818 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:33.949778 5693 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body= Oct 31 13:24:33.950257 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:33.949836 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" podUID="fb054e25-5c0a-4525-a337-1a29d0c7fe0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" Oct 31 13:24:34.950744 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:34.950501 5693 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body= Oct 31 13:24:34.950744 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:34.950561 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" podUID="fb054e25-5c0a-4525-a337-1a29d0c7fe0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" Oct 31 13:24:35.949620 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:35.949589 5693 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body= Oct 31 13:24:35.949835 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:35.949673 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" podUID="fb054e25-5c0a-4525-a337-1a29d0c7fe0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" Oct 31 13:24:36.954182 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:36.954154 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" Oct 31 13:24:36.954671 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:36.954620 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" Oct 31 13:24:36.955461 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:36.955428 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" Oct 31 13:24:48.717387 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:24:48.717351 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:24:54.034600 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:54.034555 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-384dbe-57cd8fcd7-rsx6p"] Oct 31 13:24:54.042859 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:54.042826 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-384dbe-57cd8fcd7-rsx6p" Oct 31 13:24:54.051718 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:54.051519 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-384dbe\"" Oct 31 13:24:54.063109 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:54.063079 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-384dbe-dockercfg-c874z\"" Oct 31 13:24:54.075073 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:54.065303 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-384dbe-57cd8fcd7-rsx6p"] Oct 31 13:24:54.138051 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:54.138020 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-384dbe-6f9d5d675-qgdfx"] Oct 31 13:24:54.254665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:54.249298 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69zlk\" (UniqueName: \"kubernetes.io/projected/8903924e-49a1-456e-a26d-052f3a5901d8-kube-api-access-69zlk\") pod \"payload-pre-processing-e2e-inf-b-384dbe-57cd8fcd7-rsx6p\" (UID: \"8903924e-49a1-456e-a26d-052f3a5901d8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-384dbe-57cd8fcd7-rsx6p" Oct 31 13:24:54.254665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:54.249600 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8903924e-49a1-456e-a26d-052f3a5901d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-384dbe-57cd8fcd7-rsx6p\" (UID: \"8903924e-49a1-456e-a26d-052f3a5901d8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-384dbe-57cd8fcd7-rsx6p" Oct 31 13:24:54.409006 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:54.408969 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-384dbe-6f9d5d675-qgdfx"] Oct 31 13:24:54.460667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:54.419802 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-384dbe-6f9d5d675-qgdfx" Oct 31 13:24:54.530554 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:54.475332 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69zlk\" (UniqueName: \"kubernetes.io/projected/8903924e-49a1-456e-a26d-052f3a5901d8-kube-api-access-69zlk\") pod \"payload-pre-processing-e2e-inf-b-384dbe-57cd8fcd7-rsx6p\" (UID: \"8903924e-49a1-456e-a26d-052f3a5901d8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-384dbe-57cd8fcd7-rsx6p" Oct 31 13:24:54.530554 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:54.509870 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8903924e-49a1-456e-a26d-052f3a5901d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-384dbe-57cd8fcd7-rsx6p\" (UID: \"8903924e-49a1-456e-a26d-052f3a5901d8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-384dbe-57cd8fcd7-rsx6p" Oct 31 13:24:54.530554 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:54.514517 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8903924e-49a1-456e-a26d-052f3a5901d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-384dbe-57cd8fcd7-rsx6p\" (UID: \"8903924e-49a1-456e-a26d-052f3a5901d8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-384dbe-57cd8fcd7-rsx6p" Oct 31 13:24:54.542439 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:54.542407 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69zlk\" (UniqueName: \"kubernetes.io/projected/8903924e-49a1-456e-a26d-052f3a5901d8-kube-api-access-69zlk\") pod \"payload-pre-processing-e2e-inf-b-384dbe-57cd8fcd7-rsx6p\" (UID: \"8903924e-49a1-456e-a26d-052f3a5901d8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-384dbe-57cd8fcd7-rsx6p" Oct 31 13:24:54.611759 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:54.611721 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8db7l\" (UniqueName: \"kubernetes.io/projected/dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8-kube-api-access-8db7l\") pod \"payload-processing-e2e-inf-b-384dbe-6f9d5d675-qgdfx\" (UID: \"dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-384dbe-6f9d5d675-qgdfx" Oct 31 13:24:54.612012 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:54.611987 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-384dbe-6f9d5d675-qgdfx\" (UID: \"dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-384dbe-6f9d5d675-qgdfx" Oct 31 13:24:54.713563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:54.713460 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-384dbe-6f9d5d675-qgdfx\" (UID: \"dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-384dbe-6f9d5d675-qgdfx" Oct 31 13:24:54.713787 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:54.713769 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8db7l\" (UniqueName: \"kubernetes.io/projected/dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8-kube-api-access-8db7l\") pod \"payload-processing-e2e-inf-b-384dbe-6f9d5d675-qgdfx\" (UID: \"dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-384dbe-6f9d5d675-qgdfx" Oct 31 13:24:54.714407 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:54.714386 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-384dbe-6f9d5d675-qgdfx\" (UID: \"dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-384dbe-6f9d5d675-qgdfx" Oct 31 13:24:54.736974 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:54.736898 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8db7l\" (UniqueName: \"kubernetes.io/projected/dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8-kube-api-access-8db7l\") pod \"payload-processing-e2e-inf-b-384dbe-6f9d5d675-qgdfx\" (UID: \"dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-384dbe-6f9d5d675-qgdfx" Oct 31 13:24:54.738685 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:54.738632 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-384dbe-57cd8fcd7-rsx6p" Oct 31 13:24:54.831384 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:54.831028 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-384dbe-6f9d5d675-qgdfx" Oct 31 13:24:55.062866 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:55.062832 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-384dbe-57cd8fcd7-rsx6p"] Oct 31 13:24:55.063433 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:24:55.063406 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8903924e_49a1_456e_a26d_052f3a5901d8.slice/crio-8eceab6b1c750da53a9dd242958c5f2b36149747523fb64743255a1a9001c380 WatchSource:0}: Error finding container 8eceab6b1c750da53a9dd242958c5f2b36149747523fb64743255a1a9001c380: Status 404 returned error can't find the container with id 8eceab6b1c750da53a9dd242958c5f2b36149747523fb64743255a1a9001c380 Oct 31 13:24:55.155766 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:24:55.155718 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc81d1ea_3772_4bbe_9ec1_c28bae8e13a8.slice/crio-2ea141dcf7d7202e581a8a0058dc4397def3ab8b221457a840a3e38f115172f0 WatchSource:0}: Error finding container 2ea141dcf7d7202e581a8a0058dc4397def3ab8b221457a840a3e38f115172f0: Status 404 returned error can't find the container with id 2ea141dcf7d7202e581a8a0058dc4397def3ab8b221457a840a3e38f115172f0 Oct 31 13:24:55.155900 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:55.155849 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-384dbe-6f9d5d675-qgdfx"] Oct 31 13:24:55.535529 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:55.535495 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-384dbe-6f9d5d675-qgdfx" event={"ID":"dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8","Type":"ContainerStarted","Data":"7ec67c3e102d4ea3aee7ee6f26dc18293867defc4089592582d204c331331530"} Oct 31 13:24:55.535759 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:55.535554 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-384dbe-6f9d5d675-qgdfx" event={"ID":"dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8","Type":"ContainerStarted","Data":"2ea141dcf7d7202e581a8a0058dc4397def3ab8b221457a840a3e38f115172f0"} Oct 31 13:24:55.535759 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:55.535583 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-384dbe-6f9d5d675-qgdfx" Oct 31 13:24:55.537438 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:55.537411 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-384dbe-57cd8fcd7-rsx6p" event={"ID":"8903924e-49a1-456e-a26d-052f3a5901d8","Type":"ContainerStarted","Data":"62ec82ee6807326506d56993dfc66e8f49c4f27cb476a47c486067965784b1f4"} Oct 31 13:24:55.537555 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:55.537457 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-384dbe-57cd8fcd7-rsx6p" event={"ID":"8903924e-49a1-456e-a26d-052f3a5901d8","Type":"ContainerStarted","Data":"8eceab6b1c750da53a9dd242958c5f2b36149747523fb64743255a1a9001c380"} Oct 31 13:24:55.537555 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:55.537489 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-384dbe-57cd8fcd7-rsx6p" Oct 31 13:24:55.560804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:55.560753 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-384dbe-6f9d5d675-qgdfx" podStartSLOduration=1.560738808 podStartE2EDuration="1.560738808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:55.558491221 +0000 UTC m=+3146.266077516" watchObservedRunningTime="2026-10-31 13:24:55.560738808 +0000 UTC m=+3146.268325103" Oct 31 13:24:55.584871 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:24:55.584815 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-384dbe-57cd8fcd7-rsx6p" podStartSLOduration=1.584798523 podStartE2EDuration="1.584798523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:55.582907593 +0000 UTC m=+3146.290493901" watchObservedRunningTime="2026-10-31 13:24:55.584798523 +0000 UTC m=+3146.292384820" Oct 31 13:25:00.059808 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:00.059765 5693 patch_prober.go:29] interesting pod/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:25:00.060247 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:00.059843 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" podUID="8c08b4ba-3066-4b09-ba62-de15be6ccea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:25:02.603958 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:02.603876 5693 generic.go:356] "Generic (PLEG): container finished" podID="8c08b4ba-3066-4b09-ba62-de15be6ccea2" containerID="37714a9885d5013797267376d3e7f865f72904ea538f01a35e946d9fbb52bb90" exitCode=0 Oct 31 13:25:02.604677 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:02.603959 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" event={"ID":"8c08b4ba-3066-4b09-ba62-de15be6ccea2","Type":"ContainerDied","Data":"37714a9885d5013797267376d3e7f865f72904ea538f01a35e946d9fbb52bb90"} Oct 31 13:25:02.604677 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:02.604387 5693 scope.go:122] "RemoveContainer" containerID="37714a9885d5013797267376d3e7f865f72904ea538f01a35e946d9fbb52bb90" Oct 31 13:25:02.606518 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:02.606490 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:25:02.606596 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:02.606569 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:25:02.606667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:02.606615 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:25:03.615322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:03.615288 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" event={"ID":"8c08b4ba-3066-4b09-ba62-de15be6ccea2","Type":"ContainerStarted","Data":"d5054a2c9a14b7b56af8fc1d7e745c8668669c6ce26c003f089c92e9136021cb"} Oct 31 13:25:04.007229 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:04.007167 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" Oct 31 13:25:04.007598 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:04.007575 5693 patch_prober.go:29] interesting pod/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Oct 31 13:25:04.007682 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:04.007624 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" podUID="8c08b4ba-3066-4b09-ba62-de15be6ccea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Oct 31 13:25:05.008097 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:05.008059 5693 patch_prober.go:29] interesting pod/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Oct 31 13:25:05.008547 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:05.008129 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" podUID="8c08b4ba-3066-4b09-ba62-de15be6ccea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Oct 31 13:25:06.008148 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:06.008104 5693 patch_prober.go:29] interesting pod/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Oct 31 13:25:06.008690 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:06.008169 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" podUID="8c08b4ba-3066-4b09-ba62-de15be6ccea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Oct 31 13:25:06.572329 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:06.572292 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-384dbe-57cd8fcd7-rsx6p" Oct 31 13:25:06.574352 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:06.574318 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-384dbe-6f9d5d675-qgdfx" Oct 31 13:25:07.007603 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:07.007569 5693 patch_prober.go:29] interesting pod/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Oct 31 13:25:07.007884 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:07.007624 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" podUID="8c08b4ba-3066-4b09-ba62-de15be6ccea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Oct 31 13:25:07.430178 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:07.429935 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb"] Oct 31 13:25:07.434884 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:07.434802 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb"] Oct 31 13:25:07.435357 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:07.434921 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb" Oct 31 13:25:07.440821 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:07.440771 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-c6c445-serving-cert\"" Oct 31 13:25:07.514634 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:07.514597 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57wpr\" (UniqueName: \"kubernetes.io/projected/a67d0a96-c7c6-416a-9569-bf8905dc6c0c-kube-api-access-57wpr\") pod \"maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb\" (UID: \"a67d0a96-c7c6-416a-9569-bf8905dc6c0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb" Oct 31 13:25:07.514833 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:07.514773 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a67d0a96-c7c6-416a-9569-bf8905dc6c0c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb\" (UID: \"a67d0a96-c7c6-416a-9569-bf8905dc6c0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb" Oct 31 13:25:07.514833 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:07.514822 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a67d0a96-c7c6-416a-9569-bf8905dc6c0c-maas-api-tls\") pod \"maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb\" (UID: \"a67d0a96-c7c6-416a-9569-bf8905dc6c0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb" Oct 31 13:25:07.514915 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:07.514877 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a67d0a96-c7c6-416a-9569-bf8905dc6c0c-metrics-certs\") pod \"maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb\" (UID: \"a67d0a96-c7c6-416a-9569-bf8905dc6c0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb" Oct 31 13:25:07.626676 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:07.623555 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57wpr\" (UniqueName: \"kubernetes.io/projected/a67d0a96-c7c6-416a-9569-bf8905dc6c0c-kube-api-access-57wpr\") pod \"maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb\" (UID: \"a67d0a96-c7c6-416a-9569-bf8905dc6c0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb" Oct 31 13:25:07.626676 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:07.623750 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a67d0a96-c7c6-416a-9569-bf8905dc6c0c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb\" (UID: \"a67d0a96-c7c6-416a-9569-bf8905dc6c0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb" Oct 31 13:25:07.640852 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:07.640781 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a67d0a96-c7c6-416a-9569-bf8905dc6c0c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb\" (UID: \"a67d0a96-c7c6-416a-9569-bf8905dc6c0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb" Oct 31 13:25:07.641427 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:07.641373 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a67d0a96-c7c6-416a-9569-bf8905dc6c0c-maas-api-tls\") pod \"maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb\" (UID: \"a67d0a96-c7c6-416a-9569-bf8905dc6c0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb" Oct 31 13:25:07.641879 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:07.641626 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a67d0a96-c7c6-416a-9569-bf8905dc6c0c-metrics-certs\") pod \"maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb\" (UID: \"a67d0a96-c7c6-416a-9569-bf8905dc6c0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb" Oct 31 13:25:07.647700 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:07.647443 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a67d0a96-c7c6-416a-9569-bf8905dc6c0c-metrics-certs\") pod \"maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb\" (UID: \"a67d0a96-c7c6-416a-9569-bf8905dc6c0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb" Oct 31 13:25:07.665300 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:07.665243 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a67d0a96-c7c6-416a-9569-bf8905dc6c0c-maas-api-tls\") pod \"maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb\" (UID: \"a67d0a96-c7c6-416a-9569-bf8905dc6c0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb" Oct 31 13:25:07.666331 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:07.666287 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57wpr\" (UniqueName: \"kubernetes.io/projected/a67d0a96-c7c6-416a-9569-bf8905dc6c0c-kube-api-access-57wpr\") pod \"maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb\" (UID: \"a67d0a96-c7c6-416a-9569-bf8905dc6c0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb" Oct 31 13:25:07.791704 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:07.790898 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb" Oct 31 13:25:08.008078 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:08.008040 5693 patch_prober.go:29] interesting pod/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Oct 31 13:25:08.008250 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:08.008107 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" podUID="8c08b4ba-3066-4b09-ba62-de15be6ccea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Oct 31 13:25:08.471573 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:08.471513 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb"] Oct 31 13:25:08.473335 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:08.472380 5693 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 13:25:08.729722 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:08.724947 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb" event={"ID":"a67d0a96-c7c6-416a-9569-bf8905dc6c0c","Type":"ContainerStarted","Data":"2bec6739e06f82a17c388d794419a3052b53cf9a8bc5a6c62e3e492f2eb49a34"} Oct 31 13:25:09.033080 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:09.031860 5693 patch_prober.go:29] interesting pod/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Oct 31 13:25:09.033080 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:09.031921 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" podUID="8c08b4ba-3066-4b09-ba62-de15be6ccea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Oct 31 13:25:09.735152 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:09.735115 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb" event={"ID":"a67d0a96-c7c6-416a-9569-bf8905dc6c0c","Type":"ContainerStarted","Data":"6d5fd54a2c0f63e68f79da2779f0679c63995c4315c543211f82c317a2ee9e42"} Oct 31 13:25:09.735632 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:09.735177 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb" Oct 31 13:25:10.012333 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:10.012252 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" Oct 31 13:25:10.012804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:10.012788 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" Oct 31 13:25:10.013587 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:10.013569 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" Oct 31 13:25:10.040780 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:10.040726 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb" podStartSLOduration=2.553976965 podStartE2EDuration="3.040711404s" totalImagesPullingTime="486.734439ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:25:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:25:09.76103837 +0000 UTC m=+3160.468624667" watchObservedRunningTime="2026-10-31 13:25:10.040711404 +0000 UTC m=+3160.748297747" Oct 31 13:25:15.776954 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:15.776926 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb" Oct 31 13:25:18.096632 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:18.096594 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-384dbe-674f58d889-lqz29"] Oct 31 13:25:18.105186 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:18.105154 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-384dbe-674f58d889-lqz29" Oct 31 13:25:18.113681 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:18.113634 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-384dbe-serving-cert\"" Oct 31 13:25:18.120254 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:18.120229 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-384dbe-674f58d889-lqz29"] Oct 31 13:25:18.262519 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:18.262481 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50a97ab4-3037-46d1-9cc5-68cc95955e9f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-384dbe-674f58d889-lqz29\" (UID: \"50a97ab4-3037-46d1-9cc5-68cc95955e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-384dbe-674f58d889-lqz29" Oct 31 13:25:18.264825 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:18.264793 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50a97ab4-3037-46d1-9cc5-68cc95955e9f-metrics-certs\") pod \"maas-api-e2e-inf-b-384dbe-674f58d889-lqz29\" (UID: \"50a97ab4-3037-46d1-9cc5-68cc95955e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-384dbe-674f58d889-lqz29" Oct 31 13:25:18.279665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:18.276924 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50a97ab4-3037-46d1-9cc5-68cc95955e9f-maas-api-tls\") pod \"maas-api-e2e-inf-b-384dbe-674f58d889-lqz29\" (UID: \"50a97ab4-3037-46d1-9cc5-68cc95955e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-384dbe-674f58d889-lqz29" Oct 31 13:25:18.279665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:18.277275 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7fdr\" (UniqueName: \"kubernetes.io/projected/50a97ab4-3037-46d1-9cc5-68cc95955e9f-kube-api-access-d7fdr\") pod \"maas-api-e2e-inf-b-384dbe-674f58d889-lqz29\" (UID: \"50a97ab4-3037-46d1-9cc5-68cc95955e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-384dbe-674f58d889-lqz29" Oct 31 13:25:18.378628 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:18.378542 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7fdr\" (UniqueName: \"kubernetes.io/projected/50a97ab4-3037-46d1-9cc5-68cc95955e9f-kube-api-access-d7fdr\") pod \"maas-api-e2e-inf-b-384dbe-674f58d889-lqz29\" (UID: \"50a97ab4-3037-46d1-9cc5-68cc95955e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-384dbe-674f58d889-lqz29" Oct 31 13:25:18.378845 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:18.378699 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50a97ab4-3037-46d1-9cc5-68cc95955e9f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-384dbe-674f58d889-lqz29\" (UID: \"50a97ab4-3037-46d1-9cc5-68cc95955e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-384dbe-674f58d889-lqz29" Oct 31 13:25:18.378845 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:18.378791 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50a97ab4-3037-46d1-9cc5-68cc95955e9f-metrics-certs\") pod \"maas-api-e2e-inf-b-384dbe-674f58d889-lqz29\" (UID: \"50a97ab4-3037-46d1-9cc5-68cc95955e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-384dbe-674f58d889-lqz29" Oct 31 13:25:18.378985 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:18.378925 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50a97ab4-3037-46d1-9cc5-68cc95955e9f-maas-api-tls\") pod \"maas-api-e2e-inf-b-384dbe-674f58d889-lqz29\" (UID: \"50a97ab4-3037-46d1-9cc5-68cc95955e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-384dbe-674f58d889-lqz29" Oct 31 13:25:18.379363 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:18.379338 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50a97ab4-3037-46d1-9cc5-68cc95955e9f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-384dbe-674f58d889-lqz29\" (UID: \"50a97ab4-3037-46d1-9cc5-68cc95955e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-384dbe-674f58d889-lqz29" Oct 31 13:25:18.381465 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:18.381447 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50a97ab4-3037-46d1-9cc5-68cc95955e9f-maas-api-tls\") pod \"maas-api-e2e-inf-b-384dbe-674f58d889-lqz29\" (UID: \"50a97ab4-3037-46d1-9cc5-68cc95955e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-384dbe-674f58d889-lqz29" Oct 31 13:25:18.381624 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:18.381600 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50a97ab4-3037-46d1-9cc5-68cc95955e9f-metrics-certs\") pod \"maas-api-e2e-inf-b-384dbe-674f58d889-lqz29\" (UID: \"50a97ab4-3037-46d1-9cc5-68cc95955e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-384dbe-674f58d889-lqz29" Oct 31 13:25:18.391107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:18.391085 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7fdr\" (UniqueName: \"kubernetes.io/projected/50a97ab4-3037-46d1-9cc5-68cc95955e9f-kube-api-access-d7fdr\") pod \"maas-api-e2e-inf-b-384dbe-674f58d889-lqz29\" (UID: \"50a97ab4-3037-46d1-9cc5-68cc95955e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-384dbe-674f58d889-lqz29" Oct 31 13:25:18.442311 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:18.442272 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-384dbe-674f58d889-lqz29" Oct 31 13:25:18.810114 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:25:18.810068 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:25:18.846099 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:18.846061 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-384dbe-674f58d889-lqz29"] Oct 31 13:25:18.854706 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:25:18.854676 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50a97ab4_3037_46d1_9cc5_68cc95955e9f.slice/crio-e9182135b45e4560d716381ff5b1743b2a4e249d1e2bee18ad96c1acae16a549 WatchSource:0}: Error finding container e9182135b45e4560d716381ff5b1743b2a4e249d1e2bee18ad96c1acae16a549: Status 404 returned error can't find the container with id e9182135b45e4560d716381ff5b1743b2a4e249d1e2bee18ad96c1acae16a549 Oct 31 13:25:19.858099 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:19.858059 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-384dbe-674f58d889-lqz29" event={"ID":"50a97ab4-3037-46d1-9cc5-68cc95955e9f","Type":"ContainerStarted","Data":"11ac8f935bfc0e56753855e6a58d7e252cf97258f9d2efb524524a2fe0fd7bb1"} Oct 31 13:25:19.858493 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:19.858116 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-384dbe-674f58d889-lqz29" event={"ID":"50a97ab4-3037-46d1-9cc5-68cc95955e9f","Type":"ContainerStarted","Data":"e9182135b45e4560d716381ff5b1743b2a4e249d1e2bee18ad96c1acae16a549"} Oct 31 13:25:19.858493 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:19.858189 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-384dbe-674f58d889-lqz29" Oct 31 13:25:19.891093 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:19.891036 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-384dbe-674f58d889-lqz29" podStartSLOduration=1.362704322 podStartE2EDuration="1.89102088s" totalImagesPullingTime="528.316558ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:25:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:25:19.888129428 +0000 UTC m=+3170.595715735" watchObservedRunningTime="2026-10-31 13:25:19.89102088 +0000 UTC m=+3170.598607225" Oct 31 13:25:25.893072 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:25.893044 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-384dbe-674f58d889-lqz29" Oct 31 13:25:28.038723 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:28.038687 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb"] Oct 31 13:25:28.039177 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:28.038890 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" podUID="0109302a-8951-450d-a280-98f4ab4fd093" containerName="istio-proxy" containerID="cri-o://acf7dbe173f51d07eeea94b98dc33b9a2eb190503f877531ddab9c3c296a85e2" gracePeriod=30 Oct 31 13:25:33.339876 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:33.339848 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" Oct 31 13:25:33.466402 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:33.466366 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0109302a-8951-450d-a280-98f4ab4fd093-kube-api-access-4pk9j\" (UniqueName: \"kubernetes.io/projected/0109302a-8951-450d-a280-98f4ab4fd093-kube-api-access-4pk9j\") pod \"0109302a-8951-450d-a280-98f4ab4fd093\" (UID: \"0109302a-8951-450d-a280-98f4ab4fd093\") " Oct 31 13:25:33.466608 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:33.466425 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0109302a-8951-450d-a280-98f4ab4fd093-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0109302a-8951-450d-a280-98f4ab4fd093-istiod-ca-cert\") pod \"0109302a-8951-450d-a280-98f4ab4fd093\" (UID: \"0109302a-8951-450d-a280-98f4ab4fd093\") " Oct 31 13:25:33.466608 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:33.466572 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0109302a-8951-450d-a280-98f4ab4fd093-istio-token\" (UniqueName: \"kubernetes.io/projected/0109302a-8951-450d-a280-98f4ab4fd093-istio-token\") pod \"0109302a-8951-450d-a280-98f4ab4fd093\" (UID: \"0109302a-8951-450d-a280-98f4ab4fd093\") " Oct 31 13:25:33.466751 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:33.466620 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0109302a-8951-450d-a280-98f4ab4fd093-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0109302a-8951-450d-a280-98f4ab4fd093-credential-socket\") pod \"0109302a-8951-450d-a280-98f4ab4fd093\" (UID: \"0109302a-8951-450d-a280-98f4ab4fd093\") " Oct 31 13:25:33.466751 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:33.466707 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0109302a-8951-450d-a280-98f4ab4fd093-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0109302a-8951-450d-a280-98f4ab4fd093-workload-certs\") pod \"0109302a-8951-450d-a280-98f4ab4fd093\" (UID: \"0109302a-8951-450d-a280-98f4ab4fd093\") " Oct 31 13:25:33.466899 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:33.466876 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0109302a-8951-450d-a280-98f4ab4fd093-credential-socket" pod "0109302a-8951-450d-a280-98f4ab4fd093" (UID: "0109302a-8951-450d-a280-98f4ab4fd093"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:25:33.466979 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:33.466876 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0109302a-8951-450d-a280-98f4ab4fd093-istiod-ca-cert" pod "0109302a-8951-450d-a280-98f4ab4fd093" (UID: "0109302a-8951-450d-a280-98f4ab4fd093"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:25:33.467042 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:33.466976 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0109302a-8951-450d-a280-98f4ab4fd093-workload-certs" pod "0109302a-8951-450d-a280-98f4ab4fd093" (UID: "0109302a-8951-450d-a280-98f4ab4fd093"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:25:33.467088 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:33.467048 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0109302a-8951-450d-a280-98f4ab4fd093-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0109302a-8951-450d-a280-98f4ab4fd093-istio-envoy\") pod \"0109302a-8951-450d-a280-98f4ab4fd093\" (UID: \"0109302a-8951-450d-a280-98f4ab4fd093\") " Oct 31 13:25:33.467088 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:33.467081 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0109302a-8951-450d-a280-98f4ab4fd093-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0109302a-8951-450d-a280-98f4ab4fd093-workload-socket\") pod \"0109302a-8951-450d-a280-98f4ab4fd093\" (UID: \"0109302a-8951-450d-a280-98f4ab4fd093\") " Oct 31 13:25:33.467194 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:33.467166 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0109302a-8951-450d-a280-98f4ab4fd093-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0109302a-8951-450d-a280-98f4ab4fd093-istio-podinfo\") pod \"0109302a-8951-450d-a280-98f4ab4fd093\" (UID: \"0109302a-8951-450d-a280-98f4ab4fd093\") " Oct 31 13:25:33.467248 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:33.467233 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0109302a-8951-450d-a280-98f4ab4fd093-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0109302a-8951-450d-a280-98f4ab4fd093-istio-data\") pod \"0109302a-8951-450d-a280-98f4ab4fd093\" (UID: \"0109302a-8951-450d-a280-98f4ab4fd093\") " Oct 31 13:25:33.467456 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:33.467430 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0109302a-8951-450d-a280-98f4ab4fd093-workload-socket" pod "0109302a-8951-450d-a280-98f4ab4fd093" (UID: "0109302a-8951-450d-a280-98f4ab4fd093"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:25:33.467724 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:33.467699 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0109302a-8951-450d-a280-98f4ab4fd093-istio-data" pod "0109302a-8951-450d-a280-98f4ab4fd093" (UID: "0109302a-8951-450d-a280-98f4ab4fd093"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:25:33.468118 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:33.468064 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0109302a-8951-450d-a280-98f4ab4fd093-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:25:33.468118 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:33.468088 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0109302a-8951-450d-a280-98f4ab4fd093-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:25:33.468118 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:33.468099 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0109302a-8951-450d-a280-98f4ab4fd093-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:25:33.468118 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:33.468108 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0109302a-8951-450d-a280-98f4ab4fd093-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:25:33.468118 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:33.468117 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0109302a-8951-450d-a280-98f4ab4fd093-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:25:33.469954 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:33.469927 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0109302a-8951-450d-a280-98f4ab4fd093-istio-envoy" pod "0109302a-8951-450d-a280-98f4ab4fd093" (UID: "0109302a-8951-450d-a280-98f4ab4fd093"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:25:33.470343 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:33.470321 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0109302a-8951-450d-a280-98f4ab4fd093-istio-token" pod "0109302a-8951-450d-a280-98f4ab4fd093" (UID: "0109302a-8951-450d-a280-98f4ab4fd093"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:25:33.470343 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:33.470334 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0109302a-8951-450d-a280-98f4ab4fd093-istio-podinfo" pod "0109302a-8951-450d-a280-98f4ab4fd093" (UID: "0109302a-8951-450d-a280-98f4ab4fd093"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:25:33.470456 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:33.470357 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0109302a-8951-450d-a280-98f4ab4fd093-kube-api-access-4pk9j" pod "0109302a-8951-450d-a280-98f4ab4fd093" (UID: "0109302a-8951-450d-a280-98f4ab4fd093"). InnerVolumeSpecName "kube-api-access-4pk9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:25:33.576966 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:33.570065 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pk9j\" (UniqueName: \"kubernetes.io/projected/0109302a-8951-450d-a280-98f4ab4fd093-kube-api-access-4pk9j\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:25:33.576966 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:33.570110 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0109302a-8951-450d-a280-98f4ab4fd093-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:25:33.576966 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:33.570125 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0109302a-8951-450d-a280-98f4ab4fd093-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:25:33.576966 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:33.570140 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0109302a-8951-450d-a280-98f4ab4fd093-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:25:34.022127 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:34.022093 5693 generic.go:356] "Generic (PLEG): container finished" podID="0109302a-8951-450d-a280-98f4ab4fd093" containerID="acf7dbe173f51d07eeea94b98dc33b9a2eb190503f877531ddab9c3c296a85e2" exitCode=0 Oct 31 13:25:34.022335 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:34.022150 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" Oct 31 13:25:34.022335 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:34.022167 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" event={"ID":"0109302a-8951-450d-a280-98f4ab4fd093","Type":"ContainerDied","Data":"acf7dbe173f51d07eeea94b98dc33b9a2eb190503f877531ddab9c3c296a85e2"} Oct 31 13:25:34.022335 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:34.022207 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb" event={"ID":"0109302a-8951-450d-a280-98f4ab4fd093","Type":"ContainerDied","Data":"e3eeff3bd202759aa63e639fccc3f979423d5d7c031b22ad8d80f9ebe0db33ad"} Oct 31 13:25:34.022335 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:34.022229 5693 scope.go:122] "RemoveContainer" containerID="acf7dbe173f51d07eeea94b98dc33b9a2eb190503f877531ddab9c3c296a85e2" Oct 31 13:25:34.036807 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:34.036785 5693 scope.go:122] "RemoveContainer" containerID="acf7dbe173f51d07eeea94b98dc33b9a2eb190503f877531ddab9c3c296a85e2" Oct 31 13:25:34.037115 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:25:34.037088 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acf7dbe173f51d07eeea94b98dc33b9a2eb190503f877531ddab9c3c296a85e2\": container with ID starting with acf7dbe173f51d07eeea94b98dc33b9a2eb190503f877531ddab9c3c296a85e2 not found: ID does not exist" containerID="acf7dbe173f51d07eeea94b98dc33b9a2eb190503f877531ddab9c3c296a85e2" Oct 31 13:25:34.037231 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:34.037127 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acf7dbe173f51d07eeea94b98dc33b9a2eb190503f877531ddab9c3c296a85e2"} err="failed to get container status \"acf7dbe173f51d07eeea94b98dc33b9a2eb190503f877531ddab9c3c296a85e2\": rpc error: code = NotFound desc = could not find container \"acf7dbe173f51d07eeea94b98dc33b9a2eb190503f877531ddab9c3c296a85e2\": container with ID starting with acf7dbe173f51d07eeea94b98dc33b9a2eb190503f877531ddab9c3c296a85e2 not found: ID does not exist" Oct 31 13:25:34.054595 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:34.054560 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb"] Oct 31 13:25:34.062238 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:34.062212 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-c2ffb"] Oct 31 13:25:34.819560 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:34.819529 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7"] Oct 31 13:25:35.412930 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:35.412896 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0109302a-8951-450d-a280-98f4ab4fd093" path="/var/lib/kubelet/pods/0109302a-8951-450d-a280-98f4ab4fd093/volumes" Oct 31 13:25:39.820150 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:39.820091 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7" podUID="5bee661a-1cac-4b73-9d3b-05cf50e47813" containerName="payload-processing" containerID="cri-o://3c110a86502eea8e0cdfef98a2b53bb1d048e3fd9d1e187e37c6372eeff472da" gracePeriod=25 Oct 31 13:25:40.170019 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:40.169990 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7" Oct 31 13:25:40.174902 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:40.174876 5693 generic.go:356] "Generic (PLEG): container finished" podID="5bee661a-1cac-4b73-9d3b-05cf50e47813" containerID="3c110a86502eea8e0cdfef98a2b53bb1d048e3fd9d1e187e37c6372eeff472da" exitCode=0 Oct 31 13:25:40.175051 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:40.174918 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7" Oct 31 13:25:40.175051 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:40.174958 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7" event={"ID":"5bee661a-1cac-4b73-9d3b-05cf50e47813","Type":"ContainerDied","Data":"3c110a86502eea8e0cdfef98a2b53bb1d048e3fd9d1e187e37c6372eeff472da"} Oct 31 13:25:40.175051 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:40.175011 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7" event={"ID":"5bee661a-1cac-4b73-9d3b-05cf50e47813","Type":"ContainerDied","Data":"7afb3e8b323745ad0f70ea17ac8df42aaba6988020b6644e5f99b853871fcb9b"} Oct 31 13:25:40.175051 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:40.175040 5693 scope.go:122] "RemoveContainer" containerID="3c110a86502eea8e0cdfef98a2b53bb1d048e3fd9d1e187e37c6372eeff472da" Oct 31 13:25:40.186224 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:40.186203 5693 scope.go:122] "RemoveContainer" containerID="3c110a86502eea8e0cdfef98a2b53bb1d048e3fd9d1e187e37c6372eeff472da" Oct 31 13:25:40.186487 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:25:40.186470 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c110a86502eea8e0cdfef98a2b53bb1d048e3fd9d1e187e37c6372eeff472da\": container with ID starting with 3c110a86502eea8e0cdfef98a2b53bb1d048e3fd9d1e187e37c6372eeff472da not found: ID does not exist" containerID="3c110a86502eea8e0cdfef98a2b53bb1d048e3fd9d1e187e37c6372eeff472da" Oct 31 13:25:40.186541 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:40.186498 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c110a86502eea8e0cdfef98a2b53bb1d048e3fd9d1e187e37c6372eeff472da"} err="failed to get container status \"3c110a86502eea8e0cdfef98a2b53bb1d048e3fd9d1e187e37c6372eeff472da\": rpc error: code = NotFound desc = could not find container \"3c110a86502eea8e0cdfef98a2b53bb1d048e3fd9d1e187e37c6372eeff472da\": container with ID starting with 3c110a86502eea8e0cdfef98a2b53bb1d048e3fd9d1e187e37c6372eeff472da not found: ID does not exist" Oct 31 13:25:40.259553 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:40.259523 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bee661a-1cac-4b73-9d3b-05cf50e47813-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5bee661a-1cac-4b73-9d3b-05cf50e47813-routing-overlay-volume\") pod \"5bee661a-1cac-4b73-9d3b-05cf50e47813\" (UID: \"5bee661a-1cac-4b73-9d3b-05cf50e47813\") " Oct 31 13:25:40.259799 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:40.259670 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bee661a-1cac-4b73-9d3b-05cf50e47813-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5bee661a-1cac-4b73-9d3b-05cf50e47813-provider-credentials\") pod \"5bee661a-1cac-4b73-9d3b-05cf50e47813\" (UID: \"5bee661a-1cac-4b73-9d3b-05cf50e47813\") " Oct 31 13:25:40.259889 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:40.259811 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bee661a-1cac-4b73-9d3b-05cf50e47813-kube-api-access-wc8xg\" (UniqueName: \"kubernetes.io/projected/5bee661a-1cac-4b73-9d3b-05cf50e47813-kube-api-access-wc8xg\") pod \"5bee661a-1cac-4b73-9d3b-05cf50e47813\" (UID: \"5bee661a-1cac-4b73-9d3b-05cf50e47813\") " Oct 31 13:25:40.259957 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:40.259910 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bee661a-1cac-4b73-9d3b-05cf50e47813-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bee661a-1cac-4b73-9d3b-05cf50e47813-plugins-config-volume\") pod \"5bee661a-1cac-4b73-9d3b-05cf50e47813\" (UID: \"5bee661a-1cac-4b73-9d3b-05cf50e47813\") " Oct 31 13:25:40.260014 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:40.259981 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bee661a-1cac-4b73-9d3b-05cf50e47813-routing-overlay-volume" pod "5bee661a-1cac-4b73-9d3b-05cf50e47813" (UID: "5bee661a-1cac-4b73-9d3b-05cf50e47813"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:25:40.260514 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:40.260492 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5bee661a-1cac-4b73-9d3b-05cf50e47813-routing-overlay-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:25:40.260701 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:40.260586 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bee661a-1cac-4b73-9d3b-05cf50e47813-plugins-config-volume" pod "5bee661a-1cac-4b73-9d3b-05cf50e47813" (UID: "5bee661a-1cac-4b73-9d3b-05cf50e47813"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:25:40.262304 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:40.262280 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bee661a-1cac-4b73-9d3b-05cf50e47813-provider-credentials" pod "5bee661a-1cac-4b73-9d3b-05cf50e47813" (UID: "5bee661a-1cac-4b73-9d3b-05cf50e47813"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:25:40.262467 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:40.262446 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bee661a-1cac-4b73-9d3b-05cf50e47813-kube-api-access-wc8xg" pod "5bee661a-1cac-4b73-9d3b-05cf50e47813" (UID: "5bee661a-1cac-4b73-9d3b-05cf50e47813"). InnerVolumeSpecName "kube-api-access-wc8xg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:25:40.362808 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:40.362764 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5bee661a-1cac-4b73-9d3b-05cf50e47813-provider-credentials\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:25:40.362989 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:40.362813 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc8xg\" (UniqueName: \"kubernetes.io/projected/5bee661a-1cac-4b73-9d3b-05cf50e47813-kube-api-access-wc8xg\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:25:40.362989 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:40.362830 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bee661a-1cac-4b73-9d3b-05cf50e47813-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:25:40.502523 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:40.502484 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7"] Oct 31 13:25:40.509108 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:40.509081 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-8458q7bm7"] Oct 31 13:25:41.411497 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:41.411466 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bee661a-1cac-4b73-9d3b-05cf50e47813" path="/var/lib/kubelet/pods/5bee661a-1cac-4b73-9d3b-05cf50e47813/volumes" Oct 31 13:25:45.201664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:45.198716 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7"] Oct 31 13:25:45.201664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:45.200307 5693 cpu_manager.go:414] "containerMap: removing container" podUID="0109302a-8951-450d-a280-98f4ab4fd093" containerName="istio-proxy" Oct 31 13:25:45.201664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:45.200330 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0109302a-8951-450d-a280-98f4ab4fd093" containerName="istio-proxy" Oct 31 13:25:45.201664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:45.200419 5693 cpu_manager.go:414] "containerMap: removing container" podUID="5bee661a-1cac-4b73-9d3b-05cf50e47813" containerName="payload-processing" Oct 31 13:25:45.201664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:45.200427 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bee661a-1cac-4b73-9d3b-05cf50e47813" containerName="payload-processing" Oct 31 13:25:45.201664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:45.200671 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bee661a-1cac-4b73-9d3b-05cf50e47813" containerName="payload-processing" Oct 31 13:25:45.201664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:45.200700 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="0109302a-8951-450d-a280-98f4ab4fd093" containerName="istio-proxy" Oct 31 13:25:45.264664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:45.243864 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7"] Oct 31 13:25:45.264664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:45.243987 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7" Oct 31 13:25:45.264664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:45.254042 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2ef686\"/\"kube-root-ca.crt\"" Oct 31 13:25:45.264664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:45.256120 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2ef686\"/\"payload-processing-external-model-plugins-e2e-extmod-a-2ef686\"" Oct 31 13:25:45.264664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:45.256170 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2ef686\"/\"agc-cred-e7ebb35f27c3\"" Oct 31 13:25:45.264664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:45.256208 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2ef686\"/\"routing-overlay\"" Oct 31 13:25:45.264664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:45.256266 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2ef686\"/\"openshift-service-ca.crt\"" Oct 31 13:25:45.264664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:45.256314 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2ef686\"/\"payload-processing-external-model-e2e--362fde4e-dockercfg-m5cqq\"" Oct 31 13:25:45.478288 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:45.478192 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48fa6091-7341-4168-83da-93f772457206-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7\" (UID: \"48fa6091-7341-4168-83da-93f772457206\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7" Oct 31 13:25:45.541848 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:45.541772 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48fa6091-7341-4168-83da-93f772457206-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7\" (UID: \"48fa6091-7341-4168-83da-93f772457206\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7" Oct 31 13:25:45.542071 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:45.541987 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/48fa6091-7341-4168-83da-93f772457206-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7\" (UID: \"48fa6091-7341-4168-83da-93f772457206\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7" Oct 31 13:25:45.542244 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:45.542216 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkrlg\" (UniqueName: \"kubernetes.io/projected/48fa6091-7341-4168-83da-93f772457206-kube-api-access-kkrlg\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7\" (UID: \"48fa6091-7341-4168-83da-93f772457206\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7" Oct 31 13:25:45.648627 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:45.646262 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkrlg\" (UniqueName: \"kubernetes.io/projected/48fa6091-7341-4168-83da-93f772457206-kube-api-access-kkrlg\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7\" (UID: \"48fa6091-7341-4168-83da-93f772457206\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7" Oct 31 13:25:45.648627 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:45.646538 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48fa6091-7341-4168-83da-93f772457206-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7\" (UID: \"48fa6091-7341-4168-83da-93f772457206\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7" Oct 31 13:25:45.648627 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:45.646588 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48fa6091-7341-4168-83da-93f772457206-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7\" (UID: \"48fa6091-7341-4168-83da-93f772457206\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7" Oct 31 13:25:45.648627 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:45.647228 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/48fa6091-7341-4168-83da-93f772457206-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7\" (UID: \"48fa6091-7341-4168-83da-93f772457206\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7" Oct 31 13:25:45.649057 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:45.648694 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48fa6091-7341-4168-83da-93f772457206-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7\" (UID: \"48fa6091-7341-4168-83da-93f772457206\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7" Oct 31 13:25:45.650525 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:45.650498 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48fa6091-7341-4168-83da-93f772457206-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7\" (UID: \"48fa6091-7341-4168-83da-93f772457206\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7" Oct 31 13:25:45.652591 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:45.652559 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/48fa6091-7341-4168-83da-93f772457206-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7\" (UID: \"48fa6091-7341-4168-83da-93f772457206\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7" Oct 31 13:25:45.662864 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:45.662838 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkrlg\" (UniqueName: \"kubernetes.io/projected/48fa6091-7341-4168-83da-93f772457206-kube-api-access-kkrlg\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7\" (UID: \"48fa6091-7341-4168-83da-93f772457206\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7" Oct 31 13:25:45.893838 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:45.893799 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7" Oct 31 13:25:46.270318 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:46.270093 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7"] Oct 31 13:25:47.428059 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:47.428022 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7" event={"ID":"48fa6091-7341-4168-83da-93f772457206","Type":"ContainerStarted","Data":"f0272db704c281abdbbecbfc522e345bc2a5cf1d7ef9dbeddc80c93fce3b31fd"} Oct 31 13:25:47.428635 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:47.428085 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7" event={"ID":"48fa6091-7341-4168-83da-93f772457206","Type":"ContainerStarted","Data":"e208b228577d06bfbd23b9ee863ce34aaad831ba592b1c025caaa7f88d68c00d"} Oct 31 13:25:47.428635 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:47.428140 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7" Oct 31 13:25:47.447667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:47.447590 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7" podStartSLOduration=2.447577006 podStartE2EDuration="2.447577006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:25:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:25:47.447521893 +0000 UTC m=+3198.155108188" watchObservedRunningTime="2026-10-31 13:25:47.447577006 +0000 UTC m=+3198.155163341" Oct 31 13:25:48.701917 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:25:48.701883 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:25:53.636730 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:53.636696 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-c6c445/test-model-c6c445-kserve-69b4dc6d69-nxx4n"] Oct 31 13:25:53.660031 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:53.659996 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c6c445/test-model-c6c445-kserve-69b4dc6d69-nxx4n"] Oct 31 13:25:53.660230 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:53.660107 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c6c445/test-model-c6c445-kserve-69b4dc6d69-nxx4n" Oct 31 13:25:53.669304 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:53.667346 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c6c445\"/\"openshift-service-ca.crt\"" Oct 31 13:25:53.669304 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:53.667450 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c6c445\"/\"test-model-c6c445-kserve-self-signed-certs\"" Oct 31 13:25:53.669304 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:53.667542 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c6c445\"/\"default-dockercfg-h6z52\"" Oct 31 13:25:53.669304 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:53.667699 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c6c445\"/\"kube-root-ca.crt\"" Oct 31 13:25:53.777101 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:53.777067 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/768af44a-2a4a-4254-a497-e2969f376cf7-tls-certs\") pod \"test-model-c6c445-kserve-69b4dc6d69-nxx4n\" (UID: \"768af44a-2a4a-4254-a497-e2969f376cf7\") " pod="ai-tenant-e2e-inf-b-c6c445/test-model-c6c445-kserve-69b4dc6d69-nxx4n" Oct 31 13:25:53.777316 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:53.777145 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/768af44a-2a4a-4254-a497-e2969f376cf7-dshm\") pod \"test-model-c6c445-kserve-69b4dc6d69-nxx4n\" (UID: \"768af44a-2a4a-4254-a497-e2969f376cf7\") " pod="ai-tenant-e2e-inf-b-c6c445/test-model-c6c445-kserve-69b4dc6d69-nxx4n" Oct 31 13:25:53.777316 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:53.777288 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/768af44a-2a4a-4254-a497-e2969f376cf7-tmp-dir\") pod \"test-model-c6c445-kserve-69b4dc6d69-nxx4n\" (UID: \"768af44a-2a4a-4254-a497-e2969f376cf7\") " pod="ai-tenant-e2e-inf-b-c6c445/test-model-c6c445-kserve-69b4dc6d69-nxx4n" Oct 31 13:25:53.777441 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:53.777355 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/768af44a-2a4a-4254-a497-e2969f376cf7-model-cache\") pod \"test-model-c6c445-kserve-69b4dc6d69-nxx4n\" (UID: \"768af44a-2a4a-4254-a497-e2969f376cf7\") " pod="ai-tenant-e2e-inf-b-c6c445/test-model-c6c445-kserve-69b4dc6d69-nxx4n" Oct 31 13:25:53.777497 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:53.777442 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/768af44a-2a4a-4254-a497-e2969f376cf7-home\") pod \"test-model-c6c445-kserve-69b4dc6d69-nxx4n\" (UID: \"768af44a-2a4a-4254-a497-e2969f376cf7\") " pod="ai-tenant-e2e-inf-b-c6c445/test-model-c6c445-kserve-69b4dc6d69-nxx4n" Oct 31 13:25:53.777497 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:53.777486 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46dtx\" (UniqueName: \"kubernetes.io/projected/768af44a-2a4a-4254-a497-e2969f376cf7-kube-api-access-46dtx\") pod \"test-model-c6c445-kserve-69b4dc6d69-nxx4n\" (UID: \"768af44a-2a4a-4254-a497-e2969f376cf7\") " pod="ai-tenant-e2e-inf-b-c6c445/test-model-c6c445-kserve-69b4dc6d69-nxx4n" Oct 31 13:25:53.879003 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:53.878963 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/768af44a-2a4a-4254-a497-e2969f376cf7-model-cache\") pod \"test-model-c6c445-kserve-69b4dc6d69-nxx4n\" (UID: \"768af44a-2a4a-4254-a497-e2969f376cf7\") " pod="ai-tenant-e2e-inf-b-c6c445/test-model-c6c445-kserve-69b4dc6d69-nxx4n" Oct 31 13:25:53.879204 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:53.879172 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/768af44a-2a4a-4254-a497-e2969f376cf7-home\") pod \"test-model-c6c445-kserve-69b4dc6d69-nxx4n\" (UID: \"768af44a-2a4a-4254-a497-e2969f376cf7\") " pod="ai-tenant-e2e-inf-b-c6c445/test-model-c6c445-kserve-69b4dc6d69-nxx4n" Oct 31 13:25:53.879339 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:53.879237 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46dtx\" (UniqueName: \"kubernetes.io/projected/768af44a-2a4a-4254-a497-e2969f376cf7-kube-api-access-46dtx\") pod \"test-model-c6c445-kserve-69b4dc6d69-nxx4n\" (UID: \"768af44a-2a4a-4254-a497-e2969f376cf7\") " pod="ai-tenant-e2e-inf-b-c6c445/test-model-c6c445-kserve-69b4dc6d69-nxx4n" Oct 31 13:25:53.879418 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:53.879365 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/768af44a-2a4a-4254-a497-e2969f376cf7-tls-certs\") pod \"test-model-c6c445-kserve-69b4dc6d69-nxx4n\" (UID: \"768af44a-2a4a-4254-a497-e2969f376cf7\") " pod="ai-tenant-e2e-inf-b-c6c445/test-model-c6c445-kserve-69b4dc6d69-nxx4n" Oct 31 13:25:53.879480 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:53.879424 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/768af44a-2a4a-4254-a497-e2969f376cf7-model-cache\") pod \"test-model-c6c445-kserve-69b4dc6d69-nxx4n\" (UID: \"768af44a-2a4a-4254-a497-e2969f376cf7\") " pod="ai-tenant-e2e-inf-b-c6c445/test-model-c6c445-kserve-69b4dc6d69-nxx4n" Oct 31 13:25:53.879480 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:53.879431 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/768af44a-2a4a-4254-a497-e2969f376cf7-dshm\") pod \"test-model-c6c445-kserve-69b4dc6d69-nxx4n\" (UID: \"768af44a-2a4a-4254-a497-e2969f376cf7\") " pod="ai-tenant-e2e-inf-b-c6c445/test-model-c6c445-kserve-69b4dc6d69-nxx4n" Oct 31 13:25:53.879628 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:53.879598 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/768af44a-2a4a-4254-a497-e2969f376cf7-tmp-dir\") pod \"test-model-c6c445-kserve-69b4dc6d69-nxx4n\" (UID: \"768af44a-2a4a-4254-a497-e2969f376cf7\") " pod="ai-tenant-e2e-inf-b-c6c445/test-model-c6c445-kserve-69b4dc6d69-nxx4n" Oct 31 13:25:53.879746 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:53.879721 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/768af44a-2a4a-4254-a497-e2969f376cf7-home\") pod \"test-model-c6c445-kserve-69b4dc6d69-nxx4n\" (UID: \"768af44a-2a4a-4254-a497-e2969f376cf7\") " pod="ai-tenant-e2e-inf-b-c6c445/test-model-c6c445-kserve-69b4dc6d69-nxx4n" Oct 31 13:25:53.879979 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:53.879958 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/768af44a-2a4a-4254-a497-e2969f376cf7-tmp-dir\") pod \"test-model-c6c445-kserve-69b4dc6d69-nxx4n\" (UID: \"768af44a-2a4a-4254-a497-e2969f376cf7\") " pod="ai-tenant-e2e-inf-b-c6c445/test-model-c6c445-kserve-69b4dc6d69-nxx4n" Oct 31 13:25:53.882447 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:53.882422 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/768af44a-2a4a-4254-a497-e2969f376cf7-dshm\") pod \"test-model-c6c445-kserve-69b4dc6d69-nxx4n\" (UID: \"768af44a-2a4a-4254-a497-e2969f376cf7\") " pod="ai-tenant-e2e-inf-b-c6c445/test-model-c6c445-kserve-69b4dc6d69-nxx4n" Oct 31 13:25:53.882846 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:53.882822 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/768af44a-2a4a-4254-a497-e2969f376cf7-tls-certs\") pod \"test-model-c6c445-kserve-69b4dc6d69-nxx4n\" (UID: \"768af44a-2a4a-4254-a497-e2969f376cf7\") " pod="ai-tenant-e2e-inf-b-c6c445/test-model-c6c445-kserve-69b4dc6d69-nxx4n" Oct 31 13:25:53.892524 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:53.892468 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46dtx\" (UniqueName: \"kubernetes.io/projected/768af44a-2a4a-4254-a497-e2969f376cf7-kube-api-access-46dtx\") pod \"test-model-c6c445-kserve-69b4dc6d69-nxx4n\" (UID: \"768af44a-2a4a-4254-a497-e2969f376cf7\") " pod="ai-tenant-e2e-inf-b-c6c445/test-model-c6c445-kserve-69b4dc6d69-nxx4n" Oct 31 13:25:54.000727 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:53.998103 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c6c445/test-model-c6c445-kserve-69b4dc6d69-nxx4n" Oct 31 13:25:54.400097 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:25:54.400062 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod768af44a_2a4a_4254_a497_e2969f376cf7.slice/crio-3f694af94b5133386e16811f4a7d2ce857e512a8c1a266e5ee2adaa19bdff45a WatchSource:0}: Error finding container 3f694af94b5133386e16811f4a7d2ce857e512a8c1a266e5ee2adaa19bdff45a: Status 404 returned error can't find the container with id 3f694af94b5133386e16811f4a7d2ce857e512a8c1a266e5ee2adaa19bdff45a Oct 31 13:25:54.400669 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:54.400612 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c6c445/test-model-c6c445-kserve-69b4dc6d69-nxx4n"] Oct 31 13:25:54.514820 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:54.514785 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c6c445/test-model-c6c445-kserve-69b4dc6d69-nxx4n" event={"ID":"768af44a-2a4a-4254-a497-e2969f376cf7","Type":"ContainerStarted","Data":"d0d1a27779f09a735fb98bffbaf9b0bd090b5c660136b3f9421f98598e5db960"} Oct 31 13:25:54.515083 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:54.514831 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c6c445/test-model-c6c445-kserve-69b4dc6d69-nxx4n" event={"ID":"768af44a-2a4a-4254-a497-e2969f376cf7","Type":"ContainerStarted","Data":"3f694af94b5133386e16811f4a7d2ce857e512a8c1a266e5ee2adaa19bdff45a"} Oct 31 13:25:54.557556 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:54.557497 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-c6c445/test-model-c6c445-kserve-69b4dc6d69-nxx4n" podStartSLOduration=1.557483341 podStartE2EDuration="1.557483341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:25:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:25:54.553735085 +0000 UTC m=+3205.261321405" watchObservedRunningTime="2026-10-31 13:25:54.557483341 +0000 UTC m=+3205.265069635" Oct 31 13:25:55.000856 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:55.000820 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c6c445/test-model-c6c445-kserve-69b4dc6d69-nxx4n" Oct 31 13:25:58.455712 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:25:58.440324 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7" Oct 31 13:26:04.000266 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:04.000231 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-c6c445/test-model-c6c445-kserve-69b4dc6d69-nxx4n" Oct 31 13:26:04.012891 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:04.012864 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-c6c445/test-model-c6c445-kserve-69b4dc6d69-nxx4n" Oct 31 13:26:04.641756 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:04.641730 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c6c445/test-model-c6c445-kserve-69b4dc6d69-nxx4n" Oct 31 13:26:07.556374 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:07.556337 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch"] Oct 31 13:26:07.572355 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:07.572321 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp"] Oct 31 13:26:12.556915 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:12.556877 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch" podUID="9940bb55-a0b2-4d49-80ab-7b369e67394d" containerName="payload-processing" containerID="cri-o://d24d714d00d1963229d86fac43065ae6b8b76d4f373b71db4f282cea2e3408e3" gracePeriod=25 Oct 31 13:26:12.572875 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:12.572846 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp" podUID="809353f9-0eec-4cdc-89f2-e82af19aecd7" containerName="payload-processing" containerID="cri-o://4dd47c144a029e0a5a7300d219428e1456b55aa52b68e13b4fa7c068f039d94f" gracePeriod=25 Oct 31 13:26:12.711414 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:12.711369 5693 generic.go:356] "Generic (PLEG): container finished" podID="809353f9-0eec-4cdc-89f2-e82af19aecd7" containerID="4dd47c144a029e0a5a7300d219428e1456b55aa52b68e13b4fa7c068f039d94f" exitCode=0 Oct 31 13:26:12.711601 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:12.711478 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp" event={"ID":"809353f9-0eec-4cdc-89f2-e82af19aecd7","Type":"ContainerDied","Data":"4dd47c144a029e0a5a7300d219428e1456b55aa52b68e13b4fa7c068f039d94f"} Oct 31 13:26:12.715213 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:12.715184 5693 generic.go:356] "Generic (PLEG): container finished" podID="9940bb55-a0b2-4d49-80ab-7b369e67394d" containerID="d24d714d00d1963229d86fac43065ae6b8b76d4f373b71db4f282cea2e3408e3" exitCode=0 Oct 31 13:26:12.715521 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:12.715459 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch" event={"ID":"9940bb55-a0b2-4d49-80ab-7b369e67394d","Type":"ContainerDied","Data":"d24d714d00d1963229d86fac43065ae6b8b76d4f373b71db4f282cea2e3408e3"} Oct 31 13:26:12.992340 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:12.992316 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch" Oct 31 13:26:12.996826 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:12.996801 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp" Oct 31 13:26:13.101044 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:13.100965 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/809353f9-0eec-4cdc-89f2-e82af19aecd7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/809353f9-0eec-4cdc-89f2-e82af19aecd7-provider-credentials\") pod \"809353f9-0eec-4cdc-89f2-e82af19aecd7\" (UID: \"809353f9-0eec-4cdc-89f2-e82af19aecd7\") " Oct 31 13:26:13.101044 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:13.101028 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9940bb55-a0b2-4d49-80ab-7b369e67394d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9940bb55-a0b2-4d49-80ab-7b369e67394d-routing-overlay-volume\") pod \"9940bb55-a0b2-4d49-80ab-7b369e67394d\" (UID: \"9940bb55-a0b2-4d49-80ab-7b369e67394d\") " Oct 31 13:26:13.101386 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:13.101115 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/809353f9-0eec-4cdc-89f2-e82af19aecd7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/809353f9-0eec-4cdc-89f2-e82af19aecd7-routing-overlay-volume\") pod \"809353f9-0eec-4cdc-89f2-e82af19aecd7\" (UID: \"809353f9-0eec-4cdc-89f2-e82af19aecd7\") " Oct 31 13:26:13.101386 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:13.101213 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9940bb55-a0b2-4d49-80ab-7b369e67394d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9940bb55-a0b2-4d49-80ab-7b369e67394d-plugins-config-volume\") pod \"9940bb55-a0b2-4d49-80ab-7b369e67394d\" (UID: \"9940bb55-a0b2-4d49-80ab-7b369e67394d\") " Oct 31 13:26:13.101386 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:13.101361 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9940bb55-a0b2-4d49-80ab-7b369e67394d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9940bb55-a0b2-4d49-80ab-7b369e67394d-provider-credentials\") pod \"9940bb55-a0b2-4d49-80ab-7b369e67394d\" (UID: \"9940bb55-a0b2-4d49-80ab-7b369e67394d\") " Oct 31 13:26:13.102269 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:13.101419 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/809353f9-0eec-4cdc-89f2-e82af19aecd7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/809353f9-0eec-4cdc-89f2-e82af19aecd7-plugins-config-volume\") pod \"809353f9-0eec-4cdc-89f2-e82af19aecd7\" (UID: \"809353f9-0eec-4cdc-89f2-e82af19aecd7\") " Oct 31 13:26:13.102269 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:13.101477 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9940bb55-a0b2-4d49-80ab-7b369e67394d-kube-api-access-4qkjc\" (UniqueName: \"kubernetes.io/projected/9940bb55-a0b2-4d49-80ab-7b369e67394d-kube-api-access-4qkjc\") pod \"9940bb55-a0b2-4d49-80ab-7b369e67394d\" (UID: \"9940bb55-a0b2-4d49-80ab-7b369e67394d\") " Oct 31 13:26:13.102269 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:13.101431 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9940bb55-a0b2-4d49-80ab-7b369e67394d-routing-overlay-volume" pod "9940bb55-a0b2-4d49-80ab-7b369e67394d" (UID: "9940bb55-a0b2-4d49-80ab-7b369e67394d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:26:13.102269 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:13.101524 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/809353f9-0eec-4cdc-89f2-e82af19aecd7-routing-overlay-volume" pod "809353f9-0eec-4cdc-89f2-e82af19aecd7" (UID: "809353f9-0eec-4cdc-89f2-e82af19aecd7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:26:13.102269 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:13.101585 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/809353f9-0eec-4cdc-89f2-e82af19aecd7-kube-api-access-l94sw\" (UniqueName: \"kubernetes.io/projected/809353f9-0eec-4cdc-89f2-e82af19aecd7-kube-api-access-l94sw\") pod \"809353f9-0eec-4cdc-89f2-e82af19aecd7\" (UID: \"809353f9-0eec-4cdc-89f2-e82af19aecd7\") " Oct 31 13:26:13.102269 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:13.101796 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9940bb55-a0b2-4d49-80ab-7b369e67394d-plugins-config-volume" pod "9940bb55-a0b2-4d49-80ab-7b369e67394d" (UID: "9940bb55-a0b2-4d49-80ab-7b369e67394d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:26:13.102269 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:13.101893 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/809353f9-0eec-4cdc-89f2-e82af19aecd7-plugins-config-volume" pod "809353f9-0eec-4cdc-89f2-e82af19aecd7" (UID: "809353f9-0eec-4cdc-89f2-e82af19aecd7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:26:13.102882 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:13.102456 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/809353f9-0eec-4cdc-89f2-e82af19aecd7-routing-overlay-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:13.102882 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:13.102473 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9940bb55-a0b2-4d49-80ab-7b369e67394d-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:13.102882 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:13.102482 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/809353f9-0eec-4cdc-89f2-e82af19aecd7-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:13.102882 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:13.102492 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9940bb55-a0b2-4d49-80ab-7b369e67394d-routing-overlay-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:13.104297 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:13.104273 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9940bb55-a0b2-4d49-80ab-7b369e67394d-provider-credentials" pod "9940bb55-a0b2-4d49-80ab-7b369e67394d" (UID: "9940bb55-a0b2-4d49-80ab-7b369e67394d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:26:13.104427 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:13.104408 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/809353f9-0eec-4cdc-89f2-e82af19aecd7-provider-credentials" pod "809353f9-0eec-4cdc-89f2-e82af19aecd7" (UID: "809353f9-0eec-4cdc-89f2-e82af19aecd7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:26:13.104484 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:13.104429 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9940bb55-a0b2-4d49-80ab-7b369e67394d-kube-api-access-4qkjc" pod "9940bb55-a0b2-4d49-80ab-7b369e67394d" (UID: "9940bb55-a0b2-4d49-80ab-7b369e67394d"). InnerVolumeSpecName "kube-api-access-4qkjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:26:13.104559 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:13.104543 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/809353f9-0eec-4cdc-89f2-e82af19aecd7-kube-api-access-l94sw" pod "809353f9-0eec-4cdc-89f2-e82af19aecd7" (UID: "809353f9-0eec-4cdc-89f2-e82af19aecd7"). InnerVolumeSpecName "kube-api-access-l94sw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:26:13.204618 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:13.204568 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9940bb55-a0b2-4d49-80ab-7b369e67394d-provider-credentials\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:13.204618 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:13.204597 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qkjc\" (UniqueName: \"kubernetes.io/projected/9940bb55-a0b2-4d49-80ab-7b369e67394d-kube-api-access-4qkjc\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:13.204618 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:13.204608 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l94sw\" (UniqueName: \"kubernetes.io/projected/809353f9-0eec-4cdc-89f2-e82af19aecd7-kube-api-access-l94sw\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:13.204618 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:13.204618 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/809353f9-0eec-4cdc-89f2-e82af19aecd7-provider-credentials\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:13.728107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:13.728025 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp" Oct 31 13:26:13.728536 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:13.728017 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp" event={"ID":"809353f9-0eec-4cdc-89f2-e82af19aecd7","Type":"ContainerDied","Data":"c3ed194bc66b2cbd9f8dd2e238e73f3ded2e48ba232ae64eb6e2c501b2003f83"} Oct 31 13:26:13.728536 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:13.728232 5693 scope.go:122] "RemoveContainer" containerID="4dd47c144a029e0a5a7300d219428e1456b55aa52b68e13b4fa7c068f039d94f" Oct 31 13:26:13.729776 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:13.729657 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch" Oct 31 13:26:13.729776 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:13.729734 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch" event={"ID":"9940bb55-a0b2-4d49-80ab-7b369e67394d","Type":"ContainerDied","Data":"ad42778a463a7dbc3a75f1155d987b93d3496dd4861934adef0429851c2974ab"} Oct 31 13:26:13.740208 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:13.740186 5693 scope.go:122] "RemoveContainer" containerID="d24d714d00d1963229d86fac43065ae6b8b76d4f373b71db4f282cea2e3408e3" Oct 31 13:26:13.758682 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:13.758656 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp"] Oct 31 13:26:13.771961 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:13.771928 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557bnggwp"] Oct 31 13:26:13.780202 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:13.780171 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch"] Oct 31 13:26:13.790895 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:13.790865 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-7c7bx2gch"] Oct 31 13:26:15.455671 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:15.453215 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="809353f9-0eec-4cdc-89f2-e82af19aecd7" path="/var/lib/kubelet/pods/809353f9-0eec-4cdc-89f2-e82af19aecd7/volumes" Oct 31 13:26:15.455671 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:15.453952 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9940bb55-a0b2-4d49-80ab-7b369e67394d" path="/var/lib/kubelet/pods/9940bb55-a0b2-4d49-80ab-7b369e67394d/volumes" Oct 31 13:26:18.693497 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:18.693456 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:26:41.690723 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:41.689702 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm"] Oct 31 13:26:41.691532 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:41.691498 5693 cpu_manager.go:414] "containerMap: removing container" podUID="9940bb55-a0b2-4d49-80ab-7b369e67394d" containerName="payload-processing" Oct 31 13:26:41.691748 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:41.691728 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9940bb55-a0b2-4d49-80ab-7b369e67394d" containerName="payload-processing" Oct 31 13:26:41.692000 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:41.691987 5693 cpu_manager.go:414] "containerMap: removing container" podUID="809353f9-0eec-4cdc-89f2-e82af19aecd7" containerName="payload-processing" Oct 31 13:26:41.692117 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:41.692106 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="809353f9-0eec-4cdc-89f2-e82af19aecd7" containerName="payload-processing" Oct 31 13:26:41.692588 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:41.692560 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="809353f9-0eec-4cdc-89f2-e82af19aecd7" containerName="payload-processing" Oct 31 13:26:41.692780 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:41.692767 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="9940bb55-a0b2-4d49-80ab-7b369e67394d" containerName="payload-processing" Oct 31 13:26:41.813077 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:41.807093 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm"] Oct 31 13:26:41.815873 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:41.815783 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm" Oct 31 13:26:41.967876 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:41.967788 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4534300-b54b-441b-a78d-d9577ee14a29-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm\" (UID: \"f4534300-b54b-441b-a78d-d9577ee14a29\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm" Oct 31 13:26:41.968083 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:41.968054 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4534300-b54b-441b-a78d-d9577ee14a29-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm\" (UID: \"f4534300-b54b-441b-a78d-d9577ee14a29\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm" Oct 31 13:26:41.968196 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:41.968181 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4534300-b54b-441b-a78d-d9577ee14a29-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm\" (UID: \"f4534300-b54b-441b-a78d-d9577ee14a29\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm" Oct 31 13:26:41.968247 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:41.968224 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjdq8\" (UniqueName: \"kubernetes.io/projected/f4534300-b54b-441b-a78d-d9577ee14a29-kube-api-access-wjdq8\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm\" (UID: \"f4534300-b54b-441b-a78d-d9577ee14a29\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm" Oct 31 13:26:42.046267 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:42.046234 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc"] Oct 31 13:26:42.051446 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:42.051422 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc" Oct 31 13:26:42.066898 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:42.064778 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc"] Oct 31 13:26:42.070463 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:42.070431 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4534300-b54b-441b-a78d-d9577ee14a29-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm\" (UID: \"f4534300-b54b-441b-a78d-d9577ee14a29\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm" Oct 31 13:26:42.070627 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:42.070606 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc\" (UID: \"397bdc0f-9466-4bf1-ab28-e31d0dd308d1\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc" Oct 31 13:26:42.070733 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:42.070717 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc\" (UID: \"397bdc0f-9466-4bf1-ab28-e31d0dd308d1\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc" Oct 31 13:26:42.070806 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:42.070774 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4534300-b54b-441b-a78d-d9577ee14a29-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm\" (UID: \"f4534300-b54b-441b-a78d-d9577ee14a29\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm" Oct 31 13:26:42.071044 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:42.071021 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98jgd\" (UniqueName: \"kubernetes.io/projected/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-kube-api-access-98jgd\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc\" (UID: \"397bdc0f-9466-4bf1-ab28-e31d0dd308d1\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc" Oct 31 13:26:42.071176 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:42.071080 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4534300-b54b-441b-a78d-d9577ee14a29-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm\" (UID: \"f4534300-b54b-441b-a78d-d9577ee14a29\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm" Oct 31 13:26:42.071176 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:42.071138 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjdq8\" (UniqueName: \"kubernetes.io/projected/f4534300-b54b-441b-a78d-d9577ee14a29-kube-api-access-wjdq8\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm\" (UID: \"f4534300-b54b-441b-a78d-d9577ee14a29\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm" Oct 31 13:26:42.071176 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:42.071166 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc\" (UID: \"397bdc0f-9466-4bf1-ab28-e31d0dd308d1\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc" Oct 31 13:26:42.077712 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:42.077683 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4534300-b54b-441b-a78d-d9577ee14a29-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm\" (UID: \"f4534300-b54b-441b-a78d-d9577ee14a29\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm" Oct 31 13:26:42.079317 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:42.079288 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4534300-b54b-441b-a78d-d9577ee14a29-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm\" (UID: \"f4534300-b54b-441b-a78d-d9577ee14a29\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm" Oct 31 13:26:42.079576 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:42.079552 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4534300-b54b-441b-a78d-d9577ee14a29-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm\" (UID: \"f4534300-b54b-441b-a78d-d9577ee14a29\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm" Oct 31 13:26:42.091420 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:42.091392 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjdq8\" (UniqueName: \"kubernetes.io/projected/f4534300-b54b-441b-a78d-d9577ee14a29-kube-api-access-wjdq8\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm\" (UID: \"f4534300-b54b-441b-a78d-d9577ee14a29\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm" Oct 31 13:26:42.145189 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:42.143980 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm" Oct 31 13:26:42.258332 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:42.251237 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc\" (UID: \"397bdc0f-9466-4bf1-ab28-e31d0dd308d1\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc" Oct 31 13:26:42.258332 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:42.251349 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc\" (UID: \"397bdc0f-9466-4bf1-ab28-e31d0dd308d1\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc" Oct 31 13:26:42.258332 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:42.251459 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98jgd\" (UniqueName: \"kubernetes.io/projected/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-kube-api-access-98jgd\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc\" (UID: \"397bdc0f-9466-4bf1-ab28-e31d0dd308d1\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc" Oct 31 13:26:42.258332 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:42.251532 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc\" (UID: \"397bdc0f-9466-4bf1-ab28-e31d0dd308d1\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc" Oct 31 13:26:42.259668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:42.252698 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc\" (UID: \"397bdc0f-9466-4bf1-ab28-e31d0dd308d1\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc" Oct 31 13:26:42.260925 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:42.260473 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc\" (UID: \"397bdc0f-9466-4bf1-ab28-e31d0dd308d1\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc" Oct 31 13:26:42.412898 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:42.409390 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc\" (UID: \"397bdc0f-9466-4bf1-ab28-e31d0dd308d1\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc" Oct 31 13:26:42.483676 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:42.483385 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98jgd\" (UniqueName: \"kubernetes.io/projected/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-kube-api-access-98jgd\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc\" (UID: \"397bdc0f-9466-4bf1-ab28-e31d0dd308d1\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc" Oct 31 13:26:42.676529 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:42.676494 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc" Oct 31 13:26:42.803046 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:42.802954 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm"] Oct 31 13:26:42.981395 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:42.981362 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc"] Oct 31 13:26:42.981674 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:26:42.981620 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod397bdc0f_9466_4bf1_ab28_e31d0dd308d1.slice/crio-5b33d28c8361ed3f98122e238ae4548eecea9917d5238cdd1aff7ad70d1388b8 WatchSource:0}: Error finding container 5b33d28c8361ed3f98122e238ae4548eecea9917d5238cdd1aff7ad70d1388b8: Status 404 returned error can't find the container with id 5b33d28c8361ed3f98122e238ae4548eecea9917d5238cdd1aff7ad70d1388b8 Oct 31 13:26:43.421332 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:43.421300 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc" event={"ID":"397bdc0f-9466-4bf1-ab28-e31d0dd308d1","Type":"ContainerStarted","Data":"01b35679c27e98f37af94be75d2f528f7508bad2136e43adcf468418e356ad05"} Oct 31 13:26:43.421575 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:43.421355 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc" event={"ID":"397bdc0f-9466-4bf1-ab28-e31d0dd308d1","Type":"ContainerStarted","Data":"5b33d28c8361ed3f98122e238ae4548eecea9917d5238cdd1aff7ad70d1388b8"} Oct 31 13:26:43.421575 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:43.421384 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc" Oct 31 13:26:43.423169 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:43.423145 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm" event={"ID":"f4534300-b54b-441b-a78d-d9577ee14a29","Type":"ContainerStarted","Data":"6598b6ed53d22fe47bf715e798d5f4a7125808375f5463eac97a7f0d6bb4e3b2"} Oct 31 13:26:43.423302 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:43.423186 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm" event={"ID":"f4534300-b54b-441b-a78d-d9577ee14a29","Type":"ContainerStarted","Data":"7887e2b2c4891b043c8ae6f87f67aa0ee1727e82af044276cfccc887237e4803"} Oct 31 13:26:43.423302 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:43.423206 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm" Oct 31 13:26:43.444357 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:43.444311 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc" podStartSLOduration=1.444297105 podStartE2EDuration="1.444297105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:26:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:26:43.440847965 +0000 UTC m=+3254.148434262" watchObservedRunningTime="2026-10-31 13:26:43.444297105 +0000 UTC m=+3254.151883400" Oct 31 13:26:43.462408 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:43.462343 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm" podStartSLOduration=2.462325815 podStartE2EDuration="2.462325815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:26:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:26:43.461424517 +0000 UTC m=+3254.169010814" watchObservedRunningTime="2026-10-31 13:26:43.462325815 +0000 UTC m=+3254.169912112" Oct 31 13:26:43.690627 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:43.690543 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm"] Oct 31 13:26:43.704433 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:43.704386 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7"] Oct 31 13:26:44.132241 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:44.132154 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm"] Oct 31 13:26:44.145660 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:44.139167 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc"] Oct 31 13:26:44.473663 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:44.468864 5693 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc" secret="" err="secret \"payload-processing-external-model-e2e--e205175e-dockercfg-4dwt6\" not found" Oct 31 13:26:44.473663 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:44.472794 5693 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm" secret="" err="secret \"payload-processing-external-model-e2e--4724aaf0-dockercfg-rtn5f\" not found" Oct 31 13:26:44.598665 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:44.588844 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-9ef744/agc-cred-d0bf873b9006: secret "agc-cred-d0bf873b9006" not found Oct 31 13:26:44.598665 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:44.588873 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm: secret "agc-cred-d0bf873b9006" not found Oct 31 13:26:44.598665 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:44.588934 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4534300-b54b-441b-a78d-d9577ee14a29-provider-credentials podName:f4534300-b54b-441b-a78d-d9577ee14a29 nodeName:}" failed. No retries permitted until 2026-10-31 13:26:45.088913064 +0000 UTC m=+3255.796499337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f4534300-b54b-441b-a78d-d9577ee14a29-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm" (UID: "f4534300-b54b-441b-a78d-d9577ee14a29") : secret "agc-cred-d0bf873b9006" not found Oct 31 13:26:44.598665 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:44.588985 5693 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-plugins-e2e-extmod-b-29e3a0: configmap "payload-processing-external-model-plugins-e2e-extmod-b-29e3a0" not found Oct 31 13:26:44.598665 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:44.589019 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-plugins-config-volume podName:397bdc0f-9466-4bf1-ab28-e31d0dd308d1 nodeName:}" failed. No retries permitted until 2026-10-31 13:26:45.089009001 +0000 UTC m=+3255.796595289 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc" (UID: "397bdc0f-9466-4bf1-ab28-e31d0dd308d1") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-29e3a0" not found Oct 31 13:26:44.598665 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:44.589086 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-29e3a0/agc-cred-29aa17c26ce2: secret "agc-cred-29aa17c26ce2" not found Oct 31 13:26:44.598665 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:44.589096 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc: secret "agc-cred-29aa17c26ce2" not found Oct 31 13:26:44.598665 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:44.589128 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-provider-credentials podName:397bdc0f-9466-4bf1-ab28-e31d0dd308d1 nodeName:}" failed. No retries permitted until 2026-10-31 13:26:45.089118168 +0000 UTC m=+3255.796704456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc" (UID: "397bdc0f-9466-4bf1-ab28-e31d0dd308d1") : secret "agc-cred-29aa17c26ce2" not found Oct 31 13:26:44.598665 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:44.589659 5693 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-plugins-e2e-extmod-a-9ef744: configmap "payload-processing-external-model-plugins-e2e-extmod-a-9ef744" not found Oct 31 13:26:44.598665 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:44.589703 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4534300-b54b-441b-a78d-d9577ee14a29-plugins-config-volume podName:f4534300-b54b-441b-a78d-d9577ee14a29 nodeName:}" failed. No retries permitted until 2026-10-31 13:26:45.089689554 +0000 UTC m=+3255.797275842 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/f4534300-b54b-441b-a78d-d9577ee14a29-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm" (UID: "f4534300-b54b-441b-a78d-d9577ee14a29") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-9ef744" not found Oct 31 13:26:45.098502 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:45.098463 5693 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-plugins-e2e-extmod-b-29e3a0: configmap "payload-processing-external-model-plugins-e2e-extmod-b-29e3a0" not found Oct 31 13:26:45.098740 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:45.098527 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-plugins-config-volume podName:397bdc0f-9466-4bf1-ab28-e31d0dd308d1 nodeName:}" failed. No retries permitted until 2026-10-31 13:26:46.09851279 +0000 UTC m=+3256.806099063 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc" (UID: "397bdc0f-9466-4bf1-ab28-e31d0dd308d1") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-29e3a0" not found Oct 31 13:26:45.098740 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:45.098557 5693 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-plugins-e2e-extmod-a-9ef744: configmap "payload-processing-external-model-plugins-e2e-extmod-a-9ef744" not found Oct 31 13:26:45.098740 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:45.098558 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-29e3a0/agc-cred-29aa17c26ce2: secret "agc-cred-29aa17c26ce2" not found Oct 31 13:26:45.098740 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:45.098623 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc: secret "agc-cred-29aa17c26ce2" not found Oct 31 13:26:45.098740 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:45.098628 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4534300-b54b-441b-a78d-d9577ee14a29-plugins-config-volume podName:f4534300-b54b-441b-a78d-d9577ee14a29 nodeName:}" failed. No retries permitted until 2026-10-31 13:26:46.098612498 +0000 UTC m=+3256.806198788 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/f4534300-b54b-441b-a78d-d9577ee14a29-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm" (UID: "f4534300-b54b-441b-a78d-d9577ee14a29") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-9ef744" not found Oct 31 13:26:45.098740 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:45.098684 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-provider-credentials podName:397bdc0f-9466-4bf1-ab28-e31d0dd308d1 nodeName:}" failed. No retries permitted until 2026-10-31 13:26:46.098673924 +0000 UTC m=+3256.806260199 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc" (UID: "397bdc0f-9466-4bf1-ab28-e31d0dd308d1") : secret "agc-cred-29aa17c26ce2" not found Oct 31 13:26:45.099051 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:45.098873 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-9ef744/agc-cred-d0bf873b9006: secret "agc-cred-d0bf873b9006" not found Oct 31 13:26:45.099051 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:45.098887 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm: secret "agc-cred-d0bf873b9006" not found Oct 31 13:26:45.099051 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:45.098933 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4534300-b54b-441b-a78d-d9577ee14a29-provider-credentials podName:f4534300-b54b-441b-a78d-d9577ee14a29 nodeName:}" failed. No retries permitted until 2026-10-31 13:26:46.098920061 +0000 UTC m=+3256.806506354 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f4534300-b54b-441b-a78d-d9577ee14a29-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm" (UID: "f4534300-b54b-441b-a78d-d9577ee14a29") : secret "agc-cred-d0bf873b9006" not found Oct 31 13:26:46.122670 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:46.122618 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-29e3a0/agc-cred-29aa17c26ce2: secret "agc-cred-29aa17c26ce2" not found Oct 31 13:26:46.122670 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:46.122668 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc: secret "agc-cred-29aa17c26ce2" not found Oct 31 13:26:46.122670 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:46.122632 5693 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-plugins-e2e-extmod-a-9ef744: configmap "payload-processing-external-model-plugins-e2e-extmod-a-9ef744" not found Oct 31 13:26:46.123262 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:46.122739 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4534300-b54b-441b-a78d-d9577ee14a29-plugins-config-volume podName:f4534300-b54b-441b-a78d-d9577ee14a29 nodeName:}" failed. No retries permitted until 2026-10-31 13:26:48.122717983 +0000 UTC m=+3258.830304259 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/f4534300-b54b-441b-a78d-d9577ee14a29-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm" (UID: "f4534300-b54b-441b-a78d-d9577ee14a29") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-9ef744" not found Oct 31 13:26:46.123262 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:46.122796 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-9ef744/agc-cred-d0bf873b9006: secret "agc-cred-d0bf873b9006" not found Oct 31 13:26:46.123262 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:46.122805 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-provider-credentials podName:397bdc0f-9466-4bf1-ab28-e31d0dd308d1 nodeName:}" failed. No retries permitted until 2026-10-31 13:26:48.122785637 +0000 UTC m=+3258.830371923 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc" (UID: "397bdc0f-9466-4bf1-ab28-e31d0dd308d1") : secret "agc-cred-29aa17c26ce2" not found Oct 31 13:26:46.123262 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:46.122815 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm: secret "agc-cred-d0bf873b9006" not found Oct 31 13:26:46.123262 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:46.122881 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4534300-b54b-441b-a78d-d9577ee14a29-provider-credentials podName:f4534300-b54b-441b-a78d-d9577ee14a29 nodeName:}" failed. No retries permitted until 2026-10-31 13:26:48.122864545 +0000 UTC m=+3258.830450831 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f4534300-b54b-441b-a78d-d9577ee14a29-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm" (UID: "f4534300-b54b-441b-a78d-d9577ee14a29") : secret "agc-cred-d0bf873b9006" not found Oct 31 13:26:46.123262 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:46.122899 5693 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-plugins-e2e-extmod-b-29e3a0: configmap "payload-processing-external-model-plugins-e2e-extmod-b-29e3a0" not found Oct 31 13:26:46.123262 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:46.122938 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-plugins-config-volume podName:397bdc0f-9466-4bf1-ab28-e31d0dd308d1 nodeName:}" failed. No retries permitted until 2026-10-31 13:26:48.122928257 +0000 UTC m=+3258.830514530 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc" (UID: "397bdc0f-9466-4bf1-ab28-e31d0dd308d1") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-29e3a0" not found Oct 31 13:26:48.158827 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:48.158795 5693 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-plugins-e2e-extmod-b-29e3a0: configmap "payload-processing-external-model-plugins-e2e-extmod-b-29e3a0" not found Oct 31 13:26:48.159304 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:48.158880 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-plugins-config-volume podName:397bdc0f-9466-4bf1-ab28-e31d0dd308d1 nodeName:}" failed. No retries permitted until 2026-10-31 13:26:52.158861313 +0000 UTC m=+3262.866447601 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc" (UID: "397bdc0f-9466-4bf1-ab28-e31d0dd308d1") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-29e3a0" not found Oct 31 13:26:48.159304 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:48.158942 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-29e3a0/agc-cred-29aa17c26ce2: secret "agc-cred-29aa17c26ce2" not found Oct 31 13:26:48.159304 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:48.158965 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc: secret "agc-cred-29aa17c26ce2" not found Oct 31 13:26:48.159304 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:48.159022 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-provider-credentials podName:397bdc0f-9466-4bf1-ab28-e31d0dd308d1 nodeName:}" failed. No retries permitted until 2026-10-31 13:26:52.159002607 +0000 UTC m=+3262.866588894 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc" (UID: "397bdc0f-9466-4bf1-ab28-e31d0dd308d1") : secret "agc-cred-29aa17c26ce2" not found Oct 31 13:26:48.159304 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:48.159032 5693 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-plugins-e2e-extmod-a-9ef744: configmap "payload-processing-external-model-plugins-e2e-extmod-a-9ef744" not found Oct 31 13:26:48.159304 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:48.159065 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4534300-b54b-441b-a78d-d9577ee14a29-plugins-config-volume podName:f4534300-b54b-441b-a78d-d9577ee14a29 nodeName:}" failed. No retries permitted until 2026-10-31 13:26:52.159055143 +0000 UTC m=+3262.866641416 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/f4534300-b54b-441b-a78d-d9577ee14a29-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm" (UID: "f4534300-b54b-441b-a78d-d9577ee14a29") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-9ef744" not found Oct 31 13:26:48.159304 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:48.159250 5693 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-9ef744/agc-cred-d0bf873b9006: secret "agc-cred-d0bf873b9006" not found Oct 31 13:26:48.159304 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:48.159266 5693 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm: secret "agc-cred-d0bf873b9006" not found Oct 31 13:26:48.159304 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:48.159311 5693 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4534300-b54b-441b-a78d-d9577ee14a29-provider-credentials podName:f4534300-b54b-441b-a78d-d9577ee14a29 nodeName:}" failed. No retries permitted until 2026-10-31 13:26:52.159296845 +0000 UTC m=+3262.866883135 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f4534300-b54b-441b-a78d-d9577ee14a29-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm" (UID: "f4534300-b54b-441b-a78d-d9577ee14a29") : secret "agc-cred-d0bf873b9006" not found Oct 31 13:26:48.689353 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:48.689312 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:26:48.704976 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:48.704939 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7" podUID="9c115210-830a-4c8c-b8d3-b661941f535c" containerName="payload-processing" containerID="cri-o://362cb8a23c8d9094beded9e873c2af5c5472f8c92af8c6a1f578c405d2032d6c" gracePeriod=25 Oct 31 13:26:49.001746 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.001718 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7" Oct 31 13:26:49.111124 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.111079 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c115210-830a-4c8c-b8d3-b661941f535c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9c115210-830a-4c8c-b8d3-b661941f535c-routing-overlay-volume\") pod \"9c115210-830a-4c8c-b8d3-b661941f535c\" (UID: \"9c115210-830a-4c8c-b8d3-b661941f535c\") " Oct 31 13:26:49.111387 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.111180 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c115210-830a-4c8c-b8d3-b661941f535c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9c115210-830a-4c8c-b8d3-b661941f535c-provider-credentials\") pod \"9c115210-830a-4c8c-b8d3-b661941f535c\" (UID: \"9c115210-830a-4c8c-b8d3-b661941f535c\") " Oct 31 13:26:49.111387 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.111216 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c115210-830a-4c8c-b8d3-b661941f535c-kube-api-access-gwh2w\" (UniqueName: \"kubernetes.io/projected/9c115210-830a-4c8c-b8d3-b661941f535c-kube-api-access-gwh2w\") pod \"9c115210-830a-4c8c-b8d3-b661941f535c\" (UID: \"9c115210-830a-4c8c-b8d3-b661941f535c\") " Oct 31 13:26:49.111547 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.111391 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c115210-830a-4c8c-b8d3-b661941f535c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c115210-830a-4c8c-b8d3-b661941f535c-plugins-config-volume\") pod \"9c115210-830a-4c8c-b8d3-b661941f535c\" (UID: \"9c115210-830a-4c8c-b8d3-b661941f535c\") " Oct 31 13:26:49.112667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.112593 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c115210-830a-4c8c-b8d3-b661941f535c-plugins-config-volume" pod "9c115210-830a-4c8c-b8d3-b661941f535c" (UID: "9c115210-830a-4c8c-b8d3-b661941f535c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:26:49.114051 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.114030 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c115210-830a-4c8c-b8d3-b661941f535c-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:49.114184 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.114087 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c115210-830a-4c8c-b8d3-b661941f535c-routing-overlay-volume" pod "9c115210-830a-4c8c-b8d3-b661941f535c" (UID: "9c115210-830a-4c8c-b8d3-b661941f535c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:26:49.116261 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.116231 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c115210-830a-4c8c-b8d3-b661941f535c-kube-api-access-gwh2w" pod "9c115210-830a-4c8c-b8d3-b661941f535c" (UID: "9c115210-830a-4c8c-b8d3-b661941f535c"). InnerVolumeSpecName "kube-api-access-gwh2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:26:49.117216 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.117185 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c115210-830a-4c8c-b8d3-b661941f535c-provider-credentials" pod "9c115210-830a-4c8c-b8d3-b661941f535c" (UID: "9c115210-830a-4c8c-b8d3-b661941f535c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:26:49.158348 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.157022 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm" podUID="76fa58b1-d342-45ee-9b00-5aca023ac806" containerName="payload-processing" containerID="cri-o://68efc67710ef2187efc90c12a93cf219e2020b9f85e28dbc35bf80ab5b452869" gracePeriod=25 Oct 31 13:26:49.215673 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.215568 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9c115210-830a-4c8c-b8d3-b661941f535c-routing-overlay-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:49.215673 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.215597 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9c115210-830a-4c8c-b8d3-b661941f535c-provider-credentials\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:49.215673 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.215607 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwh2w\" (UniqueName: \"kubernetes.io/projected/9c115210-830a-4c8c-b8d3-b661941f535c-kube-api-access-gwh2w\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:49.441001 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.440973 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm" Oct 31 13:26:49.510898 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.510801 5693 generic.go:356] "Generic (PLEG): container finished" podID="76fa58b1-d342-45ee-9b00-5aca023ac806" containerID="68efc67710ef2187efc90c12a93cf219e2020b9f85e28dbc35bf80ab5b452869" exitCode=0 Oct 31 13:26:49.510898 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.510840 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm" event={"ID":"76fa58b1-d342-45ee-9b00-5aca023ac806","Type":"ContainerDied","Data":"68efc67710ef2187efc90c12a93cf219e2020b9f85e28dbc35bf80ab5b452869"} Oct 31 13:26:49.511115 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.510904 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm" Oct 31 13:26:49.511465 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.511439 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm" event={"ID":"76fa58b1-d342-45ee-9b00-5aca023ac806","Type":"ContainerDied","Data":"018b24304e7bea02f4dfc266488b29d1c65691baaf8a2d617bc64742405b358b"} Oct 31 13:26:49.511698 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.511681 5693 scope.go:122] "RemoveContainer" containerID="68efc67710ef2187efc90c12a93cf219e2020b9f85e28dbc35bf80ab5b452869" Oct 31 13:26:49.513037 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.512970 5693 generic.go:356] "Generic (PLEG): container finished" podID="9c115210-830a-4c8c-b8d3-b661941f535c" containerID="362cb8a23c8d9094beded9e873c2af5c5472f8c92af8c6a1f578c405d2032d6c" exitCode=0 Oct 31 13:26:49.513037 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.513023 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7" Oct 31 13:26:49.513238 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.513220 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7" event={"ID":"9c115210-830a-4c8c-b8d3-b661941f535c","Type":"ContainerDied","Data":"362cb8a23c8d9094beded9e873c2af5c5472f8c92af8c6a1f578c405d2032d6c"} Oct 31 13:26:49.513332 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.513263 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7" event={"ID":"9c115210-830a-4c8c-b8d3-b661941f535c","Type":"ContainerDied","Data":"b705c0454293b1a5a715e1c519850093c132c77a99ab6108d12f58dac7c98dcc"} Oct 31 13:26:49.520521 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.520500 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76fa58b1-d342-45ee-9b00-5aca023ac806-kube-api-access-rxs27\" (UniqueName: \"kubernetes.io/projected/76fa58b1-d342-45ee-9b00-5aca023ac806-kube-api-access-rxs27\") pod \"76fa58b1-d342-45ee-9b00-5aca023ac806\" (UID: \"76fa58b1-d342-45ee-9b00-5aca023ac806\") " Oct 31 13:26:49.520629 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.520612 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76fa58b1-d342-45ee-9b00-5aca023ac806-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76fa58b1-d342-45ee-9b00-5aca023ac806-plugins-config-volume\") pod \"76fa58b1-d342-45ee-9b00-5aca023ac806\" (UID: \"76fa58b1-d342-45ee-9b00-5aca023ac806\") " Oct 31 13:26:49.520731 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.520655 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76fa58b1-d342-45ee-9b00-5aca023ac806-provider-credentials\" (UniqueName: \"kubernetes.io/projected/76fa58b1-d342-45ee-9b00-5aca023ac806-provider-credentials\") pod \"76fa58b1-d342-45ee-9b00-5aca023ac806\" (UID: \"76fa58b1-d342-45ee-9b00-5aca023ac806\") " Oct 31 13:26:49.520954 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.520929 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76fa58b1-d342-45ee-9b00-5aca023ac806-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/76fa58b1-d342-45ee-9b00-5aca023ac806-routing-overlay-volume\") pod \"76fa58b1-d342-45ee-9b00-5aca023ac806\" (UID: \"76fa58b1-d342-45ee-9b00-5aca023ac806\") " Oct 31 13:26:49.521371 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.521345 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76fa58b1-d342-45ee-9b00-5aca023ac806-routing-overlay-volume" pod "76fa58b1-d342-45ee-9b00-5aca023ac806" (UID: "76fa58b1-d342-45ee-9b00-5aca023ac806"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:26:49.522710 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.522686 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76fa58b1-d342-45ee-9b00-5aca023ac806-plugins-config-volume" pod "76fa58b1-d342-45ee-9b00-5aca023ac806" (UID: "76fa58b1-d342-45ee-9b00-5aca023ac806"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:26:49.522710 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.522701 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/76fa58b1-d342-45ee-9b00-5aca023ac806-routing-overlay-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:49.526659 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.524403 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76fa58b1-d342-45ee-9b00-5aca023ac806-kube-api-access-rxs27" pod "76fa58b1-d342-45ee-9b00-5aca023ac806" (UID: "76fa58b1-d342-45ee-9b00-5aca023ac806"). InnerVolumeSpecName "kube-api-access-rxs27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:26:49.526659 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.526592 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76fa58b1-d342-45ee-9b00-5aca023ac806-provider-credentials" pod "76fa58b1-d342-45ee-9b00-5aca023ac806" (UID: "76fa58b1-d342-45ee-9b00-5aca023ac806"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:26:49.528506 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.528483 5693 scope.go:122] "RemoveContainer" containerID="68efc67710ef2187efc90c12a93cf219e2020b9f85e28dbc35bf80ab5b452869" Oct 31 13:26:49.528901 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:49.528876 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68efc67710ef2187efc90c12a93cf219e2020b9f85e28dbc35bf80ab5b452869\": container with ID starting with 68efc67710ef2187efc90c12a93cf219e2020b9f85e28dbc35bf80ab5b452869 not found: ID does not exist" containerID="68efc67710ef2187efc90c12a93cf219e2020b9f85e28dbc35bf80ab5b452869" Oct 31 13:26:49.529020 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.528906 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68efc67710ef2187efc90c12a93cf219e2020b9f85e28dbc35bf80ab5b452869"} err="failed to get container status \"68efc67710ef2187efc90c12a93cf219e2020b9f85e28dbc35bf80ab5b452869\": rpc error: code = NotFound desc = could not find container \"68efc67710ef2187efc90c12a93cf219e2020b9f85e28dbc35bf80ab5b452869\": container with ID starting with 68efc67710ef2187efc90c12a93cf219e2020b9f85e28dbc35bf80ab5b452869 not found: ID does not exist" Oct 31 13:26:49.529020 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.528923 5693 scope.go:122] "RemoveContainer" containerID="362cb8a23c8d9094beded9e873c2af5c5472f8c92af8c6a1f578c405d2032d6c" Oct 31 13:26:49.543489 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.543466 5693 scope.go:122] "RemoveContainer" containerID="362cb8a23c8d9094beded9e873c2af5c5472f8c92af8c6a1f578c405d2032d6c" Oct 31 13:26:49.543827 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:49.543803 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"362cb8a23c8d9094beded9e873c2af5c5472f8c92af8c6a1f578c405d2032d6c\": container with ID starting with 362cb8a23c8d9094beded9e873c2af5c5472f8c92af8c6a1f578c405d2032d6c not found: ID does not exist" containerID="362cb8a23c8d9094beded9e873c2af5c5472f8c92af8c6a1f578c405d2032d6c" Oct 31 13:26:49.543950 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.543832 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"362cb8a23c8d9094beded9e873c2af5c5472f8c92af8c6a1f578c405d2032d6c"} err="failed to get container status \"362cb8a23c8d9094beded9e873c2af5c5472f8c92af8c6a1f578c405d2032d6c\": rpc error: code = NotFound desc = could not find container \"362cb8a23c8d9094beded9e873c2af5c5472f8c92af8c6a1f578c405d2032d6c\": container with ID starting with 362cb8a23c8d9094beded9e873c2af5c5472f8c92af8c6a1f578c405d2032d6c not found: ID does not exist" Oct 31 13:26:49.543950 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.543898 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7"] Oct 31 13:26:49.549194 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.549171 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-f7567fcs7"] Oct 31 13:26:49.624759 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.624717 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxs27\" (UniqueName: \"kubernetes.io/projected/76fa58b1-d342-45ee-9b00-5aca023ac806-kube-api-access-rxs27\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:49.624759 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.624757 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76fa58b1-d342-45ee-9b00-5aca023ac806-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:49.624759 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.624771 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/76fa58b1-d342-45ee-9b00-5aca023ac806-provider-credentials\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:49.838250 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.838220 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm"] Oct 31 13:26:49.842754 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:49.842731 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f65z42dm"] Oct 31 13:26:50.475723 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:50.475682 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm" podUID="f4534300-b54b-441b-a78d-d9577ee14a29" containerName="payload-processing" containerID="cri-o://6598b6ed53d22fe47bf715e798d5f4a7125808375f5463eac97a7f0d6bb4e3b2" gracePeriod=25 Oct 31 13:26:50.475723 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:50.475682 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc" podUID="397bdc0f-9466-4bf1-ab28-e31d0dd308d1" containerName="payload-processing" containerID="cri-o://01b35679c27e98f37af94be75d2f528f7508bad2136e43adcf468418e356ad05" gracePeriod=25 Oct 31 13:26:50.826221 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:50.826193 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc" Oct 31 13:26:50.879844 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:50.879777 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm" Oct 31 13:26:50.964735 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:50.964704 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-provider-credentials\") pod \"397bdc0f-9466-4bf1-ab28-e31d0dd308d1\" (UID: \"397bdc0f-9466-4bf1-ab28-e31d0dd308d1\") " Oct 31 13:26:50.964949 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:50.964757 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-kube-api-access-98jgd\" (UniqueName: \"kubernetes.io/projected/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-kube-api-access-98jgd\") pod \"397bdc0f-9466-4bf1-ab28-e31d0dd308d1\" (UID: \"397bdc0f-9466-4bf1-ab28-e31d0dd308d1\") " Oct 31 13:26:50.964949 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:50.964808 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4534300-b54b-441b-a78d-d9577ee14a29-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4534300-b54b-441b-a78d-d9577ee14a29-plugins-config-volume\") pod \"f4534300-b54b-441b-a78d-d9577ee14a29\" (UID: \"f4534300-b54b-441b-a78d-d9577ee14a29\") " Oct 31 13:26:50.964949 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:50.964849 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4534300-b54b-441b-a78d-d9577ee14a29-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4534300-b54b-441b-a78d-d9577ee14a29-provider-credentials\") pod \"f4534300-b54b-441b-a78d-d9577ee14a29\" (UID: \"f4534300-b54b-441b-a78d-d9577ee14a29\") " Oct 31 13:26:50.965125 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:50.964951 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4534300-b54b-441b-a78d-d9577ee14a29-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4534300-b54b-441b-a78d-d9577ee14a29-routing-overlay-volume\") pod \"f4534300-b54b-441b-a78d-d9577ee14a29\" (UID: \"f4534300-b54b-441b-a78d-d9577ee14a29\") " Oct 31 13:26:50.965125 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:50.965013 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-routing-overlay-volume\") pod \"397bdc0f-9466-4bf1-ab28-e31d0dd308d1\" (UID: \"397bdc0f-9466-4bf1-ab28-e31d0dd308d1\") " Oct 31 13:26:50.965233 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:50.965119 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4534300-b54b-441b-a78d-d9577ee14a29-kube-api-access-wjdq8\" (UniqueName: \"kubernetes.io/projected/f4534300-b54b-441b-a78d-d9577ee14a29-kube-api-access-wjdq8\") pod \"f4534300-b54b-441b-a78d-d9577ee14a29\" (UID: \"f4534300-b54b-441b-a78d-d9577ee14a29\") " Oct 31 13:26:50.965322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:50.965283 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-plugins-config-volume\") pod \"397bdc0f-9466-4bf1-ab28-e31d0dd308d1\" (UID: \"397bdc0f-9466-4bf1-ab28-e31d0dd308d1\") " Oct 31 13:26:50.965436 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:50.965331 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4534300-b54b-441b-a78d-d9577ee14a29-plugins-config-volume" pod "f4534300-b54b-441b-a78d-d9577ee14a29" (UID: "f4534300-b54b-441b-a78d-d9577ee14a29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:26:50.965500 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:50.965429 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4534300-b54b-441b-a78d-d9577ee14a29-routing-overlay-volume" pod "f4534300-b54b-441b-a78d-d9577ee14a29" (UID: "f4534300-b54b-441b-a78d-d9577ee14a29"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:26:50.965556 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:50.965492 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-routing-overlay-volume" pod "397bdc0f-9466-4bf1-ab28-e31d0dd308d1" (UID: "397bdc0f-9466-4bf1-ab28-e31d0dd308d1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:26:50.966132 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:50.966106 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4534300-b54b-441b-a78d-d9577ee14a29-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:50.966132 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:50.966135 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4534300-b54b-441b-a78d-d9577ee14a29-routing-overlay-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:50.966712 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:50.966149 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-routing-overlay-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:50.966712 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:50.966270 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-plugins-config-volume" pod "397bdc0f-9466-4bf1-ab28-e31d0dd308d1" (UID: "397bdc0f-9466-4bf1-ab28-e31d0dd308d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:26:50.967560 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:50.967535 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-kube-api-access-98jgd" pod "397bdc0f-9466-4bf1-ab28-e31d0dd308d1" (UID: "397bdc0f-9466-4bf1-ab28-e31d0dd308d1"). InnerVolumeSpecName "kube-api-access-98jgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:26:50.967909 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:50.967885 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4534300-b54b-441b-a78d-d9577ee14a29-provider-credentials" pod "f4534300-b54b-441b-a78d-d9577ee14a29" (UID: "f4534300-b54b-441b-a78d-d9577ee14a29"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:26:50.967987 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:50.967924 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-provider-credentials" pod "397bdc0f-9466-4bf1-ab28-e31d0dd308d1" (UID: "397bdc0f-9466-4bf1-ab28-e31d0dd308d1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:26:50.968259 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:50.968238 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4534300-b54b-441b-a78d-d9577ee14a29-kube-api-access-wjdq8" pod "f4534300-b54b-441b-a78d-d9577ee14a29" (UID: "f4534300-b54b-441b-a78d-d9577ee14a29"). InnerVolumeSpecName "kube-api-access-wjdq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:26:51.068284 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:51.068253 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-provider-credentials\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:51.068284 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:51.068284 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98jgd\" (UniqueName: \"kubernetes.io/projected/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-kube-api-access-98jgd\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:51.068575 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:51.068300 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4534300-b54b-441b-a78d-d9577ee14a29-provider-credentials\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:51.068575 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:51.068312 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjdq8\" (UniqueName: \"kubernetes.io/projected/f4534300-b54b-441b-a78d-d9577ee14a29-kube-api-access-wjdq8\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:51.068575 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:51.068324 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/397bdc0f-9466-4bf1-ab28-e31d0dd308d1-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:51.462494 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:51.462458 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76fa58b1-d342-45ee-9b00-5aca023ac806" path="/var/lib/kubelet/pods/76fa58b1-d342-45ee-9b00-5aca023ac806/volumes" Oct 31 13:26:51.463287 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:51.463259 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c115210-830a-4c8c-b8d3-b661941f535c" path="/var/lib/kubelet/pods/9c115210-830a-4c8c-b8d3-b661941f535c/volumes" Oct 31 13:26:51.538393 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:51.538362 5693 generic.go:356] "Generic (PLEG): container finished" podID="397bdc0f-9466-4bf1-ab28-e31d0dd308d1" containerID="01b35679c27e98f37af94be75d2f528f7508bad2136e43adcf468418e356ad05" exitCode=0 Oct 31 13:26:51.538910 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:51.538426 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc" Oct 31 13:26:51.538910 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:51.538443 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc" event={"ID":"397bdc0f-9466-4bf1-ab28-e31d0dd308d1","Type":"ContainerDied","Data":"01b35679c27e98f37af94be75d2f528f7508bad2136e43adcf468418e356ad05"} Oct 31 13:26:51.538910 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:51.538496 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc" event={"ID":"397bdc0f-9466-4bf1-ab28-e31d0dd308d1","Type":"ContainerDied","Data":"5b33d28c8361ed3f98122e238ae4548eecea9917d5238cdd1aff7ad70d1388b8"} Oct 31 13:26:51.538910 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:51.538520 5693 scope.go:122] "RemoveContainer" containerID="01b35679c27e98f37af94be75d2f528f7508bad2136e43adcf468418e356ad05" Oct 31 13:26:51.540114 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:51.540091 5693 generic.go:356] "Generic (PLEG): container finished" podID="f4534300-b54b-441b-a78d-d9577ee14a29" containerID="6598b6ed53d22fe47bf715e798d5f4a7125808375f5463eac97a7f0d6bb4e3b2" exitCode=0 Oct 31 13:26:51.540252 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:51.540169 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm" event={"ID":"f4534300-b54b-441b-a78d-d9577ee14a29","Type":"ContainerDied","Data":"6598b6ed53d22fe47bf715e798d5f4a7125808375f5463eac97a7f0d6bb4e3b2"} Oct 31 13:26:51.540252 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:51.540202 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm" event={"ID":"f4534300-b54b-441b-a78d-d9577ee14a29","Type":"ContainerDied","Data":"7887e2b2c4891b043c8ae6f87f67aa0ee1727e82af044276cfccc887237e4803"} Oct 31 13:26:51.540252 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:51.540246 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm" Oct 31 13:26:51.550129 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:51.550109 5693 scope.go:122] "RemoveContainer" containerID="01b35679c27e98f37af94be75d2f528f7508bad2136e43adcf468418e356ad05" Oct 31 13:26:51.550379 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:51.550356 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01b35679c27e98f37af94be75d2f528f7508bad2136e43adcf468418e356ad05\": container with ID starting with 01b35679c27e98f37af94be75d2f528f7508bad2136e43adcf468418e356ad05 not found: ID does not exist" containerID="01b35679c27e98f37af94be75d2f528f7508bad2136e43adcf468418e356ad05" Oct 31 13:26:51.550468 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:51.550385 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01b35679c27e98f37af94be75d2f528f7508bad2136e43adcf468418e356ad05"} err="failed to get container status \"01b35679c27e98f37af94be75d2f528f7508bad2136e43adcf468418e356ad05\": rpc error: code = NotFound desc = could not find container \"01b35679c27e98f37af94be75d2f528f7508bad2136e43adcf468418e356ad05\": container with ID starting with 01b35679c27e98f37af94be75d2f528f7508bad2136e43adcf468418e356ad05 not found: ID does not exist" Oct 31 13:26:51.550468 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:51.550405 5693 scope.go:122] "RemoveContainer" containerID="6598b6ed53d22fe47bf715e798d5f4a7125808375f5463eac97a7f0d6bb4e3b2" Oct 31 13:26:51.561524 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:51.561499 5693 scope.go:122] "RemoveContainer" containerID="6598b6ed53d22fe47bf715e798d5f4a7125808375f5463eac97a7f0d6bb4e3b2" Oct 31 13:26:51.561866 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:26:51.561845 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6598b6ed53d22fe47bf715e798d5f4a7125808375f5463eac97a7f0d6bb4e3b2\": container with ID starting with 6598b6ed53d22fe47bf715e798d5f4a7125808375f5463eac97a7f0d6bb4e3b2 not found: ID does not exist" containerID="6598b6ed53d22fe47bf715e798d5f4a7125808375f5463eac97a7f0d6bb4e3b2" Oct 31 13:26:51.561979 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:51.561871 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6598b6ed53d22fe47bf715e798d5f4a7125808375f5463eac97a7f0d6bb4e3b2"} err="failed to get container status \"6598b6ed53d22fe47bf715e798d5f4a7125808375f5463eac97a7f0d6bb4e3b2\": rpc error: code = NotFound desc = could not find container \"6598b6ed53d22fe47bf715e798d5f4a7125808375f5463eac97a7f0d6bb4e3b2\": container with ID starting with 6598b6ed53d22fe47bf715e798d5f4a7125808375f5463eac97a7f0d6bb4e3b2 not found: ID does not exist" Oct 31 13:26:51.565071 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:51.565049 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc"] Oct 31 13:26:51.574469 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:51.574444 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-5f97bngmc"] Oct 31 13:26:51.587222 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:51.587197 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm"] Oct 31 13:26:51.591293 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:51.591276 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfccbjm"] Oct 31 13:26:52.458088 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.458054 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m"] Oct 31 13:26:52.458989 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.458973 5693 cpu_manager.go:414] "containerMap: removing container" podUID="76fa58b1-d342-45ee-9b00-5aca023ac806" containerName="payload-processing" Oct 31 13:26:52.459038 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.458990 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76fa58b1-d342-45ee-9b00-5aca023ac806" containerName="payload-processing" Oct 31 13:26:52.459038 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.459000 5693 cpu_manager.go:414] "containerMap: removing container" podUID="397bdc0f-9466-4bf1-ab28-e31d0dd308d1" containerName="payload-processing" Oct 31 13:26:52.459038 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.459005 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="397bdc0f-9466-4bf1-ab28-e31d0dd308d1" containerName="payload-processing" Oct 31 13:26:52.459038 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.459036 5693 cpu_manager.go:414] "containerMap: removing container" podUID="9c115210-830a-4c8c-b8d3-b661941f535c" containerName="payload-processing" Oct 31 13:26:52.459173 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.459041 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c115210-830a-4c8c-b8d3-b661941f535c" containerName="payload-processing" Oct 31 13:26:52.459173 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.459048 5693 cpu_manager.go:414] "containerMap: removing container" podUID="f4534300-b54b-441b-a78d-d9577ee14a29" containerName="payload-processing" Oct 31 13:26:52.459173 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.459053 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4534300-b54b-441b-a78d-d9577ee14a29" containerName="payload-processing" Oct 31 13:26:52.459266 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.459189 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="76fa58b1-d342-45ee-9b00-5aca023ac806" containerName="payload-processing" Oct 31 13:26:52.459266 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.459198 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="397bdc0f-9466-4bf1-ab28-e31d0dd308d1" containerName="payload-processing" Oct 31 13:26:52.459266 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.459211 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c115210-830a-4c8c-b8d3-b661941f535c" containerName="payload-processing" Oct 31 13:26:52.459266 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.459217 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4534300-b54b-441b-a78d-d9577ee14a29" containerName="payload-processing" Oct 31 13:26:52.464711 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.464674 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" Oct 31 13:26:52.470126 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.470048 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a39960-openshift-default-dockercfg-fjt9b\"" Oct 31 13:26:52.476355 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.476315 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m"] Oct 31 13:26:52.697672 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.688770 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9735d8ba-9f3e-434c-bb9f-5876f11d9336-credential-socket\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m\" (UID: \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" Oct 31 13:26:52.697672 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.688861 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9735d8ba-9f3e-434c-bb9f-5876f11d9336-workload-certs\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m\" (UID: \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" Oct 31 13:26:52.697672 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.688927 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9735d8ba-9f3e-434c-bb9f-5876f11d9336-workload-socket\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m\" (UID: \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" Oct 31 13:26:52.697672 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.689119 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9735d8ba-9f3e-434c-bb9f-5876f11d9336-istiod-ca-cert\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m\" (UID: \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" Oct 31 13:26:52.697672 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.689174 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c8lx\" (UniqueName: \"kubernetes.io/projected/9735d8ba-9f3e-434c-bb9f-5876f11d9336-kube-api-access-8c8lx\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m\" (UID: \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" Oct 31 13:26:52.697672 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.689246 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9735d8ba-9f3e-434c-bb9f-5876f11d9336-istio-token\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m\" (UID: \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" Oct 31 13:26:52.697672 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.689451 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9735d8ba-9f3e-434c-bb9f-5876f11d9336-istio-envoy\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m\" (UID: \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" Oct 31 13:26:52.697672 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.689501 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9735d8ba-9f3e-434c-bb9f-5876f11d9336-istio-podinfo\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m\" (UID: \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" Oct 31 13:26:52.697672 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.689586 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9735d8ba-9f3e-434c-bb9f-5876f11d9336-istio-data\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m\" (UID: \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" Oct 31 13:26:52.797865 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.797769 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9735d8ba-9f3e-434c-bb9f-5876f11d9336-istio-envoy\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m\" (UID: \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" Oct 31 13:26:52.798112 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.798090 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9735d8ba-9f3e-434c-bb9f-5876f11d9336-istio-podinfo\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m\" (UID: \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" Oct 31 13:26:52.798286 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.798271 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9735d8ba-9f3e-434c-bb9f-5876f11d9336-istio-data\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m\" (UID: \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" Oct 31 13:26:52.798543 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.798507 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9735d8ba-9f3e-434c-bb9f-5876f11d9336-credential-socket\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m\" (UID: \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" Oct 31 13:26:52.798818 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.798801 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9735d8ba-9f3e-434c-bb9f-5876f11d9336-workload-certs\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m\" (UID: \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" Oct 31 13:26:52.799043 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.799021 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9735d8ba-9f3e-434c-bb9f-5876f11d9336-workload-socket\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m\" (UID: \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" Oct 31 13:26:52.799283 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.799264 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9735d8ba-9f3e-434c-bb9f-5876f11d9336-istiod-ca-cert\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m\" (UID: \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" Oct 31 13:26:52.799438 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.799414 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c8lx\" (UniqueName: \"kubernetes.io/projected/9735d8ba-9f3e-434c-bb9f-5876f11d9336-kube-api-access-8c8lx\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m\" (UID: \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" Oct 31 13:26:52.799608 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.799586 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9735d8ba-9f3e-434c-bb9f-5876f11d9336-istio-token\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m\" (UID: \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" Oct 31 13:26:52.801451 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.801415 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9735d8ba-9f3e-434c-bb9f-5876f11d9336-workload-certs\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m\" (UID: \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" Oct 31 13:26:52.802361 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.802340 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9735d8ba-9f3e-434c-bb9f-5876f11d9336-istio-data\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m\" (UID: \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" Oct 31 13:26:52.803185 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.803155 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9735d8ba-9f3e-434c-bb9f-5876f11d9336-istiod-ca-cert\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m\" (UID: \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" Oct 31 13:26:52.803554 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.803529 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9735d8ba-9f3e-434c-bb9f-5876f11d9336-credential-socket\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m\" (UID: \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" Oct 31 13:26:52.803867 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.803847 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9735d8ba-9f3e-434c-bb9f-5876f11d9336-workload-socket\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m\" (UID: \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" Oct 31 13:26:52.807224 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.807200 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9735d8ba-9f3e-434c-bb9f-5876f11d9336-istio-envoy\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m\" (UID: \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" Oct 31 13:26:52.818770 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.817458 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9735d8ba-9f3e-434c-bb9f-5876f11d9336-istio-podinfo\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m\" (UID: \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" Oct 31 13:26:52.832361 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.832319 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c8lx\" (UniqueName: \"kubernetes.io/projected/9735d8ba-9f3e-434c-bb9f-5876f11d9336-kube-api-access-8c8lx\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m\" (UID: \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" Oct 31 13:26:52.833194 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:52.833156 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9735d8ba-9f3e-434c-bb9f-5876f11d9336-istio-token\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m\" (UID: \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" Oct 31 13:26:53.083468 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:53.083386 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" Oct 31 13:26:53.402197 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:26:53.402167 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9735d8ba_9f3e_434c_bb9f_5876f11d9336.slice/crio-fb7b5043990ad54796cdd123b5f20d3e5de087b00eae660486ed59524fe85187 WatchSource:0}: Error finding container fb7b5043990ad54796cdd123b5f20d3e5de087b00eae660486ed59524fe85187: Status 404 returned error can't find the container with id fb7b5043990ad54796cdd123b5f20d3e5de087b00eae660486ed59524fe85187 Oct 31 13:26:53.405795 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:53.405756 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:26:53.406084 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:53.406034 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:26:53.406199 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:53.406110 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:26:53.416100 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:53.416075 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="397bdc0f-9466-4bf1-ab28-e31d0dd308d1" path="/var/lib/kubelet/pods/397bdc0f-9466-4bf1-ab28-e31d0dd308d1/volumes" Oct 31 13:26:53.416823 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:53.416804 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4534300-b54b-441b-a78d-d9577ee14a29" path="/var/lib/kubelet/pods/f4534300-b54b-441b-a78d-d9577ee14a29/volumes" Oct 31 13:26:53.417306 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:53.417289 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m"] Oct 31 13:26:53.667424 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:53.667341 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" event={"ID":"9735d8ba-9f3e-434c-bb9f-5876f11d9336","Type":"ContainerStarted","Data":"523ae8670acc553ccb1385c8cfcad2524d46d9d019df1f3545ff8e15a0d6416e"} Oct 31 13:26:53.667424 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:53.667397 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" event={"ID":"9735d8ba-9f3e-434c-bb9f-5876f11d9336","Type":"ContainerStarted","Data":"fb7b5043990ad54796cdd123b5f20d3e5de087b00eae660486ed59524fe85187"} Oct 31 13:26:53.693355 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:53.693305 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" podStartSLOduration=1.693291138 podStartE2EDuration="1.693291138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:26:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:26:53.691917331 +0000 UTC m=+3264.399503626" watchObservedRunningTime="2026-10-31 13:26:53.693291138 +0000 UTC m=+3264.400877433" Oct 31 13:26:54.083653 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:54.083551 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" Oct 31 13:26:54.090345 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:54.090316 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" Oct 31 13:26:54.676931 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:54.676905 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" Oct 31 13:26:54.677902 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:54.677882 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" Oct 31 13:26:57.511368 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:57.511338 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a39960-84b6db594c-lf44b"] Oct 31 13:26:57.516531 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:57.516513 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a39960-84b6db594c-lf44b" Oct 31 13:26:57.531748 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:57.531712 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a39960\"" Oct 31 13:26:57.533483 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:57.533455 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a39960-84b6db594c-lf44b"] Oct 31 13:26:57.560222 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:57.560185 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a39960-59f9f54fd9-qlvqh"] Oct 31 13:26:57.570760 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:57.570731 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a39960-59f9f54fd9-qlvqh" Oct 31 13:26:57.571612 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:57.571585 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a39960-59f9f54fd9-qlvqh"] Oct 31 13:26:57.578044 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:57.578021 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a39960-dockercfg-8v5zb\"" Oct 31 13:26:57.608611 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:57.608567 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e759d3e-611f-43e3-b520-2a5d34517d91-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a39960-84b6db594c-lf44b\" (UID: \"0e759d3e-611f-43e3-b520-2a5d34517d91\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a39960-84b6db594c-lf44b" Oct 31 13:26:57.608611 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:57.608606 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26dmg\" (UniqueName: \"kubernetes.io/projected/0e759d3e-611f-43e3-b520-2a5d34517d91-kube-api-access-26dmg\") pod \"payload-pre-processing-e2e-inf-a-a39960-84b6db594c-lf44b\" (UID: \"0e759d3e-611f-43e3-b520-2a5d34517d91\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a39960-84b6db594c-lf44b" Oct 31 13:26:57.727718 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:57.725331 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e759d3e-611f-43e3-b520-2a5d34517d91-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a39960-84b6db594c-lf44b\" (UID: \"0e759d3e-611f-43e3-b520-2a5d34517d91\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a39960-84b6db594c-lf44b" Oct 31 13:26:57.727718 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:57.725384 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26dmg\" (UniqueName: \"kubernetes.io/projected/0e759d3e-611f-43e3-b520-2a5d34517d91-kube-api-access-26dmg\") pod \"payload-pre-processing-e2e-inf-a-a39960-84b6db594c-lf44b\" (UID: \"0e759d3e-611f-43e3-b520-2a5d34517d91\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a39960-84b6db594c-lf44b" Oct 31 13:26:57.727718 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:57.725622 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khk6w\" (UniqueName: \"kubernetes.io/projected/2852337d-cf60-41cf-a3c1-5d6ff380c97f-kube-api-access-khk6w\") pod \"payload-processing-e2e-inf-a-a39960-59f9f54fd9-qlvqh\" (UID: \"2852337d-cf60-41cf-a3c1-5d6ff380c97f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a39960-59f9f54fd9-qlvqh" Oct 31 13:26:57.727718 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:57.725749 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2852337d-cf60-41cf-a3c1-5d6ff380c97f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a39960-59f9f54fd9-qlvqh\" (UID: \"2852337d-cf60-41cf-a3c1-5d6ff380c97f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a39960-59f9f54fd9-qlvqh" Oct 31 13:26:57.727718 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:57.726823 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e759d3e-611f-43e3-b520-2a5d34517d91-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a39960-84b6db594c-lf44b\" (UID: \"0e759d3e-611f-43e3-b520-2a5d34517d91\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a39960-84b6db594c-lf44b" Oct 31 13:26:57.785721 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:57.782292 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26dmg\" (UniqueName: \"kubernetes.io/projected/0e759d3e-611f-43e3-b520-2a5d34517d91-kube-api-access-26dmg\") pod \"payload-pre-processing-e2e-inf-a-a39960-84b6db594c-lf44b\" (UID: \"0e759d3e-611f-43e3-b520-2a5d34517d91\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a39960-84b6db594c-lf44b" Oct 31 13:26:57.828663 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:57.828206 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khk6w\" (UniqueName: \"kubernetes.io/projected/2852337d-cf60-41cf-a3c1-5d6ff380c97f-kube-api-access-khk6w\") pod \"payload-processing-e2e-inf-a-a39960-59f9f54fd9-qlvqh\" (UID: \"2852337d-cf60-41cf-a3c1-5d6ff380c97f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a39960-59f9f54fd9-qlvqh" Oct 31 13:26:57.828663 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:57.828317 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2852337d-cf60-41cf-a3c1-5d6ff380c97f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a39960-59f9f54fd9-qlvqh\" (UID: \"2852337d-cf60-41cf-a3c1-5d6ff380c97f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a39960-59f9f54fd9-qlvqh" Oct 31 13:26:57.832770 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:57.829305 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2852337d-cf60-41cf-a3c1-5d6ff380c97f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a39960-59f9f54fd9-qlvqh\" (UID: \"2852337d-cf60-41cf-a3c1-5d6ff380c97f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a39960-59f9f54fd9-qlvqh" Oct 31 13:26:57.872662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:57.869935 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a39960-84b6db594c-lf44b" Oct 31 13:26:57.880130 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:57.879102 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khk6w\" (UniqueName: \"kubernetes.io/projected/2852337d-cf60-41cf-a3c1-5d6ff380c97f-kube-api-access-khk6w\") pod \"payload-processing-e2e-inf-a-a39960-59f9f54fd9-qlvqh\" (UID: \"2852337d-cf60-41cf-a3c1-5d6ff380c97f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a39960-59f9f54fd9-qlvqh" Oct 31 13:26:57.890701 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:57.887594 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a39960-59f9f54fd9-qlvqh" Oct 31 13:26:58.285395 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:58.285360 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a39960-84b6db594c-lf44b"] Oct 31 13:26:58.286015 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:26:58.285986 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e759d3e_611f_43e3_b520_2a5d34517d91.slice/crio-4e1150c5b9dfe2b229e455fe4bf890ce0f4ffe67357a2085596b2948b3233dce WatchSource:0}: Error finding container 4e1150c5b9dfe2b229e455fe4bf890ce0f4ffe67357a2085596b2948b3233dce: Status 404 returned error can't find the container with id 4e1150c5b9dfe2b229e455fe4bf890ce0f4ffe67357a2085596b2948b3233dce Oct 31 13:26:58.293264 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:26:58.293223 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2852337d_cf60_41cf_a3c1_5d6ff380c97f.slice/crio-6dc8d807f1d709450b0361602c46288ee3db177e4a254c0c41bb278757404bd2 WatchSource:0}: Error finding container 6dc8d807f1d709450b0361602c46288ee3db177e4a254c0c41bb278757404bd2: Status 404 returned error can't find the container with id 6dc8d807f1d709450b0361602c46288ee3db177e4a254c0c41bb278757404bd2 Oct 31 13:26:58.295211 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:58.295172 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a39960-59f9f54fd9-qlvqh"] Oct 31 13:26:58.777466 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:58.777428 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a39960-59f9f54fd9-qlvqh" event={"ID":"2852337d-cf60-41cf-a3c1-5d6ff380c97f","Type":"ContainerStarted","Data":"f303f4be3db8255547408ac0caa77b0d7264643eeb22b30dc93ad4de888f3334"} Oct 31 13:26:58.777997 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:58.777538 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a39960-59f9f54fd9-qlvqh" event={"ID":"2852337d-cf60-41cf-a3c1-5d6ff380c97f","Type":"ContainerStarted","Data":"6dc8d807f1d709450b0361602c46288ee3db177e4a254c0c41bb278757404bd2"} Oct 31 13:26:58.777997 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:58.777568 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a39960-59f9f54fd9-qlvqh" Oct 31 13:26:58.779760 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:58.779731 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a39960-84b6db594c-lf44b" event={"ID":"0e759d3e-611f-43e3-b520-2a5d34517d91","Type":"ContainerStarted","Data":"c205c54e0e7cd8a502a036b71bcf839981d0dc6cadeee50f5d7bd35574dd6857"} Oct 31 13:26:58.779920 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:58.779785 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a39960-84b6db594c-lf44b" event={"ID":"0e759d3e-611f-43e3-b520-2a5d34517d91","Type":"ContainerStarted","Data":"4e1150c5b9dfe2b229e455fe4bf890ce0f4ffe67357a2085596b2948b3233dce"} Oct 31 13:26:58.779920 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:58.779832 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a39960-84b6db594c-lf44b" Oct 31 13:26:58.799368 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:58.799314 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a39960-59f9f54fd9-qlvqh" podStartSLOduration=1.7992994329999998 podStartE2EDuration="1.799299433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:26:58.79595482 +0000 UTC m=+3269.503541117" watchObservedRunningTime="2026-10-31 13:26:58.799299433 +0000 UTC m=+3269.506885729" Oct 31 13:26:58.821104 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:26:58.821055 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a39960-84b6db594c-lf44b" podStartSLOduration=1.821041254 podStartE2EDuration="1.821041254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:26:58.81766062 +0000 UTC m=+3269.525246912" watchObservedRunningTime="2026-10-31 13:26:58.821041254 +0000 UTC m=+3269.528627549" Oct 31 13:27:08.931666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:08.929930 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg"] Oct 31 13:27:08.945454 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:08.945424 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv"] Oct 31 13:27:08.959996 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:08.950214 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" Oct 31 13:27:08.959996 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:08.958829 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-5dbd39-openshift-default-dockercfg-l8k75\"" Oct 31 13:27:08.967888 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:08.967860 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg"] Oct 31 13:27:08.968180 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:08.968147 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv"] Oct 31 13:27:08.968310 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:08.968268 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" Oct 31 13:27:09.013864 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.013829 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9410570-a345-441e-9978-2f9ce48d7a09-istiod-ca-cert\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg\" (UID: \"a9410570-a345-441e-9978-2f9ce48d7a09\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" Oct 31 13:27:09.014025 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.013885 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6791b24c-6569-46a0-af4c-3b7b7b356a9d-istio-data\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv\" (UID: \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" Oct 31 13:27:09.014025 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.014001 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9410570-a345-441e-9978-2f9ce48d7a09-credential-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg\" (UID: \"a9410570-a345-441e-9978-2f9ce48d7a09\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" Oct 31 13:27:09.014342 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.014073 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9410570-a345-441e-9978-2f9ce48d7a09-istio-envoy\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg\" (UID: \"a9410570-a345-441e-9978-2f9ce48d7a09\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" Oct 31 13:27:09.014342 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.014115 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b425g\" (UniqueName: \"kubernetes.io/projected/a9410570-a345-441e-9978-2f9ce48d7a09-kube-api-access-b425g\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg\" (UID: \"a9410570-a345-441e-9978-2f9ce48d7a09\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" Oct 31 13:27:09.014342 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.014161 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6791b24c-6569-46a0-af4c-3b7b7b356a9d-istio-podinfo\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv\" (UID: \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" Oct 31 13:27:09.014342 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.014207 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6791b24c-6569-46a0-af4c-3b7b7b356a9d-istiod-ca-cert\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv\" (UID: \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" Oct 31 13:27:09.014342 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.014254 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6791b24c-6569-46a0-af4c-3b7b7b356a9d-istio-envoy\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv\" (UID: \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" Oct 31 13:27:09.015039 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.014349 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9410570-a345-441e-9978-2f9ce48d7a09-istio-data\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg\" (UID: \"a9410570-a345-441e-9978-2f9ce48d7a09\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" Oct 31 13:27:09.015039 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.014410 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6791b24c-6569-46a0-af4c-3b7b7b356a9d-istio-token\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv\" (UID: \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" Oct 31 13:27:09.015039 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.014436 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6791b24c-6569-46a0-af4c-3b7b7b356a9d-workload-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv\" (UID: \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" Oct 31 13:27:09.015039 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.014479 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9410570-a345-441e-9978-2f9ce48d7a09-istio-token\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg\" (UID: \"a9410570-a345-441e-9978-2f9ce48d7a09\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" Oct 31 13:27:09.015039 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.014533 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9410570-a345-441e-9978-2f9ce48d7a09-istio-podinfo\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg\" (UID: \"a9410570-a345-441e-9978-2f9ce48d7a09\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" Oct 31 13:27:09.015039 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.014579 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9410570-a345-441e-9978-2f9ce48d7a09-workload-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg\" (UID: \"a9410570-a345-441e-9978-2f9ce48d7a09\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" Oct 31 13:27:09.015039 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.014601 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9410570-a345-441e-9978-2f9ce48d7a09-workload-certs\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg\" (UID: \"a9410570-a345-441e-9978-2f9ce48d7a09\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" Oct 31 13:27:09.015039 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.014733 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6791b24c-6569-46a0-af4c-3b7b7b356a9d-credential-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv\" (UID: \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" Oct 31 13:27:09.015039 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.014833 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6791b24c-6569-46a0-af4c-3b7b7b356a9d-workload-certs\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv\" (UID: \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" Oct 31 13:27:09.015039 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.014865 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7gk7\" (UniqueName: \"kubernetes.io/projected/6791b24c-6569-46a0-af4c-3b7b7b356a9d-kube-api-access-v7gk7\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv\" (UID: \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" Oct 31 13:27:09.116451 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.116417 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9410570-a345-441e-9978-2f9ce48d7a09-istio-envoy\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg\" (UID: \"a9410570-a345-441e-9978-2f9ce48d7a09\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" Oct 31 13:27:09.116634 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.116457 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b425g\" (UniqueName: \"kubernetes.io/projected/a9410570-a345-441e-9978-2f9ce48d7a09-kube-api-access-b425g\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg\" (UID: \"a9410570-a345-441e-9978-2f9ce48d7a09\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" Oct 31 13:27:09.116634 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.116489 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6791b24c-6569-46a0-af4c-3b7b7b356a9d-istio-podinfo\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv\" (UID: \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" Oct 31 13:27:09.116634 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.116519 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6791b24c-6569-46a0-af4c-3b7b7b356a9d-istiod-ca-cert\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv\" (UID: \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" Oct 31 13:27:09.116634 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.116548 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6791b24c-6569-46a0-af4c-3b7b7b356a9d-istio-envoy\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv\" (UID: \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" Oct 31 13:27:09.116634 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.116609 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9410570-a345-441e-9978-2f9ce48d7a09-istio-data\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg\" (UID: \"a9410570-a345-441e-9978-2f9ce48d7a09\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" Oct 31 13:27:09.116945 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.116662 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6791b24c-6569-46a0-af4c-3b7b7b356a9d-istio-token\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv\" (UID: \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" Oct 31 13:27:09.116945 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.116689 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6791b24c-6569-46a0-af4c-3b7b7b356a9d-workload-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv\" (UID: \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" Oct 31 13:27:09.116945 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.116729 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9410570-a345-441e-9978-2f9ce48d7a09-istio-token\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg\" (UID: \"a9410570-a345-441e-9978-2f9ce48d7a09\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" Oct 31 13:27:09.116945 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.116780 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9410570-a345-441e-9978-2f9ce48d7a09-istio-podinfo\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg\" (UID: \"a9410570-a345-441e-9978-2f9ce48d7a09\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" Oct 31 13:27:09.116945 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.116824 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9410570-a345-441e-9978-2f9ce48d7a09-workload-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg\" (UID: \"a9410570-a345-441e-9978-2f9ce48d7a09\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" Oct 31 13:27:09.116945 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.116849 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9410570-a345-441e-9978-2f9ce48d7a09-workload-certs\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg\" (UID: \"a9410570-a345-441e-9978-2f9ce48d7a09\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" Oct 31 13:27:09.116945 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.116932 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6791b24c-6569-46a0-af4c-3b7b7b356a9d-credential-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv\" (UID: \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" Oct 31 13:27:09.117343 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.117022 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6791b24c-6569-46a0-af4c-3b7b7b356a9d-workload-certs\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv\" (UID: \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" Oct 31 13:27:09.117343 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.117029 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9410570-a345-441e-9978-2f9ce48d7a09-istio-data\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg\" (UID: \"a9410570-a345-441e-9978-2f9ce48d7a09\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" Oct 31 13:27:09.117343 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.117052 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7gk7\" (UniqueName: \"kubernetes.io/projected/6791b24c-6569-46a0-af4c-3b7b7b356a9d-kube-api-access-v7gk7\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv\" (UID: \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" Oct 31 13:27:09.117343 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.117131 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9410570-a345-441e-9978-2f9ce48d7a09-istiod-ca-cert\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg\" (UID: \"a9410570-a345-441e-9978-2f9ce48d7a09\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" Oct 31 13:27:09.117343 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.117140 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6791b24c-6569-46a0-af4c-3b7b7b356a9d-workload-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv\" (UID: \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" Oct 31 13:27:09.117568 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.117339 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9410570-a345-441e-9978-2f9ce48d7a09-workload-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg\" (UID: \"a9410570-a345-441e-9978-2f9ce48d7a09\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" Oct 31 13:27:09.117568 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.117355 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6791b24c-6569-46a0-af4c-3b7b7b356a9d-istio-data\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv\" (UID: \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" Oct 31 13:27:09.117568 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.117422 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9410570-a345-441e-9978-2f9ce48d7a09-credential-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg\" (UID: \"a9410570-a345-441e-9978-2f9ce48d7a09\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" Oct 31 13:27:09.117761 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.117740 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9410570-a345-441e-9978-2f9ce48d7a09-workload-certs\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg\" (UID: \"a9410570-a345-441e-9978-2f9ce48d7a09\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" Oct 31 13:27:09.117821 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.117796 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6791b24c-6569-46a0-af4c-3b7b7b356a9d-credential-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv\" (UID: \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" Oct 31 13:27:09.117881 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.117830 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6791b24c-6569-46a0-af4c-3b7b7b356a9d-workload-certs\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv\" (UID: \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" Oct 31 13:27:09.118027 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.117839 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9410570-a345-441e-9978-2f9ce48d7a09-credential-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg\" (UID: \"a9410570-a345-441e-9978-2f9ce48d7a09\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" Oct 31 13:27:09.118027 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.118013 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6791b24c-6569-46a0-af4c-3b7b7b356a9d-istio-data\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv\" (UID: \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" Oct 31 13:27:09.118218 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.118086 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6791b24c-6569-46a0-af4c-3b7b7b356a9d-istiod-ca-cert\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv\" (UID: \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" Oct 31 13:27:09.118496 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.118469 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9410570-a345-441e-9978-2f9ce48d7a09-istiod-ca-cert\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg\" (UID: \"a9410570-a345-441e-9978-2f9ce48d7a09\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" Oct 31 13:27:09.119271 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.119248 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9410570-a345-441e-9978-2f9ce48d7a09-istio-envoy\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg\" (UID: \"a9410570-a345-441e-9978-2f9ce48d7a09\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" Oct 31 13:27:09.119895 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.119871 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6791b24c-6569-46a0-af4c-3b7b7b356a9d-istio-podinfo\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv\" (UID: \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" Oct 31 13:27:09.120003 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.119977 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6791b24c-6569-46a0-af4c-3b7b7b356a9d-istio-envoy\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv\" (UID: \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" Oct 31 13:27:09.120257 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.120236 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9410570-a345-441e-9978-2f9ce48d7a09-istio-podinfo\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg\" (UID: \"a9410570-a345-441e-9978-2f9ce48d7a09\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" Oct 31 13:27:09.131478 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.131448 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9410570-a345-441e-9978-2f9ce48d7a09-istio-token\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg\" (UID: \"a9410570-a345-441e-9978-2f9ce48d7a09\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" Oct 31 13:27:09.132011 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.131986 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b425g\" (UniqueName: \"kubernetes.io/projected/a9410570-a345-441e-9978-2f9ce48d7a09-kube-api-access-b425g\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg\" (UID: \"a9410570-a345-441e-9978-2f9ce48d7a09\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" Oct 31 13:27:09.132701 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.132682 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6791b24c-6569-46a0-af4c-3b7b7b356a9d-istio-token\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv\" (UID: \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" Oct 31 13:27:09.132788 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.132716 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7gk7\" (UniqueName: \"kubernetes.io/projected/6791b24c-6569-46a0-af4c-3b7b7b356a9d-kube-api-access-v7gk7\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv\" (UID: \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" Oct 31 13:27:09.310915 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.310831 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" Oct 31 13:27:09.318767 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.318745 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" Oct 31 13:27:09.621585 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:27:09.621540 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9410570_a345_441e_9978_2f9ce48d7a09.slice/crio-4a1119f5dead40badd8cbdc210e4e77dc4d4ca5d2998693762e34054bf920d69 WatchSource:0}: Error finding container 4a1119f5dead40badd8cbdc210e4e77dc4d4ca5d2998693762e34054bf920d69: Status 404 returned error can't find the container with id 4a1119f5dead40badd8cbdc210e4e77dc4d4ca5d2998693762e34054bf920d69 Oct 31 13:27:09.621759 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.621698 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg"] Oct 31 13:27:09.625982 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.625947 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:27:09.626078 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.626018 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:27:09.626078 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.626048 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:27:09.660863 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.660781 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv"] Oct 31 13:27:09.668211 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:27:09.668184 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6791b24c_6569_46a0_af4c_3b7b7b356a9d.slice/crio-32f4b40e7ca1f5e1a08109c60f1af2a9ff21e0999c8861b68284fd415938d3d6 WatchSource:0}: Error finding container 32f4b40e7ca1f5e1a08109c60f1af2a9ff21e0999c8861b68284fd415938d3d6: Status 404 returned error can't find the container with id 32f4b40e7ca1f5e1a08109c60f1af2a9ff21e0999c8861b68284fd415938d3d6 Oct 31 13:27:09.672332 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.672291 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:27:09.672408 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.672391 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:27:09.672472 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.672433 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:27:09.806298 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.806271 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a39960-84b6db594c-lf44b" Oct 31 13:27:09.810764 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:09.809955 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a39960-59f9f54fd9-qlvqh" Oct 31 13:27:10.032804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:10.032733 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" event={"ID":"a9410570-a345-441e-9978-2f9ce48d7a09","Type":"ContainerStarted","Data":"557ce5b49869163d06f9cba2506416807d41c914db847c420e7ca9bb17aa44f7"} Oct 31 13:27:10.033784 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:10.033752 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" event={"ID":"a9410570-a345-441e-9978-2f9ce48d7a09","Type":"ContainerStarted","Data":"4a1119f5dead40badd8cbdc210e4e77dc4d4ca5d2998693762e34054bf920d69"} Oct 31 13:27:10.036507 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:10.036481 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" event={"ID":"6791b24c-6569-46a0-af4c-3b7b7b356a9d","Type":"ContainerStarted","Data":"3390338bc12883f630fca0b2e5329f29428205b72b1d46e984c6ff4a5edbf0bb"} Oct 31 13:27:10.036627 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:10.036530 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" event={"ID":"6791b24c-6569-46a0-af4c-3b7b7b356a9d","Type":"ContainerStarted","Data":"32f4b40e7ca1f5e1a08109c60f1af2a9ff21e0999c8861b68284fd415938d3d6"} Oct 31 13:27:10.060891 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:10.060837 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" podStartSLOduration=2.060821784 podStartE2EDuration="2.060821784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:27:10.058241634 +0000 UTC m=+3280.765827929" watchObservedRunningTime="2026-10-31 13:27:10.060821784 +0000 UTC m=+3280.768408135" Oct 31 13:27:10.088769 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:10.088706 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" podStartSLOduration=2.088689743 podStartE2EDuration="2.088689743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:27:10.084072099 +0000 UTC m=+3280.791658394" watchObservedRunningTime="2026-10-31 13:27:10.088689743 +0000 UTC m=+3280.796276037" Oct 31 13:27:10.313928 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:10.311518 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" Oct 31 13:27:10.313928 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:10.313361 5693 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body= Oct 31 13:27:10.313928 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:10.313408 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" podUID="a9410570-a345-441e-9978-2f9ce48d7a09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" Oct 31 13:27:10.319581 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:10.319553 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" Oct 31 13:27:10.321045 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:10.321022 5693 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Oct 31 13:27:10.321177 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:10.321073 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" podUID="6791b24c-6569-46a0-af4c-3b7b7b356a9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Oct 31 13:27:11.316666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:11.315922 5693 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body= Oct 31 13:27:11.316666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:11.315983 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" podUID="a9410570-a345-441e-9978-2f9ce48d7a09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" Oct 31 13:27:11.324666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:11.323064 5693 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Oct 31 13:27:11.324666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:11.323122 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" podUID="6791b24c-6569-46a0-af4c-3b7b7b356a9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Oct 31 13:27:12.319667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:12.314269 5693 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body= Oct 31 13:27:12.319667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:12.314333 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" podUID="a9410570-a345-441e-9978-2f9ce48d7a09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" Oct 31 13:27:12.331661 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:12.329532 5693 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Oct 31 13:27:12.331661 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:12.329588 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" podUID="6791b24c-6569-46a0-af4c-3b7b7b356a9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Oct 31 13:27:13.313082 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:13.313044 5693 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body= Oct 31 13:27:13.314741 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:13.314706 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" podUID="a9410570-a345-441e-9978-2f9ce48d7a09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" Oct 31 13:27:13.324041 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:13.324009 5693 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Oct 31 13:27:13.325074 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:13.325047 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" podUID="6791b24c-6569-46a0-af4c-3b7b7b356a9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Oct 31 13:27:14.321668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:14.312092 5693 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body= Oct 31 13:27:14.321668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:14.312161 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" podUID="a9410570-a345-441e-9978-2f9ce48d7a09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" Oct 31 13:27:14.327666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:14.327589 5693 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Oct 31 13:27:14.328201 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:14.327670 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" podUID="6791b24c-6569-46a0-af4c-3b7b7b356a9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Oct 31 13:27:15.317667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:15.315042 5693 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body= Oct 31 13:27:15.317667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:15.315121 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" podUID="a9410570-a345-441e-9978-2f9ce48d7a09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" Oct 31 13:27:15.325717 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:15.322341 5693 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Oct 31 13:27:15.325717 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:15.322396 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" podUID="6791b24c-6569-46a0-af4c-3b7b7b356a9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Oct 31 13:27:16.321801 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:16.321765 5693 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body= Oct 31 13:27:16.328432 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:16.327906 5693 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Oct 31 13:27:16.328432 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:16.327992 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" podUID="a9410570-a345-441e-9978-2f9ce48d7a09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" Oct 31 13:27:16.328432 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:16.328058 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" podUID="6791b24c-6569-46a0-af4c-3b7b7b356a9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Oct 31 13:27:17.317801 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:17.317769 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" Oct 31 13:27:17.318249 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:17.318224 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" Oct 31 13:27:17.318982 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:17.318955 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" Oct 31 13:27:17.323732 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:17.323707 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" Oct 31 13:27:17.324107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:17.323980 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" Oct 31 13:27:17.324620 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:27:17.324602 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" Oct 31 13:27:18.691085 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:27:18.691049 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:27:48.691732 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:27:48.691693 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:28:03.872332 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:03.872299 5693 generic.go:356] "Generic (PLEG): container finished" podID="9735d8ba-9f3e-434c-bb9f-5876f11d9336" containerID="523ae8670acc553ccb1385c8cfcad2524d46d9d019df1f3545ff8e15a0d6416e" exitCode=0 Oct 31 13:28:03.872860 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:03.872372 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" event={"ID":"9735d8ba-9f3e-434c-bb9f-5876f11d9336","Type":"ContainerDied","Data":"523ae8670acc553ccb1385c8cfcad2524d46d9d019df1f3545ff8e15a0d6416e"} Oct 31 13:28:03.872960 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:03.872880 5693 scope.go:122] "RemoveContainer" containerID="523ae8670acc553ccb1385c8cfcad2524d46d9d019df1f3545ff8e15a0d6416e" Oct 31 13:28:03.875121 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:03.875084 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:28:03.875366 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:03.875335 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:28:03.875467 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:03.875404 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:28:04.886424 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:04.886379 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" event={"ID":"9735d8ba-9f3e-434c-bb9f-5876f11d9336","Type":"ContainerStarted","Data":"4388103aa62448775770f44a8297eca0e2a7b6f704fbcf1ceed08ac1dfdd587c"} Oct 31 13:28:05.091047 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:05.091011 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" Oct 31 13:28:05.096998 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:05.096964 5693 patch_prober.go:29] interesting pod/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Oct 31 13:28:05.097270 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:05.097237 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" podUID="9735d8ba-9f3e-434c-bb9f-5876f11d9336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Oct 31 13:28:06.086003 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:06.085963 5693 patch_prober.go:29] interesting pod/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Oct 31 13:28:06.086621 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:06.086588 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" podUID="9735d8ba-9f3e-434c-bb9f-5876f11d9336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Oct 31 13:28:07.096087 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:07.096049 5693 patch_prober.go:29] interesting pod/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Oct 31 13:28:07.096853 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:07.096817 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" podUID="9735d8ba-9f3e-434c-bb9f-5876f11d9336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Oct 31 13:28:08.094666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:08.084983 5693 patch_prober.go:29] interesting pod/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Oct 31 13:28:08.094666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:08.085058 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" podUID="9735d8ba-9f3e-434c-bb9f-5876f11d9336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Oct 31 13:28:09.096727 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:09.089061 5693 patch_prober.go:29] interesting pod/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Oct 31 13:28:09.096727 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:09.089263 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" podUID="9735d8ba-9f3e-434c-bb9f-5876f11d9336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Oct 31 13:28:09.697668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:09.697428 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" Oct 31 13:28:10.088978 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:10.088885 5693 patch_prober.go:29] interesting pod/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Oct 31 13:28:10.096800 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:10.096752 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" podUID="9735d8ba-9f3e-434c-bb9f-5876f11d9336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Oct 31 13:28:11.096159 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:11.096128 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" Oct 31 13:28:11.097158 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:11.097137 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" Oct 31 13:28:18.724822 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:28:18.724784 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:28:48.733792 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:28:48.733756 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:28:53.230633 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:53.230600 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t"] Oct 31 13:28:53.231208 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:53.230839 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" podUID="595d267e-2643-4570-b24c-8b03e9e34705" containerName="istio-proxy" containerID="cri-o://4544866727db81303852e515e9c8b14d5f745c1d44756019a86065e3890ee087" gracePeriod=30 Oct 31 13:28:55.371182 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:55.371134 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4"] Oct 31 13:28:55.371800 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:55.371467 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" podUID="a022831f-b584-43fe-91a7-7f541c066ace" containerName="istio-proxy" containerID="cri-o://e2d6bcbc7bf35b16702d1f2705a242775af5ab8cd6e0569ad46313a327d6b8b2" gracePeriod=30 Oct 31 13:28:58.582755 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.582722 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" Oct 31 13:28:58.628674 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.628625 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp"] Oct 31 13:28:58.629669 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.629629 5693 cpu_manager.go:414] "containerMap: removing container" podUID="595d267e-2643-4570-b24c-8b03e9e34705" containerName="istio-proxy" Oct 31 13:28:58.629669 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.629668 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="595d267e-2643-4570-b24c-8b03e9e34705" containerName="istio-proxy" Oct 31 13:28:58.629877 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.629706 5693 cpu_manager.go:414] "containerMap: removing container" podUID="595d267e-2643-4570-b24c-8b03e9e34705" containerName="istio-proxy" Oct 31 13:28:58.629877 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.629714 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="595d267e-2643-4570-b24c-8b03e9e34705" containerName="istio-proxy" Oct 31 13:28:58.630022 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.630006 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="595d267e-2643-4570-b24c-8b03e9e34705" containerName="istio-proxy" Oct 31 13:28:58.630096 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.630032 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="595d267e-2643-4570-b24c-8b03e9e34705" containerName="istio-proxy" Oct 31 13:28:58.635296 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.635255 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" Oct 31 13:28:58.660446 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.660415 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp"] Oct 31 13:28:58.684966 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.684933 5693 generic.go:356] "Generic (PLEG): container finished" podID="595d267e-2643-4570-b24c-8b03e9e34705" containerID="4544866727db81303852e515e9c8b14d5f745c1d44756019a86065e3890ee087" exitCode=0 Oct 31 13:28:58.685173 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.684991 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" Oct 31 13:28:58.685173 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.685015 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" event={"ID":"595d267e-2643-4570-b24c-8b03e9e34705","Type":"ContainerDied","Data":"4544866727db81303852e515e9c8b14d5f745c1d44756019a86065e3890ee087"} Oct 31 13:28:58.685173 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.685074 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t" event={"ID":"595d267e-2643-4570-b24c-8b03e9e34705","Type":"ContainerDied","Data":"52257c23a4a7ca6090cf083ad1df629ce02e895d1214e3a0e76f904cc1f4d65a"} Oct 31 13:28:58.685173 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.685105 5693 scope.go:122] "RemoveContainer" containerID="4544866727db81303852e515e9c8b14d5f745c1d44756019a86065e3890ee087" Oct 31 13:28:58.698383 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.698359 5693 scope.go:122] "RemoveContainer" containerID="2cf22199be26c300893f693a59db956954a1edc575787dbbd7775f19c8765834" Oct 31 13:28:58.710069 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.710043 5693 scope.go:122] "RemoveContainer" containerID="4544866727db81303852e515e9c8b14d5f745c1d44756019a86065e3890ee087" Oct 31 13:28:58.710503 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:28:58.710480 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4544866727db81303852e515e9c8b14d5f745c1d44756019a86065e3890ee087\": container with ID starting with 4544866727db81303852e515e9c8b14d5f745c1d44756019a86065e3890ee087 not found: ID does not exist" containerID="4544866727db81303852e515e9c8b14d5f745c1d44756019a86065e3890ee087" Oct 31 13:28:58.710603 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.710514 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4544866727db81303852e515e9c8b14d5f745c1d44756019a86065e3890ee087"} err="failed to get container status \"4544866727db81303852e515e9c8b14d5f745c1d44756019a86065e3890ee087\": rpc error: code = NotFound desc = could not find container \"4544866727db81303852e515e9c8b14d5f745c1d44756019a86065e3890ee087\": container with ID starting with 4544866727db81303852e515e9c8b14d5f745c1d44756019a86065e3890ee087 not found: ID does not exist" Oct 31 13:28:58.710603 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.710540 5693 scope.go:122] "RemoveContainer" containerID="2cf22199be26c300893f693a59db956954a1edc575787dbbd7775f19c8765834" Oct 31 13:28:58.710873 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:28:58.710851 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cf22199be26c300893f693a59db956954a1edc575787dbbd7775f19c8765834\": container with ID starting with 2cf22199be26c300893f693a59db956954a1edc575787dbbd7775f19c8765834 not found: ID does not exist" containerID="2cf22199be26c300893f693a59db956954a1edc575787dbbd7775f19c8765834" Oct 31 13:28:58.710936 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.710879 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cf22199be26c300893f693a59db956954a1edc575787dbbd7775f19c8765834"} err="failed to get container status \"2cf22199be26c300893f693a59db956954a1edc575787dbbd7775f19c8765834\": rpc error: code = NotFound desc = could not find container \"2cf22199be26c300893f693a59db956954a1edc575787dbbd7775f19c8765834\": container with ID starting with 2cf22199be26c300893f693a59db956954a1edc575787dbbd7775f19c8765834 not found: ID does not exist" Oct 31 13:28:58.731074 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.731039 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/595d267e-2643-4570-b24c-8b03e9e34705-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/595d267e-2643-4570-b24c-8b03e9e34705-istio-podinfo\") pod \"595d267e-2643-4570-b24c-8b03e9e34705\" (UID: \"595d267e-2643-4570-b24c-8b03e9e34705\") " Oct 31 13:28:58.731398 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.731376 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/595d267e-2643-4570-b24c-8b03e9e34705-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/595d267e-2643-4570-b24c-8b03e9e34705-workload-certs\") pod \"595d267e-2643-4570-b24c-8b03e9e34705\" (UID: \"595d267e-2643-4570-b24c-8b03e9e34705\") " Oct 31 13:28:58.731713 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.731689 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/595d267e-2643-4570-b24c-8b03e9e34705-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/595d267e-2643-4570-b24c-8b03e9e34705-workload-socket\") pod \"595d267e-2643-4570-b24c-8b03e9e34705\" (UID: \"595d267e-2643-4570-b24c-8b03e9e34705\") " Oct 31 13:28:58.731933 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.731914 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/595d267e-2643-4570-b24c-8b03e9e34705-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/595d267e-2643-4570-b24c-8b03e9e34705-credential-socket\") pod \"595d267e-2643-4570-b24c-8b03e9e34705\" (UID: \"595d267e-2643-4570-b24c-8b03e9e34705\") " Oct 31 13:28:58.732167 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.732147 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/595d267e-2643-4570-b24c-8b03e9e34705-kube-api-access-vjzc4\" (UniqueName: \"kubernetes.io/projected/595d267e-2643-4570-b24c-8b03e9e34705-kube-api-access-vjzc4\") pod \"595d267e-2643-4570-b24c-8b03e9e34705\" (UID: \"595d267e-2643-4570-b24c-8b03e9e34705\") " Oct 31 13:28:58.735020 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.732187 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/595d267e-2643-4570-b24c-8b03e9e34705-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/595d267e-2643-4570-b24c-8b03e9e34705-istiod-ca-cert\") pod \"595d267e-2643-4570-b24c-8b03e9e34705\" (UID: \"595d267e-2643-4570-b24c-8b03e9e34705\") " Oct 31 13:28:58.735020 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.734973 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/595d267e-2643-4570-b24c-8b03e9e34705-istiod-ca-cert" pod "595d267e-2643-4570-b24c-8b03e9e34705" (UID: "595d267e-2643-4570-b24c-8b03e9e34705"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:28:58.738102 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.738078 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/595d267e-2643-4570-b24c-8b03e9e34705-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/595d267e-2643-4570-b24c-8b03e9e34705-istio-data\") pod \"595d267e-2643-4570-b24c-8b03e9e34705\" (UID: \"595d267e-2643-4570-b24c-8b03e9e34705\") " Oct 31 13:28:58.738366 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.738258 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595d267e-2643-4570-b24c-8b03e9e34705-workload-socket" pod "595d267e-2643-4570-b24c-8b03e9e34705" (UID: "595d267e-2643-4570-b24c-8b03e9e34705"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:28:58.738497 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.738396 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595d267e-2643-4570-b24c-8b03e9e34705-credential-socket" pod "595d267e-2643-4570-b24c-8b03e9e34705" (UID: "595d267e-2643-4570-b24c-8b03e9e34705"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:28:58.738497 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.738418 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595d267e-2643-4570-b24c-8b03e9e34705-workload-certs" pod "595d267e-2643-4570-b24c-8b03e9e34705" (UID: "595d267e-2643-4570-b24c-8b03e9e34705"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:28:58.738497 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.738476 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595d267e-2643-4570-b24c-8b03e9e34705-istio-data" pod "595d267e-2643-4570-b24c-8b03e9e34705" (UID: "595d267e-2643-4570-b24c-8b03e9e34705"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:28:58.739070 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.738518 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/595d267e-2643-4570-b24c-8b03e9e34705-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/595d267e-2643-4570-b24c-8b03e9e34705-istio-envoy\") pod \"595d267e-2643-4570-b24c-8b03e9e34705\" (UID: \"595d267e-2643-4570-b24c-8b03e9e34705\") " Oct 31 13:28:58.739070 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.738873 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/595d267e-2643-4570-b24c-8b03e9e34705-istio-token\" (UniqueName: \"kubernetes.io/projected/595d267e-2643-4570-b24c-8b03e9e34705-istio-token\") pod \"595d267e-2643-4570-b24c-8b03e9e34705\" (UID: \"595d267e-2643-4570-b24c-8b03e9e34705\") " Oct 31 13:28:58.739391 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.739365 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/595d267e-2643-4570-b24c-8b03e9e34705-kube-api-access-vjzc4" pod "595d267e-2643-4570-b24c-8b03e9e34705" (UID: "595d267e-2643-4570-b24c-8b03e9e34705"). InnerVolumeSpecName "kube-api-access-vjzc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:28:58.739522 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.739446 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-workload-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp\" (UID: \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" Oct 31 13:28:58.739522 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.739478 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-istio-token\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp\" (UID: \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" Oct 31 13:28:58.739788 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.739759 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wktnv\" (UniqueName: \"kubernetes.io/projected/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-kube-api-access-wktnv\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp\" (UID: \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" Oct 31 13:28:58.739872 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.739829 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-credential-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp\" (UID: \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" Oct 31 13:28:58.740244 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.740159 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-istiod-ca-cert\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp\" (UID: \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" Oct 31 13:28:58.740434 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.740416 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-workload-certs\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp\" (UID: \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" Oct 31 13:28:58.740683 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.740665 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-istio-podinfo\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp\" (UID: \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" Oct 31 13:28:58.740841 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.740803 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/595d267e-2643-4570-b24c-8b03e9e34705-istio-podinfo" pod "595d267e-2643-4570-b24c-8b03e9e34705" (UID: "595d267e-2643-4570-b24c-8b03e9e34705"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:28:58.740952 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.740864 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-istio-envoy\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp\" (UID: \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" Oct 31 13:28:58.741047 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.740951 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-istio-data\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp\" (UID: \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" Oct 31 13:28:58.741180 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.741161 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/595d267e-2643-4570-b24c-8b03e9e34705-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:58.741225 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.741189 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/595d267e-2643-4570-b24c-8b03e9e34705-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:58.741225 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.741206 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/595d267e-2643-4570-b24c-8b03e9e34705-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:58.741225 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.741220 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/595d267e-2643-4570-b24c-8b03e9e34705-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:58.741359 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.741267 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjzc4\" (UniqueName: \"kubernetes.io/projected/595d267e-2643-4570-b24c-8b03e9e34705-kube-api-access-vjzc4\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:58.741359 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.741290 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/595d267e-2643-4570-b24c-8b03e9e34705-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:58.741359 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.741303 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/595d267e-2643-4570-b24c-8b03e9e34705-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:58.741952 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.741925 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/595d267e-2643-4570-b24c-8b03e9e34705-istio-token" pod "595d267e-2643-4570-b24c-8b03e9e34705" (UID: "595d267e-2643-4570-b24c-8b03e9e34705"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:28:58.742750 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.742727 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595d267e-2643-4570-b24c-8b03e9e34705-istio-envoy" pod "595d267e-2643-4570-b24c-8b03e9e34705" (UID: "595d267e-2643-4570-b24c-8b03e9e34705"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:28:58.842564 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.842470 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wktnv\" (UniqueName: \"kubernetes.io/projected/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-kube-api-access-wktnv\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp\" (UID: \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" Oct 31 13:28:58.842564 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.842537 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-credential-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp\" (UID: \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" Oct 31 13:28:58.842876 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.842666 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-istiod-ca-cert\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp\" (UID: \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" Oct 31 13:28:58.842876 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.842699 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-workload-certs\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp\" (UID: \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" Oct 31 13:28:58.842876 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.842736 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-istio-podinfo\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp\" (UID: \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" Oct 31 13:28:58.842876 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.842813 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-istio-envoy\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp\" (UID: \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" Oct 31 13:28:58.843143 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.842894 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-istio-data\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp\" (UID: \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" Oct 31 13:28:58.843143 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.843047 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-workload-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp\" (UID: \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" Oct 31 13:28:58.843143 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.843077 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-istio-token\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp\" (UID: \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" Oct 31 13:28:58.843143 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.843123 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-credential-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp\" (UID: \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" Oct 31 13:28:58.843354 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.843198 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-workload-certs\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp\" (UID: \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" Oct 31 13:28:58.843429 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.843353 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/595d267e-2643-4570-b24c-8b03e9e34705-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:58.843429 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.843379 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/595d267e-2643-4570-b24c-8b03e9e34705-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:58.843579 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.843555 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-workload-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp\" (UID: \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" Oct 31 13:28:58.843763 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.843740 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-istio-data\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp\" (UID: \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" Oct 31 13:28:58.843914 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.843894 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-istiod-ca-cert\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp\" (UID: \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" Oct 31 13:28:58.845383 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.845365 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-istio-envoy\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp\" (UID: \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" Oct 31 13:28:58.845778 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.845758 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-istio-podinfo\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp\" (UID: \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" Oct 31 13:28:58.862764 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.862725 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-istio-token\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp\" (UID: \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" Oct 31 13:28:58.864976 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.864950 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wktnv\" (UniqueName: \"kubernetes.io/projected/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-kube-api-access-wktnv\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp\" (UID: \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" Oct 31 13:28:58.965134 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:58.965099 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" Oct 31 13:28:59.044436 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:59.043573 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t"] Oct 31 13:28:59.060497 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:59.051076 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x898t"] Oct 31 13:28:59.342746 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:59.342719 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp"] Oct 31 13:28:59.345319 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:59.345279 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:28:59.345430 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:59.345361 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:28:59.345430 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:59.345401 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:28:59.413195 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:59.413166 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="595d267e-2643-4570-b24c-8b03e9e34705" path="/var/lib/kubelet/pods/595d267e-2643-4570-b24c-8b03e9e34705/volumes" Oct 31 13:28:59.699122 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:59.699075 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" event={"ID":"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6","Type":"ContainerStarted","Data":"f46697bff63215d4e7cca52a22073a800040001ac71ceb1296105962d6143ff5"} Oct 31 13:28:59.699549 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:59.699149 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" event={"ID":"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6","Type":"ContainerStarted","Data":"ba4d836b54cc4ab7ef1d3e1d34df8ff2e4cf69fe395b8068d646bc76029d5c53"} Oct 31 13:28:59.738811 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:59.738750 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" podStartSLOduration=66.738729725 podStartE2EDuration="1m6.738729725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:27:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:59.730284128 +0000 UTC m=+3390.437870424" watchObservedRunningTime="2026-10-31 13:28:59.738729725 +0000 UTC m=+3390.446316021" Oct 31 13:28:59.965720 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:28:59.965591 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" Oct 31 13:29:00.547490 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:29:00.546944 5693 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda022831f_b584_43fe_91a7_7f541c066ace.slice/crio-conmon-e2d6bcbc7bf35b16702d1f2705a242775af5ab8cd6e0569ad46313a327d6b8b2.scope\": RecentStats: unable to find data in memory cache]" Oct 31 13:29:01.010719 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.004836 5693 generic.go:356] "Generic (PLEG): container finished" podID="a022831f-b584-43fe-91a7-7f541c066ace" containerID="e2d6bcbc7bf35b16702d1f2705a242775af5ab8cd6e0569ad46313a327d6b8b2" exitCode=0 Oct 31 13:29:01.010719 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.008453 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" event={"ID":"a022831f-b584-43fe-91a7-7f541c066ace","Type":"ContainerDied","Data":"e2d6bcbc7bf35b16702d1f2705a242775af5ab8cd6e0569ad46313a327d6b8b2"} Oct 31 13:29:01.010719 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.008522 5693 scope.go:122] "RemoveContainer" containerID="c140569b2bd55c4082acba10ca650cd3758e8d04deb825997848a11005d1a9f8" Oct 31 13:29:01.021937 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.021683 5693 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 31 13:29:01.021937 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.021754 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" podUID="c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 31 13:29:01.109664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.109587 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" Oct 31 13:29:01.156697 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.149981 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp"] Oct 31 13:29:01.156697 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.151554 5693 cpu_manager.go:414] "containerMap: removing container" podUID="a022831f-b584-43fe-91a7-7f541c066ace" containerName="istio-proxy" Oct 31 13:29:01.156697 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.151577 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a022831f-b584-43fe-91a7-7f541c066ace" containerName="istio-proxy" Oct 31 13:29:01.156697 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.151609 5693 cpu_manager.go:414] "containerMap: removing container" podUID="a022831f-b584-43fe-91a7-7f541c066ace" containerName="istio-proxy" Oct 31 13:29:01.156697 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.151616 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a022831f-b584-43fe-91a7-7f541c066ace" containerName="istio-proxy" Oct 31 13:29:01.156697 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.151868 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="a022831f-b584-43fe-91a7-7f541c066ace" containerName="istio-proxy" Oct 31 13:29:01.156697 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.151888 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="a022831f-b584-43fe-91a7-7f541c066ace" containerName="istio-proxy" Oct 31 13:29:01.156697 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.156562 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" Oct 31 13:29:01.164591 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.164569 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-691f1e-openshift-default-dockercfg-bjw8b\"" Oct 31 13:29:01.192675 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.176312 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp"] Oct 31 13:29:01.218585 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.218557 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a022831f-b584-43fe-91a7-7f541c066ace-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a022831f-b584-43fe-91a7-7f541c066ace-credential-socket\") pod \"a022831f-b584-43fe-91a7-7f541c066ace\" (UID: \"a022831f-b584-43fe-91a7-7f541c066ace\") " Oct 31 13:29:01.219263 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.219037 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a022831f-b584-43fe-91a7-7f541c066ace-credential-socket" pod "a022831f-b584-43fe-91a7-7f541c066ace" (UID: "a022831f-b584-43fe-91a7-7f541c066ace"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:29:01.219596 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.219583 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a022831f-b584-43fe-91a7-7f541c066ace-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a022831f-b584-43fe-91a7-7f541c066ace-istio-podinfo\") pod \"a022831f-b584-43fe-91a7-7f541c066ace\" (UID: \"a022831f-b584-43fe-91a7-7f541c066ace\") " Oct 31 13:29:01.220566 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.220211 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a022831f-b584-43fe-91a7-7f541c066ace-istio-token\" (UniqueName: \"kubernetes.io/projected/a022831f-b584-43fe-91a7-7f541c066ace-istio-token\") pod \"a022831f-b584-43fe-91a7-7f541c066ace\" (UID: \"a022831f-b584-43fe-91a7-7f541c066ace\") " Oct 31 13:29:01.222844 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.221201 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a022831f-b584-43fe-91a7-7f541c066ace-kube-api-access-cmt6x\" (UniqueName: \"kubernetes.io/projected/a022831f-b584-43fe-91a7-7f541c066ace-kube-api-access-cmt6x\") pod \"a022831f-b584-43fe-91a7-7f541c066ace\" (UID: \"a022831f-b584-43fe-91a7-7f541c066ace\") " Oct 31 13:29:01.224230 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.222867 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a022831f-b584-43fe-91a7-7f541c066ace-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a022831f-b584-43fe-91a7-7f541c066ace-istio-data\") pod \"a022831f-b584-43fe-91a7-7f541c066ace\" (UID: \"a022831f-b584-43fe-91a7-7f541c066ace\") " Oct 31 13:29:01.224230 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.223102 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a022831f-b584-43fe-91a7-7f541c066ace-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a022831f-b584-43fe-91a7-7f541c066ace-istio-envoy\") pod \"a022831f-b584-43fe-91a7-7f541c066ace\" (UID: \"a022831f-b584-43fe-91a7-7f541c066ace\") " Oct 31 13:29:01.224230 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.223441 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a022831f-b584-43fe-91a7-7f541c066ace-istio-data" pod "a022831f-b584-43fe-91a7-7f541c066ace" (UID: "a022831f-b584-43fe-91a7-7f541c066ace"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:29:01.224230 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.223502 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a022831f-b584-43fe-91a7-7f541c066ace-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a022831f-b584-43fe-91a7-7f541c066ace-workload-socket\") pod \"a022831f-b584-43fe-91a7-7f541c066ace\" (UID: \"a022831f-b584-43fe-91a7-7f541c066ace\") " Oct 31 13:29:01.224230 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.223691 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a022831f-b584-43fe-91a7-7f541c066ace-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a022831f-b584-43fe-91a7-7f541c066ace-workload-certs\") pod \"a022831f-b584-43fe-91a7-7f541c066ace\" (UID: \"a022831f-b584-43fe-91a7-7f541c066ace\") " Oct 31 13:29:01.224230 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.224180 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a022831f-b584-43fe-91a7-7f541c066ace-workload-socket" pod "a022831f-b584-43fe-91a7-7f541c066ace" (UID: "a022831f-b584-43fe-91a7-7f541c066ace"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:29:01.224668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.224466 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a022831f-b584-43fe-91a7-7f541c066ace-workload-certs" pod "a022831f-b584-43fe-91a7-7f541c066ace" (UID: "a022831f-b584-43fe-91a7-7f541c066ace"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:29:01.224874 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.224856 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a022831f-b584-43fe-91a7-7f541c066ace-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a022831f-b584-43fe-91a7-7f541c066ace-istiod-ca-cert\") pod \"a022831f-b584-43fe-91a7-7f541c066ace\" (UID: \"a022831f-b584-43fe-91a7-7f541c066ace\") " Oct 31 13:29:01.225327 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.225295 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a022831f-b584-43fe-91a7-7f541c066ace-istiod-ca-cert" pod "a022831f-b584-43fe-91a7-7f541c066ace" (UID: "a022831f-b584-43fe-91a7-7f541c066ace"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:29:01.225498 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.225481 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a022831f-b584-43fe-91a7-7f541c066ace-istio-token" pod "a022831f-b584-43fe-91a7-7f541c066ace" (UID: "a022831f-b584-43fe-91a7-7f541c066ace"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:29:01.225602 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.225541 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-istio-data\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp\" (UID: \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" Oct 31 13:29:01.225775 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.225759 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-workload-certs\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp\" (UID: \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" Oct 31 13:29:01.226004 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.225988 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-istio-podinfo\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp\" (UID: \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" Oct 31 13:29:01.226120 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.226109 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-istiod-ca-cert\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp\" (UID: \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" Oct 31 13:29:01.226333 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.226320 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pw84\" (UniqueName: \"kubernetes.io/projected/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-kube-api-access-2pw84\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp\" (UID: \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" Oct 31 13:29:01.226428 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.226417 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-istio-token\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp\" (UID: \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" Oct 31 13:29:01.226611 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.226600 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-istio-envoy\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp\" (UID: \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" Oct 31 13:29:01.226914 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.226900 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-workload-socket\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp\" (UID: \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" Oct 31 13:29:01.227010 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.226994 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-credential-socket\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp\" (UID: \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" Oct 31 13:29:01.227197 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.227185 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a022831f-b584-43fe-91a7-7f541c066ace-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:01.227278 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.227270 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a022831f-b584-43fe-91a7-7f541c066ace-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:01.227346 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.227337 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a022831f-b584-43fe-91a7-7f541c066ace-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:01.227412 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.227403 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a022831f-b584-43fe-91a7-7f541c066ace-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:01.227470 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.227462 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a022831f-b584-43fe-91a7-7f541c066ace-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:01.227531 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.227523 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a022831f-b584-43fe-91a7-7f541c066ace-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:01.228342 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.227689 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a022831f-b584-43fe-91a7-7f541c066ace-istio-podinfo" pod "a022831f-b584-43fe-91a7-7f541c066ace" (UID: "a022831f-b584-43fe-91a7-7f541c066ace"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:29:01.235689 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.231886 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a022831f-b584-43fe-91a7-7f541c066ace-kube-api-access-cmt6x" pod "a022831f-b584-43fe-91a7-7f541c066ace" (UID: "a022831f-b584-43fe-91a7-7f541c066ace"). InnerVolumeSpecName "kube-api-access-cmt6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:29:01.235689 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.232506 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a022831f-b584-43fe-91a7-7f541c066ace-istio-envoy" pod "a022831f-b584-43fe-91a7-7f541c066ace" (UID: "a022831f-b584-43fe-91a7-7f541c066ace"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:29:01.329604 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.329564 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-istio-envoy\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp\" (UID: \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" Oct 31 13:29:01.329827 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.329618 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-workload-socket\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp\" (UID: \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" Oct 31 13:29:01.329827 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.329660 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-credential-socket\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp\" (UID: \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" Oct 31 13:29:01.329827 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.329776 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-istio-data\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp\" (UID: \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" Oct 31 13:29:01.329827 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.329800 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-workload-certs\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp\" (UID: \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" Oct 31 13:29:01.330104 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.329862 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-istio-podinfo\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp\" (UID: \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" Oct 31 13:29:01.330104 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.329880 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-istiod-ca-cert\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp\" (UID: \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" Oct 31 13:29:01.330104 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.329957 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pw84\" (UniqueName: \"kubernetes.io/projected/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-kube-api-access-2pw84\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp\" (UID: \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" Oct 31 13:29:01.330104 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.329980 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-istio-token\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp\" (UID: \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" Oct 31 13:29:01.330104 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.330084 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a022831f-b584-43fe-91a7-7f541c066ace-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:01.330104 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.330097 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmt6x\" (UniqueName: \"kubernetes.io/projected/a022831f-b584-43fe-91a7-7f541c066ace-kube-api-access-cmt6x\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:01.330104 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.330107 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a022831f-b584-43fe-91a7-7f541c066ace-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:01.330488 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.330111 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-workload-socket\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp\" (UID: \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" Oct 31 13:29:01.330488 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.330397 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-credential-socket\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp\" (UID: \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" Oct 31 13:29:01.330627 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.330603 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-istio-data\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp\" (UID: \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" Oct 31 13:29:01.330872 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.330833 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-istiod-ca-cert\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp\" (UID: \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" Oct 31 13:29:01.331690 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.331541 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-workload-certs\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp\" (UID: \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" Oct 31 13:29:01.332688 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.332614 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-istio-envoy\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp\" (UID: \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" Oct 31 13:29:01.334208 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.334187 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-istio-podinfo\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp\" (UID: \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" Oct 31 13:29:01.343683 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.343559 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-istio-token\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp\" (UID: \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" Oct 31 13:29:01.343803 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.343699 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pw84\" (UniqueName: \"kubernetes.io/projected/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-kube-api-access-2pw84\") pod \"e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp\" (UID: \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\") " pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" Oct 31 13:29:01.518948 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.518913 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" Oct 31 13:29:01.922792 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.922756 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp"] Oct 31 13:29:01.922920 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:29:01.922760 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefccc572_e9f9_4f4a_ab45_c7458b0e2e58.slice/crio-94228db49135d44ced3fb1feb111429775198883770eae5a13f12083f71f79d1 WatchSource:0}: Error finding container 94228db49135d44ced3fb1feb111429775198883770eae5a13f12083f71f79d1: Status 404 returned error can't find the container with id 94228db49135d44ced3fb1feb111429775198883770eae5a13f12083f71f79d1 Oct 31 13:29:01.926874 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.926831 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:29:01.926996 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.926924 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:29:01.926996 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.926971 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:29:01.965683 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.965650 5693 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 31 13:29:01.965836 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:01.965701 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" podUID="c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 31 13:29:02.018150 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:02.018119 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" Oct 31 13:29:02.018565 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:02.018191 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4" event={"ID":"a022831f-b584-43fe-91a7-7f541c066ace","Type":"ContainerDied","Data":"d6626697337212525e6ef7dff7209bff00c943917572cc1641ef215b2d000e6f"} Oct 31 13:29:02.018565 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:02.018250 5693 scope.go:122] "RemoveContainer" containerID="e2d6bcbc7bf35b16702d1f2705a242775af5ab8cd6e0569ad46313a327d6b8b2" Oct 31 13:29:02.020462 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:02.020434 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" event={"ID":"efccc572-e9f9-4f4a-ab45-c7458b0e2e58","Type":"ContainerStarted","Data":"94228db49135d44ced3fb1feb111429775198883770eae5a13f12083f71f79d1"} Oct 31 13:29:02.050128 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:02.050100 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4"] Oct 31 13:29:02.058195 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:02.058167 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-jp9l4"] Oct 31 13:29:02.707517 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:02.707483 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs"] Oct 31 13:29:02.707793 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:02.707720 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" podUID="6d9b984c-d94b-4d88-8b4b-48375154dfaa" containerName="istio-proxy" containerID="cri-o://8c6bcb7b3aa1e39e060d1e0dda5998d1b8b9b4efe60ee32bbfd8eebb76e4e099" gracePeriod=30 Oct 31 13:29:02.966420 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:02.966335 5693 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 31 13:29:02.966420 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:02.966396 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" podUID="c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 31 13:29:03.030597 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:03.030557 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" event={"ID":"efccc572-e9f9-4f4a-ab45-c7458b0e2e58","Type":"ContainerStarted","Data":"15a949ec75dd1373b9bbb8b3ec5121bcd53fe94ec3e99f9369b78d8e66383412"} Oct 31 13:29:03.058158 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:03.058107 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" podStartSLOduration=15.058093395 podStartE2EDuration="15.058093395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:28:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:03.056726231 +0000 UTC m=+3393.764312526" watchObservedRunningTime="2026-10-31 13:29:03.058093395 +0000 UTC m=+3393.765679689" Oct 31 13:29:03.413089 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:03.413052 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a022831f-b584-43fe-91a7-7f541c066ace" path="/var/lib/kubelet/pods/a022831f-b584-43fe-91a7-7f541c066ace/volumes" Oct 31 13:29:03.519527 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:03.519486 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" Oct 31 13:29:03.525806 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:03.525776 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" Oct 31 13:29:03.966211 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:03.966175 5693 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 31 13:29:03.966400 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:03.966241 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" podUID="c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 31 13:29:04.040973 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:04.040942 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" Oct 31 13:29:04.042062 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:04.042037 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" Oct 31 13:29:04.966248 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:04.966210 5693 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 31 13:29:04.966469 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:04.966277 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" podUID="c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 31 13:29:06.162758 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:06.162720 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" Oct 31 13:29:06.163542 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:06.163253 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" Oct 31 13:29:06.164352 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:06.164331 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" Oct 31 13:29:06.679373 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:06.679331 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-691f1e-68bd657c5-whf6d"] Oct 31 13:29:06.685879 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:06.685853 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-691f1e-68bd657c5-whf6d" Oct 31 13:29:06.695921 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:06.695891 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-691f1e-68bd657c5-whf6d"] Oct 31 13:29:06.696679 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:06.696634 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-691f1e\"" Oct 31 13:29:06.696679 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:06.696632 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-691f1e-dockercfg-2ndxx\"" Oct 31 13:29:06.697839 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:06.697582 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/497f3ecd-3d10-4ffd-ae25-490c570c4b1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-691f1e-68bd657c5-whf6d\" (UID: \"497f3ecd-3d10-4ffd-ae25-490c570c4b1f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-691f1e-68bd657c5-whf6d" Oct 31 13:29:06.697945 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:06.697877 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm5tk\" (UniqueName: \"kubernetes.io/projected/497f3ecd-3d10-4ffd-ae25-490c570c4b1f-kube-api-access-nm5tk\") pod \"payload-pre-processing-e2e-inf-a-691f1e-68bd657c5-whf6d\" (UID: \"497f3ecd-3d10-4ffd-ae25-490c570c4b1f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-691f1e-68bd657c5-whf6d" Oct 31 13:29:06.831666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:06.813029 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm5tk\" (UniqueName: \"kubernetes.io/projected/497f3ecd-3d10-4ffd-ae25-490c570c4b1f-kube-api-access-nm5tk\") pod \"payload-pre-processing-e2e-inf-a-691f1e-68bd657c5-whf6d\" (UID: \"497f3ecd-3d10-4ffd-ae25-490c570c4b1f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-691f1e-68bd657c5-whf6d" Oct 31 13:29:06.841668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:06.833333 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/497f3ecd-3d10-4ffd-ae25-490c570c4b1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-691f1e-68bd657c5-whf6d\" (UID: \"497f3ecd-3d10-4ffd-ae25-490c570c4b1f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-691f1e-68bd657c5-whf6d" Oct 31 13:29:06.841917 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:06.841885 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/497f3ecd-3d10-4ffd-ae25-490c570c4b1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-691f1e-68bd657c5-whf6d\" (UID: \"497f3ecd-3d10-4ffd-ae25-490c570c4b1f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-691f1e-68bd657c5-whf6d" Oct 31 13:29:06.860351 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:06.855761 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm5tk\" (UniqueName: \"kubernetes.io/projected/497f3ecd-3d10-4ffd-ae25-490c570c4b1f-kube-api-access-nm5tk\") pod \"payload-pre-processing-e2e-inf-a-691f1e-68bd657c5-whf6d\" (UID: \"497f3ecd-3d10-4ffd-ae25-490c570c4b1f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-691f1e-68bd657c5-whf6d" Oct 31 13:29:07.014778 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:07.007015 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-691f1e-68bd657c5-whf6d" Oct 31 13:29:07.400101 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:07.400068 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-691f1e-68bd657c5-whf6d"] Oct 31 13:29:07.400548 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:29:07.400309 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod497f3ecd_3d10_4ffd_ae25_490c570c4b1f.slice/crio-140a321a52480f28ede4750f2bddb854e027015c63cc4a370c1d361da3fd6a5e WatchSource:0}: Error finding container 140a321a52480f28ede4750f2bddb854e027015c63cc4a370c1d361da3fd6a5e: Status 404 returned error can't find the container with id 140a321a52480f28ede4750f2bddb854e027015c63cc4a370c1d361da3fd6a5e Oct 31 13:29:07.936349 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:07.935846 5693 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 13:29:07.936349 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:07.935915 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" podUID="6d9b984c-d94b-4d88-8b4b-48375154dfaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 13:29:08.078764 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.078726 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" Oct 31 13:29:08.101829 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.101795 5693 generic.go:356] "Generic (PLEG): container finished" podID="6d9b984c-d94b-4d88-8b4b-48375154dfaa" containerID="8c6bcb7b3aa1e39e060d1e0dda5998d1b8b9b4efe60ee32bbfd8eebb76e4e099" exitCode=0 Oct 31 13:29:08.102018 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.101904 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" event={"ID":"6d9b984c-d94b-4d88-8b4b-48375154dfaa","Type":"ContainerDied","Data":"8c6bcb7b3aa1e39e060d1e0dda5998d1b8b9b4efe60ee32bbfd8eebb76e4e099"} Oct 31 13:29:08.102018 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.101905 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" Oct 31 13:29:08.102018 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.101956 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs" event={"ID":"6d9b984c-d94b-4d88-8b4b-48375154dfaa","Type":"ContainerDied","Data":"3965df2c000f19af5a63179e5eac3c501927a2691b89e18e09adb8695d0195e9"} Oct 31 13:29:08.102018 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.101989 5693 scope.go:122] "RemoveContainer" containerID="8c6bcb7b3aa1e39e060d1e0dda5998d1b8b9b4efe60ee32bbfd8eebb76e4e099" Oct 31 13:29:08.106511 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.106478 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-691f1e-68bd657c5-whf6d" event={"ID":"497f3ecd-3d10-4ffd-ae25-490c570c4b1f","Type":"ContainerStarted","Data":"bdb16dab5bb358e91cad68c315c38f01eae7f3660d8c5c93a787facbe35994af"} Oct 31 13:29:08.107115 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.106564 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-691f1e-68bd657c5-whf6d" event={"ID":"497f3ecd-3d10-4ffd-ae25-490c570c4b1f","Type":"ContainerStarted","Data":"140a321a52480f28ede4750f2bddb854e027015c63cc4a370c1d361da3fd6a5e"} Oct 31 13:29:08.107115 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.106825 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-691f1e-68bd657c5-whf6d" Oct 31 13:29:08.117674 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.117626 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-691f1e-569d8fc9cc-k6fn8"] Oct 31 13:29:08.118540 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.118517 5693 scope.go:122] "RemoveContainer" containerID="8c6bcb7b3aa1e39e060d1e0dda5998d1b8b9b4efe60ee32bbfd8eebb76e4e099" Oct 31 13:29:08.118829 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.118812 5693 cpu_manager.go:414] "containerMap: removing container" podUID="6d9b984c-d94b-4d88-8b4b-48375154dfaa" containerName="istio-proxy" Oct 31 13:29:08.118920 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.118832 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d9b984c-d94b-4d88-8b4b-48375154dfaa" containerName="istio-proxy" Oct 31 13:29:08.118973 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:29:08.118952 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c6bcb7b3aa1e39e060d1e0dda5998d1b8b9b4efe60ee32bbfd8eebb76e4e099\": container with ID starting with 8c6bcb7b3aa1e39e060d1e0dda5998d1b8b9b4efe60ee32bbfd8eebb76e4e099 not found: ID does not exist" containerID="8c6bcb7b3aa1e39e060d1e0dda5998d1b8b9b4efe60ee32bbfd8eebb76e4e099" Oct 31 13:29:08.119013 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.118978 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c6bcb7b3aa1e39e060d1e0dda5998d1b8b9b4efe60ee32bbfd8eebb76e4e099"} err="failed to get container status \"8c6bcb7b3aa1e39e060d1e0dda5998d1b8b9b4efe60ee32bbfd8eebb76e4e099\": rpc error: code = NotFound desc = could not find container \"8c6bcb7b3aa1e39e060d1e0dda5998d1b8b9b4efe60ee32bbfd8eebb76e4e099\": container with ID starting with 8c6bcb7b3aa1e39e060d1e0dda5998d1b8b9b4efe60ee32bbfd8eebb76e4e099 not found: ID does not exist" Oct 31 13:29:08.119052 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.119017 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d9b984c-d94b-4d88-8b4b-48375154dfaa" containerName="istio-proxy" Oct 31 13:29:08.124445 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.124418 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-691f1e-569d8fc9cc-k6fn8" Oct 31 13:29:08.149872 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.149836 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-691f1e-569d8fc9cc-k6fn8"] Oct 31 13:29:08.165683 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.165596 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-691f1e-68bd657c5-whf6d" podStartSLOduration=2.165576039 podStartE2EDuration="2.165576039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:08.13734819 +0000 UTC m=+3398.844934485" watchObservedRunningTime="2026-10-31 13:29:08.165576039 +0000 UTC m=+3398.873162335" Oct 31 13:29:08.198739 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.198700 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d9b984c-d94b-4d88-8b4b-48375154dfaa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d9b984c-d94b-4d88-8b4b-48375154dfaa-workload-socket\") pod \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\" (UID: \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\") " Oct 31 13:29:08.198955 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.198853 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d9b984c-d94b-4d88-8b4b-48375154dfaa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d9b984c-d94b-4d88-8b4b-48375154dfaa-credential-socket\") pod \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\" (UID: \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\") " Oct 31 13:29:08.198955 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.198918 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d9b984c-d94b-4d88-8b4b-48375154dfaa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d9b984c-d94b-4d88-8b4b-48375154dfaa-istio-data\") pod \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\" (UID: \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\") " Oct 31 13:29:08.199099 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.199024 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d9b984c-d94b-4d88-8b4b-48375154dfaa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d9b984c-d94b-4d88-8b4b-48375154dfaa-istiod-ca-cert\") pod \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\" (UID: \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\") " Oct 31 13:29:08.199099 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.199031 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d9b984c-d94b-4d88-8b4b-48375154dfaa-workload-socket" pod "6d9b984c-d94b-4d88-8b4b-48375154dfaa" (UID: "6d9b984c-d94b-4d88-8b4b-48375154dfaa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:29:08.199218 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.199109 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d9b984c-d94b-4d88-8b4b-48375154dfaa-istio-token\" (UniqueName: \"kubernetes.io/projected/6d9b984c-d94b-4d88-8b4b-48375154dfaa-istio-token\") pod \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\" (UID: \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\") " Oct 31 13:29:08.199278 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.199214 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d9b984c-d94b-4d88-8b4b-48375154dfaa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d9b984c-d94b-4d88-8b4b-48375154dfaa-istio-envoy\") pod \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\" (UID: \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\") " Oct 31 13:29:08.199278 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.199220 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d9b984c-d94b-4d88-8b4b-48375154dfaa-credential-socket" pod "6d9b984c-d94b-4d88-8b4b-48375154dfaa" (UID: "6d9b984c-d94b-4d88-8b4b-48375154dfaa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:29:08.199278 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.199241 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d9b984c-d94b-4d88-8b4b-48375154dfaa-istio-data" pod "6d9b984c-d94b-4d88-8b4b-48375154dfaa" (UID: "6d9b984c-d94b-4d88-8b4b-48375154dfaa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:29:08.199440 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.199329 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d9b984c-d94b-4d88-8b4b-48375154dfaa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d9b984c-d94b-4d88-8b4b-48375154dfaa-workload-certs\") pod \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\" (UID: \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\") " Oct 31 13:29:08.199440 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.199387 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d9b984c-d94b-4d88-8b4b-48375154dfaa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d9b984c-d94b-4d88-8b4b-48375154dfaa-istio-podinfo\") pod \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\" (UID: \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\") " Oct 31 13:29:08.199440 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.199414 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d9b984c-d94b-4d88-8b4b-48375154dfaa-istiod-ca-cert" pod "6d9b984c-d94b-4d88-8b4b-48375154dfaa" (UID: "6d9b984c-d94b-4d88-8b4b-48375154dfaa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:29:08.199440 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.199424 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d9b984c-d94b-4d88-8b4b-48375154dfaa-kube-api-access-h9rwt\" (UniqueName: \"kubernetes.io/projected/6d9b984c-d94b-4d88-8b4b-48375154dfaa-kube-api-access-h9rwt\") pod \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\" (UID: \"6d9b984c-d94b-4d88-8b4b-48375154dfaa\") " Oct 31 13:29:08.199733 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.199711 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d9b984c-d94b-4d88-8b4b-48375154dfaa-workload-certs" pod "6d9b984c-d94b-4d88-8b4b-48375154dfaa" (UID: "6d9b984c-d94b-4d88-8b4b-48375154dfaa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:29:08.200412 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.199944 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4rfm\" (UniqueName: \"kubernetes.io/projected/30cc0f1f-cfc3-475d-8d48-a98f210b91f6-kube-api-access-b4rfm\") pod \"payload-processing-e2e-inf-a-691f1e-569d8fc9cc-k6fn8\" (UID: \"30cc0f1f-cfc3-475d-8d48-a98f210b91f6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-691f1e-569d8fc9cc-k6fn8" Oct 31 13:29:08.201633 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.200568 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30cc0f1f-cfc3-475d-8d48-a98f210b91f6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-691f1e-569d8fc9cc-k6fn8\" (UID: \"30cc0f1f-cfc3-475d-8d48-a98f210b91f6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-691f1e-569d8fc9cc-k6fn8" Oct 31 13:29:08.201633 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.201370 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d9b984c-d94b-4d88-8b4b-48375154dfaa-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:08.201633 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.201396 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d9b984c-d94b-4d88-8b4b-48375154dfaa-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:08.201633 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.201412 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d9b984c-d94b-4d88-8b4b-48375154dfaa-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:08.201633 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.201426 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d9b984c-d94b-4d88-8b4b-48375154dfaa-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:08.201633 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.201441 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d9b984c-d94b-4d88-8b4b-48375154dfaa-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:08.202370 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.202332 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d9b984c-d94b-4d88-8b4b-48375154dfaa-istio-token" pod "6d9b984c-d94b-4d88-8b4b-48375154dfaa" (UID: "6d9b984c-d94b-4d88-8b4b-48375154dfaa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:29:08.203034 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.203011 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d9b984c-d94b-4d88-8b4b-48375154dfaa-istio-podinfo" pod "6d9b984c-d94b-4d88-8b4b-48375154dfaa" (UID: "6d9b984c-d94b-4d88-8b4b-48375154dfaa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:29:08.203140 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.203036 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d9b984c-d94b-4d88-8b4b-48375154dfaa-kube-api-access-h9rwt" pod "6d9b984c-d94b-4d88-8b4b-48375154dfaa" (UID: "6d9b984c-d94b-4d88-8b4b-48375154dfaa"). InnerVolumeSpecName "kube-api-access-h9rwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:29:08.203533 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.203518 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d9b984c-d94b-4d88-8b4b-48375154dfaa-istio-envoy" pod "6d9b984c-d94b-4d88-8b4b-48375154dfaa" (UID: "6d9b984c-d94b-4d88-8b4b-48375154dfaa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:29:08.303497 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.303400 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30cc0f1f-cfc3-475d-8d48-a98f210b91f6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-691f1e-569d8fc9cc-k6fn8\" (UID: \"30cc0f1f-cfc3-475d-8d48-a98f210b91f6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-691f1e-569d8fc9cc-k6fn8" Oct 31 13:29:08.303666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.303622 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4rfm\" (UniqueName: \"kubernetes.io/projected/30cc0f1f-cfc3-475d-8d48-a98f210b91f6-kube-api-access-b4rfm\") pod \"payload-processing-e2e-inf-a-691f1e-569d8fc9cc-k6fn8\" (UID: \"30cc0f1f-cfc3-475d-8d48-a98f210b91f6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-691f1e-569d8fc9cc-k6fn8" Oct 31 13:29:08.303881 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.303863 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d9b984c-d94b-4d88-8b4b-48375154dfaa-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:08.303934 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.303887 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9rwt\" (UniqueName: \"kubernetes.io/projected/6d9b984c-d94b-4d88-8b4b-48375154dfaa-kube-api-access-h9rwt\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:08.303934 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.303904 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d9b984c-d94b-4d88-8b4b-48375154dfaa-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:08.303934 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.303916 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d9b984c-d94b-4d88-8b4b-48375154dfaa-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:08.304230 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.304208 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30cc0f1f-cfc3-475d-8d48-a98f210b91f6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-691f1e-569d8fc9cc-k6fn8\" (UID: \"30cc0f1f-cfc3-475d-8d48-a98f210b91f6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-691f1e-569d8fc9cc-k6fn8" Oct 31 13:29:08.318390 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.318039 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4rfm\" (UniqueName: \"kubernetes.io/projected/30cc0f1f-cfc3-475d-8d48-a98f210b91f6-kube-api-access-b4rfm\") pod \"payload-processing-e2e-inf-a-691f1e-569d8fc9cc-k6fn8\" (UID: \"30cc0f1f-cfc3-475d-8d48-a98f210b91f6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-691f1e-569d8fc9cc-k6fn8" Oct 31 13:29:08.431435 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.431403 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs"] Oct 31 13:29:08.436510 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.436482 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-twcvs"] Oct 31 13:29:08.463786 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.463730 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-691f1e-569d8fc9cc-k6fn8" Oct 31 13:29:08.831614 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:08.831584 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-691f1e-569d8fc9cc-k6fn8"] Oct 31 13:29:09.117553 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:09.117465 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-691f1e-569d8fc9cc-k6fn8" event={"ID":"30cc0f1f-cfc3-475d-8d48-a98f210b91f6","Type":"ContainerStarted","Data":"65209c72d8b5a3378ac7d6e207966f4f8df1e679bdbe7a08e5420517ecfe0e00"} Oct 31 13:29:09.117553 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:09.117528 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-691f1e-569d8fc9cc-k6fn8" event={"ID":"30cc0f1f-cfc3-475d-8d48-a98f210b91f6","Type":"ContainerStarted","Data":"60256b760ef1fecfd864811de61f9bebad84596945f75f989827cd155b48b371"} Oct 31 13:29:09.117809 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:09.117789 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-691f1e-569d8fc9cc-k6fn8" Oct 31 13:29:09.140458 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:09.140407 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-691f1e-569d8fc9cc-k6fn8" podStartSLOduration=3.1403939 podStartE2EDuration="3.1403939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:09.138022049 +0000 UTC m=+3399.845608344" watchObservedRunningTime="2026-10-31 13:29:09.1403939 +0000 UTC m=+3399.847980195" Oct 31 13:29:09.411881 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:09.411848 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d9b984c-d94b-4d88-8b4b-48375154dfaa" path="/var/lib/kubelet/pods/6d9b984c-d94b-4d88-8b4b-48375154dfaa/volumes" Oct 31 13:29:18.693881 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:29:18.693843 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:29:20.128483 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:20.128453 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-691f1e-569d8fc9cc-k6fn8" Oct 31 13:29:20.129115 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:20.128907 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-691f1e-68bd657c5-whf6d" Oct 31 13:29:47.025966 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:47.025887 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ca4b64b-7kdk6"] Oct 31 13:29:47.034919 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:47.034893 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ca4b64b-7kdk6" Oct 31 13:29:47.043451 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:47.043418 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ca4b64b-7kdk6"] Oct 31 13:29:47.045242 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:47.045211 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:29:47.045525 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:47.045507 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:29:47.160727 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:47.160632 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df80a09d-3179-4e70-ae9a-748d67f5c460-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ca4b64b-7kdk6\" (UID: \"df80a09d-3179-4e70-ae9a-748d67f5c460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ca4b64b-7kdk6" Oct 31 13:29:47.160967 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:47.160887 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcbkd\" (UniqueName: \"kubernetes.io/projected/df80a09d-3179-4e70-ae9a-748d67f5c460-kube-api-access-bcbkd\") pod \"maas-api-revoke-sub-2ca4b64b-7kdk6\" (UID: \"df80a09d-3179-4e70-ae9a-748d67f5c460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ca4b64b-7kdk6" Oct 31 13:29:47.161069 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:47.161020 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df80a09d-3179-4e70-ae9a-748d67f5c460-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ca4b64b-7kdk6\" (UID: \"df80a09d-3179-4e70-ae9a-748d67f5c460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ca4b64b-7kdk6" Oct 31 13:29:47.264600 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:47.264559 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcbkd\" (UniqueName: \"kubernetes.io/projected/df80a09d-3179-4e70-ae9a-748d67f5c460-kube-api-access-bcbkd\") pod \"maas-api-revoke-sub-2ca4b64b-7kdk6\" (UID: \"df80a09d-3179-4e70-ae9a-748d67f5c460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ca4b64b-7kdk6" Oct 31 13:29:47.264859 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:47.264728 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df80a09d-3179-4e70-ae9a-748d67f5c460-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ca4b64b-7kdk6\" (UID: \"df80a09d-3179-4e70-ae9a-748d67f5c460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ca4b64b-7kdk6" Oct 31 13:29:47.264989 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:47.264968 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df80a09d-3179-4e70-ae9a-748d67f5c460-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ca4b64b-7kdk6\" (UID: \"df80a09d-3179-4e70-ae9a-748d67f5c460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ca4b64b-7kdk6" Oct 31 13:29:47.265971 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:47.265943 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df80a09d-3179-4e70-ae9a-748d67f5c460-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ca4b64b-7kdk6\" (UID: \"df80a09d-3179-4e70-ae9a-748d67f5c460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ca4b64b-7kdk6" Oct 31 13:29:47.267623 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:47.267600 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df80a09d-3179-4e70-ae9a-748d67f5c460-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ca4b64b-7kdk6\" (UID: \"df80a09d-3179-4e70-ae9a-748d67f5c460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ca4b64b-7kdk6" Oct 31 13:29:47.278202 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:47.278128 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcbkd\" (UniqueName: \"kubernetes.io/projected/df80a09d-3179-4e70-ae9a-748d67f5c460-kube-api-access-bcbkd\") pod \"maas-api-revoke-sub-2ca4b64b-7kdk6\" (UID: \"df80a09d-3179-4e70-ae9a-748d67f5c460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ca4b64b-7kdk6" Oct 31 13:29:47.364377 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:47.364008 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ca4b64b-7kdk6" Oct 31 13:29:47.704702 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:47.704671 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ca4b64b-7kdk6"] Oct 31 13:29:48.626405 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:48.626360 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ca4b64b-7kdk6" event={"ID":"df80a09d-3179-4e70-ae9a-748d67f5c460","Type":"ContainerStarted","Data":"c590d9c3a9e8cc8d801040a1c6323d96b2842c4a69627895099adfaa6ea168e0"} Oct 31 13:29:48.626863 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:48.626433 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ca4b64b-7kdk6" event={"ID":"df80a09d-3179-4e70-ae9a-748d67f5c460","Type":"ContainerStarted","Data":"4d532c4bebe5d24695aacdfc3d0f028d9a96d0c9212f9c1b05e106950fc5e7c1"} Oct 31 13:29:48.649095 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:48.649037 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ca4b64b-7kdk6" podStartSLOduration=2.649023416 podStartE2EDuration="2.649023416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:48.644620556 +0000 UTC m=+3439.352206852" watchObservedRunningTime="2026-10-31 13:29:48.649023416 +0000 UTC m=+3439.356609710" Oct 31 13:29:48.691184 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:29:48.691141 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:29:49.636935 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:49.636900 5693 generic.go:356] "Generic (PLEG): container finished" podID="df80a09d-3179-4e70-ae9a-748d67f5c460" containerID="c590d9c3a9e8cc8d801040a1c6323d96b2842c4a69627895099adfaa6ea168e0" exitCode=0 Oct 31 13:29:49.637478 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:49.636949 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ca4b64b-7kdk6" event={"ID":"df80a09d-3179-4e70-ae9a-748d67f5c460","Type":"ContainerDied","Data":"c590d9c3a9e8cc8d801040a1c6323d96b2842c4a69627895099adfaa6ea168e0"} Oct 31 13:29:50.899184 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:50.899117 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ca4b64b-7kdk6" Oct 31 13:29:50.970802 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:50.970771 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df80a09d-3179-4e70-ae9a-748d67f5c460-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df80a09d-3179-4e70-ae9a-748d67f5c460-maas-api-xks-ca\") pod \"df80a09d-3179-4e70-ae9a-748d67f5c460\" (UID: \"df80a09d-3179-4e70-ae9a-748d67f5c460\") " Oct 31 13:29:50.970991 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:50.970914 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df80a09d-3179-4e70-ae9a-748d67f5c460-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df80a09d-3179-4e70-ae9a-748d67f5c460-maas-api-openshift-ca\") pod \"df80a09d-3179-4e70-ae9a-748d67f5c460\" (UID: \"df80a09d-3179-4e70-ae9a-748d67f5c460\") " Oct 31 13:29:50.971069 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:50.971052 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df80a09d-3179-4e70-ae9a-748d67f5c460-kube-api-access-bcbkd\" (UniqueName: \"kubernetes.io/projected/df80a09d-3179-4e70-ae9a-748d67f5c460-kube-api-access-bcbkd\") pod \"df80a09d-3179-4e70-ae9a-748d67f5c460\" (UID: \"df80a09d-3179-4e70-ae9a-748d67f5c460\") " Oct 31 13:29:50.971292 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:50.971254 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df80a09d-3179-4e70-ae9a-748d67f5c460-maas-api-openshift-ca" pod "df80a09d-3179-4e70-ae9a-748d67f5c460" (UID: "df80a09d-3179-4e70-ae9a-748d67f5c460"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:29:50.971918 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:50.971898 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df80a09d-3179-4e70-ae9a-748d67f5c460-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:50.973264 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:50.973232 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df80a09d-3179-4e70-ae9a-748d67f5c460-maas-api-xks-ca" pod "df80a09d-3179-4e70-ae9a-748d67f5c460" (UID: "df80a09d-3179-4e70-ae9a-748d67f5c460"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:29:50.973361 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:50.973316 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df80a09d-3179-4e70-ae9a-748d67f5c460-kube-api-access-bcbkd" pod "df80a09d-3179-4e70-ae9a-748d67f5c460" (UID: "df80a09d-3179-4e70-ae9a-748d67f5c460"). InnerVolumeSpecName "kube-api-access-bcbkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:29:51.073305 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:51.073274 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df80a09d-3179-4e70-ae9a-748d67f5c460-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:51.073305 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:51.073304 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcbkd\" (UniqueName: \"kubernetes.io/projected/df80a09d-3179-4e70-ae9a-748d67f5c460-kube-api-access-bcbkd\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:51.664325 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:51.664297 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ca4b64b-7kdk6" Oct 31 13:29:51.664529 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:51.664323 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ca4b64b-7kdk6" event={"ID":"df80a09d-3179-4e70-ae9a-748d67f5c460","Type":"ContainerDied","Data":"4d532c4bebe5d24695aacdfc3d0f028d9a96d0c9212f9c1b05e106950fc5e7c1"} Oct 31 13:29:51.664529 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:29:51.664374 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d532c4bebe5d24695aacdfc3d0f028d9a96d0c9212f9c1b05e106950fc5e7c1" Oct 31 13:30:00.404492 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.404453 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890890-5bssh"] Oct 31 13:30:00.406207 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.406164 5693 cpu_manager.go:414] "containerMap: removing container" podUID="df80a09d-3179-4e70-ae9a-748d67f5c460" containerName="revoke-keys" Oct 31 13:30:00.406207 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.406196 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df80a09d-3179-4e70-ae9a-748d67f5c460" containerName="revoke-keys" Oct 31 13:30:00.406384 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.406370 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="df80a09d-3179-4e70-ae9a-748d67f5c460" containerName="revoke-keys" Oct 31 13:30:00.410912 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.410889 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890890-5bssh" Oct 31 13:30:00.413930 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.413902 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890890-2k4sn"] Oct 31 13:30:00.420214 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.420195 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890890-2k4sn" Oct 31 13:30:00.443888 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.443862 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:30:00.447906 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.447880 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890890-rlpk4"] Oct 31 13:30:00.453036 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.453020 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890890-rlpk4" Oct 31 13:30:00.466210 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.466146 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890890-c6w7z"] Oct 31 13:30:00.488342 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.479698 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890890-66f9d"] Oct 31 13:30:00.488342 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.481779 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890890-c6w7z" Oct 31 13:30:00.512155 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.509046 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/875da43d-e0a5-4fba-bd88-7758e8581e04-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890890-rlpk4\" (UID: \"875da43d-e0a5-4fba-bd88-7758e8581e04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890890-rlpk4" Oct 31 13:30:00.512155 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.509149 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phnq6\" (UniqueName: \"kubernetes.io/projected/22242dc7-c483-4f14-8e16-5f5e5f1a7ad4-kube-api-access-phnq6\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890890-2k4sn\" (UID: \"22242dc7-c483-4f14-8e16-5f5e5f1a7ad4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890890-2k4sn" Oct 31 13:30:00.512155 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.509242 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tddl\" (UniqueName: \"kubernetes.io/projected/875da43d-e0a5-4fba-bd88-7758e8581e04-kube-api-access-9tddl\") pod \"maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890890-rlpk4\" (UID: \"875da43d-e0a5-4fba-bd88-7758e8581e04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890890-rlpk4" Oct 31 13:30:00.512155 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.509280 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7hxg\" (UniqueName: \"kubernetes.io/projected/e486c623-3983-406f-9f04-21a0d994c828-kube-api-access-z7hxg\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890890-c6w7z\" (UID: \"e486c623-3983-406f-9f04-21a0d994c828\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890890-c6w7z" Oct 31 13:30:00.512155 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.509405 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22242dc7-c483-4f14-8e16-5f5e5f1a7ad4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890890-2k4sn\" (UID: \"22242dc7-c483-4f14-8e16-5f5e5f1a7ad4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890890-2k4sn" Oct 31 13:30:00.512155 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.509517 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92712c0d-d49c-46fe-8bad-a023a997d5f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890890-5bssh\" (UID: \"92712c0d-d49c-46fe-8bad-a023a997d5f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890890-5bssh" Oct 31 13:30:00.512155 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.509586 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e486c623-3983-406f-9f04-21a0d994c828-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890890-c6w7z\" (UID: \"e486c623-3983-406f-9f04-21a0d994c828\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890890-c6w7z" Oct 31 13:30:00.512155 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.509705 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98ldg\" (UniqueName: \"kubernetes.io/projected/92712c0d-d49c-46fe-8bad-a023a997d5f2-kube-api-access-98ldg\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890890-5bssh\" (UID: \"92712c0d-d49c-46fe-8bad-a023a997d5f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890890-5bssh" Oct 31 13:30:00.528956 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.528919 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890890-cnvdm"] Oct 31 13:30:00.531188 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.531165 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890890-66f9d" Oct 31 13:30:00.545505 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.545471 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890890-5bssh"] Oct 31 13:30:00.545756 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.545589 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890890-cnvdm" Oct 31 13:30:00.547592 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.546564 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890890-2k4sn"] Oct 31 13:30:00.552844 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.552819 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890890-rlpk4"] Oct 31 13:30:00.574730 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.574707 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890890-66f9d"] Oct 31 13:30:00.580010 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.579977 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890890-c6w7z"] Oct 31 13:30:00.589995 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.589964 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890890-cnvdm"] Oct 31 13:30:00.610773 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.610741 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98ldg\" (UniqueName: \"kubernetes.io/projected/92712c0d-d49c-46fe-8bad-a023a997d5f2-kube-api-access-98ldg\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890890-5bssh\" (UID: \"92712c0d-d49c-46fe-8bad-a023a997d5f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890890-5bssh" Oct 31 13:30:00.611004 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.610785 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0fb78e4-7046-48f0-a737-4ca457074027-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890890-cnvdm\" (UID: \"c0fb78e4-7046-48f0-a737-4ca457074027\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890890-cnvdm" Oct 31 13:30:00.611004 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.610850 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24816799-6034-48e6-9711-d9a342c0ef28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-107955-29890890-66f9d\" (UID: \"24816799-6034-48e6-9711-d9a342c0ef28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890890-66f9d" Oct 31 13:30:00.611004 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.610873 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/875da43d-e0a5-4fba-bd88-7758e8581e04-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890890-rlpk4\" (UID: \"875da43d-e0a5-4fba-bd88-7758e8581e04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890890-rlpk4" Oct 31 13:30:00.611187 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.611041 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phnq6\" (UniqueName: \"kubernetes.io/projected/22242dc7-c483-4f14-8e16-5f5e5f1a7ad4-kube-api-access-phnq6\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890890-2k4sn\" (UID: \"22242dc7-c483-4f14-8e16-5f5e5f1a7ad4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890890-2k4sn" Oct 31 13:30:00.611187 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.611126 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tddl\" (UniqueName: \"kubernetes.io/projected/875da43d-e0a5-4fba-bd88-7758e8581e04-kube-api-access-9tddl\") pod \"maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890890-rlpk4\" (UID: \"875da43d-e0a5-4fba-bd88-7758e8581e04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890890-rlpk4" Oct 31 13:30:00.611187 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.611152 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7hxg\" (UniqueName: \"kubernetes.io/projected/e486c623-3983-406f-9f04-21a0d994c828-kube-api-access-z7hxg\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890890-c6w7z\" (UID: \"e486c623-3983-406f-9f04-21a0d994c828\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890890-c6w7z" Oct 31 13:30:00.611187 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.611180 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kcqv\" (UniqueName: \"kubernetes.io/projected/c0fb78e4-7046-48f0-a737-4ca457074027-kube-api-access-9kcqv\") pod \"maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890890-cnvdm\" (UID: \"c0fb78e4-7046-48f0-a737-4ca457074027\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890890-cnvdm" Oct 31 13:30:00.611410 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.611277 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvsd5\" (UniqueName: \"kubernetes.io/projected/24816799-6034-48e6-9711-d9a342c0ef28-kube-api-access-bvsd5\") pod \"maas-api-key-cleanup-e2e-worker-w0-107955-29890890-66f9d\" (UID: \"24816799-6034-48e6-9711-d9a342c0ef28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890890-66f9d" Oct 31 13:30:00.611410 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.611380 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22242dc7-c483-4f14-8e16-5f5e5f1a7ad4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890890-2k4sn\" (UID: \"22242dc7-c483-4f14-8e16-5f5e5f1a7ad4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890890-2k4sn" Oct 31 13:30:00.611534 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.611516 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92712c0d-d49c-46fe-8bad-a023a997d5f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890890-5bssh\" (UID: \"92712c0d-d49c-46fe-8bad-a023a997d5f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890890-5bssh" Oct 31 13:30:00.611604 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.611588 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e486c623-3983-406f-9f04-21a0d994c828-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890890-c6w7z\" (UID: \"e486c623-3983-406f-9f04-21a0d994c828\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890890-c6w7z" Oct 31 13:30:00.611749 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.611724 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/875da43d-e0a5-4fba-bd88-7758e8581e04-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890890-rlpk4\" (UID: \"875da43d-e0a5-4fba-bd88-7758e8581e04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890890-rlpk4" Oct 31 13:30:00.612119 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.612079 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22242dc7-c483-4f14-8e16-5f5e5f1a7ad4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890890-2k4sn\" (UID: \"22242dc7-c483-4f14-8e16-5f5e5f1a7ad4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890890-2k4sn" Oct 31 13:30:00.612223 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.612182 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92712c0d-d49c-46fe-8bad-a023a997d5f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890890-5bssh\" (UID: \"92712c0d-d49c-46fe-8bad-a023a997d5f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890890-5bssh" Oct 31 13:30:00.612338 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.612319 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e486c623-3983-406f-9f04-21a0d994c828-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890890-c6w7z\" (UID: \"e486c623-3983-406f-9f04-21a0d994c828\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890890-c6w7z" Oct 31 13:30:00.637914 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.637767 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98ldg\" (UniqueName: \"kubernetes.io/projected/92712c0d-d49c-46fe-8bad-a023a997d5f2-kube-api-access-98ldg\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890890-5bssh\" (UID: \"92712c0d-d49c-46fe-8bad-a023a997d5f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890890-5bssh" Oct 31 13:30:00.638149 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.638125 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tddl\" (UniqueName: \"kubernetes.io/projected/875da43d-e0a5-4fba-bd88-7758e8581e04-kube-api-access-9tddl\") pod \"maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890890-rlpk4\" (UID: \"875da43d-e0a5-4fba-bd88-7758e8581e04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890890-rlpk4" Oct 31 13:30:00.641069 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.641034 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7hxg\" (UniqueName: \"kubernetes.io/projected/e486c623-3983-406f-9f04-21a0d994c828-kube-api-access-z7hxg\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890890-c6w7z\" (UID: \"e486c623-3983-406f-9f04-21a0d994c828\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890890-c6w7z" Oct 31 13:30:00.641341 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.641315 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phnq6\" (UniqueName: \"kubernetes.io/projected/22242dc7-c483-4f14-8e16-5f5e5f1a7ad4-kube-api-access-phnq6\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890890-2k4sn\" (UID: \"22242dc7-c483-4f14-8e16-5f5e5f1a7ad4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890890-2k4sn" Oct 31 13:30:00.713362 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.713271 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kcqv\" (UniqueName: \"kubernetes.io/projected/c0fb78e4-7046-48f0-a737-4ca457074027-kube-api-access-9kcqv\") pod \"maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890890-cnvdm\" (UID: \"c0fb78e4-7046-48f0-a737-4ca457074027\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890890-cnvdm" Oct 31 13:30:00.713362 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.713351 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvsd5\" (UniqueName: \"kubernetes.io/projected/24816799-6034-48e6-9711-d9a342c0ef28-kube-api-access-bvsd5\") pod \"maas-api-key-cleanup-e2e-worker-w0-107955-29890890-66f9d\" (UID: \"24816799-6034-48e6-9711-d9a342c0ef28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890890-66f9d" Oct 31 13:30:00.713577 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.713556 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0fb78e4-7046-48f0-a737-4ca457074027-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890890-cnvdm\" (UID: \"c0fb78e4-7046-48f0-a737-4ca457074027\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890890-cnvdm" Oct 31 13:30:00.713663 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.713633 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24816799-6034-48e6-9711-d9a342c0ef28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-107955-29890890-66f9d\" (UID: \"24816799-6034-48e6-9711-d9a342c0ef28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890890-66f9d" Oct 31 13:30:00.714271 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.714247 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24816799-6034-48e6-9711-d9a342c0ef28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-107955-29890890-66f9d\" (UID: \"24816799-6034-48e6-9711-d9a342c0ef28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890890-66f9d" Oct 31 13:30:00.714379 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.714338 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0fb78e4-7046-48f0-a737-4ca457074027-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890890-cnvdm\" (UID: \"c0fb78e4-7046-48f0-a737-4ca457074027\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890890-cnvdm" Oct 31 13:30:00.733393 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.733349 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890890-5bssh" Oct 31 13:30:00.736762 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.736734 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kcqv\" (UniqueName: \"kubernetes.io/projected/c0fb78e4-7046-48f0-a737-4ca457074027-kube-api-access-9kcqv\") pod \"maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890890-cnvdm\" (UID: \"c0fb78e4-7046-48f0-a737-4ca457074027\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890890-cnvdm" Oct 31 13:30:00.737098 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.737075 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvsd5\" (UniqueName: \"kubernetes.io/projected/24816799-6034-48e6-9711-d9a342c0ef28-kube-api-access-bvsd5\") pod \"maas-api-key-cleanup-e2e-worker-w0-107955-29890890-66f9d\" (UID: \"24816799-6034-48e6-9711-d9a342c0ef28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890890-66f9d" Oct 31 13:30:00.741576 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.741550 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890890-2k4sn" Oct 31 13:30:00.837898 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.837792 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890890-rlpk4" Oct 31 13:30:00.848603 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.845015 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890890-c6w7z" Oct 31 13:30:00.863315 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.863083 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890890-66f9d" Oct 31 13:30:00.877214 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:00.876876 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890890-cnvdm" Oct 31 13:30:01.263788 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:30:01.263750 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode486c623_3983_406f_9f04_21a0d994c828.slice/crio-88a02e1cbfa2e409d0b2e6c0f12beb778e92491c7abbe07173ad17ed520f80c6 WatchSource:0}: Error finding container 88a02e1cbfa2e409d0b2e6c0f12beb778e92491c7abbe07173ad17ed520f80c6: Status 404 returned error can't find the container with id 88a02e1cbfa2e409d0b2e6c0f12beb778e92491c7abbe07173ad17ed520f80c6 Oct 31 13:30:01.265270 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:30:01.265243 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24816799_6034_48e6_9711_d9a342c0ef28.slice/crio-d2a6e0f938056fdbd5ab208cf58782cac36e9413823fe3b881df0cedf18445a7 WatchSource:0}: Error finding container d2a6e0f938056fdbd5ab208cf58782cac36e9413823fe3b881df0cedf18445a7: Status 404 returned error can't find the container with id d2a6e0f938056fdbd5ab208cf58782cac36e9413823fe3b881df0cedf18445a7 Oct 31 13:30:01.275694 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:01.275674 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890890-c6w7z"] Oct 31 13:30:01.285471 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:01.285446 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890890-66f9d"] Oct 31 13:30:01.293800 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:01.293751 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890890-2k4sn"] Oct 31 13:30:01.302723 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:01.302698 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890890-cnvdm"] Oct 31 13:30:01.307079 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:30:01.307047 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22242dc7_c483_4f14_8e16_5f5e5f1a7ad4.slice/crio-9d28cd3f69ac238d34f16c907124e7673a4dade4d682dfdc3d308ab0bda0fe22 WatchSource:0}: Error finding container 9d28cd3f69ac238d34f16c907124e7673a4dade4d682dfdc3d308ab0bda0fe22: Status 404 returned error can't find the container with id 9d28cd3f69ac238d34f16c907124e7673a4dade4d682dfdc3d308ab0bda0fe22 Oct 31 13:30:01.307742 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:30:01.307717 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0fb78e4_7046_48f0_a737_4ca457074027.slice/crio-bcd5ac5eb7829717059c16287b9b0503ead1ef52dea79c4dd0491391e15de109 WatchSource:0}: Error finding container bcd5ac5eb7829717059c16287b9b0503ead1ef52dea79c4dd0491391e15de109: Status 404 returned error can't find the container with id bcd5ac5eb7829717059c16287b9b0503ead1ef52dea79c4dd0491391e15de109 Oct 31 13:30:01.309059 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:01.309036 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890890-5bssh"] Oct 31 13:30:01.309905 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:30:01.309884 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92712c0d_d49c_46fe_8bad_a023a997d5f2.slice/crio-7959cab03024ac7d1e508c21eebecdebe462b2ff3efcb0aa63551da9d2ad9a6b WatchSource:0}: Error finding container 7959cab03024ac7d1e508c21eebecdebe462b2ff3efcb0aa63551da9d2ad9a6b: Status 404 returned error can't find the container with id 7959cab03024ac7d1e508c21eebecdebe462b2ff3efcb0aa63551da9d2ad9a6b Oct 31 13:30:01.370573 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:01.370548 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890890-rlpk4"] Oct 31 13:30:01.370708 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:30:01.370591 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod875da43d_e0a5_4fba_bd88_7758e8581e04.slice/crio-6797679d0bed8117a4b46bd50a44b9d064acbaf7234b8fcaf3f5a8de5e012c11 WatchSource:0}: Error finding container 6797679d0bed8117a4b46bd50a44b9d064acbaf7234b8fcaf3f5a8de5e012c11: Status 404 returned error can't find the container with id 6797679d0bed8117a4b46bd50a44b9d064acbaf7234b8fcaf3f5a8de5e012c11 Oct 31 13:30:01.769278 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:01.769242 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890890-66f9d" event={"ID":"24816799-6034-48e6-9711-d9a342c0ef28","Type":"ContainerStarted","Data":"d2a6e0f938056fdbd5ab208cf58782cac36e9413823fe3b881df0cedf18445a7"} Oct 31 13:30:01.770826 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:01.770799 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890890-5bssh" event={"ID":"92712c0d-d49c-46fe-8bad-a023a997d5f2","Type":"ContainerStarted","Data":"7959cab03024ac7d1e508c21eebecdebe462b2ff3efcb0aa63551da9d2ad9a6b"} Oct 31 13:30:01.772266 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:01.772241 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890890-2k4sn" event={"ID":"22242dc7-c483-4f14-8e16-5f5e5f1a7ad4","Type":"ContainerStarted","Data":"9d28cd3f69ac238d34f16c907124e7673a4dade4d682dfdc3d308ab0bda0fe22"} Oct 31 13:30:01.773611 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:01.773583 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890890-cnvdm" event={"ID":"c0fb78e4-7046-48f0-a737-4ca457074027","Type":"ContainerStarted","Data":"bcd5ac5eb7829717059c16287b9b0503ead1ef52dea79c4dd0491391e15de109"} Oct 31 13:30:01.775044 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:01.775019 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890890-rlpk4" event={"ID":"875da43d-e0a5-4fba-bd88-7758e8581e04","Type":"ContainerStarted","Data":"6797679d0bed8117a4b46bd50a44b9d064acbaf7234b8fcaf3f5a8de5e012c11"} Oct 31 13:30:01.776287 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:01.776265 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890890-c6w7z" event={"ID":"e486c623-3983-406f-9f04-21a0d994c828","Type":"ContainerStarted","Data":"88a02e1cbfa2e409d0b2e6c0f12beb778e92491c7abbe07173ad17ed520f80c6"} Oct 31 13:30:02.788585 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:02.788547 5693 generic.go:356] "Generic (PLEG): container finished" podID="e486c623-3983-406f-9f04-21a0d994c828" containerID="16134cd9535caebe3dce66bd3cdb7b2fde0182b82b814e99f1b97543ac9f334c" exitCode=60 Oct 31 13:30:02.789060 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:02.788630 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890890-c6w7z" event={"ID":"e486c623-3983-406f-9f04-21a0d994c828","Type":"ContainerDied","Data":"16134cd9535caebe3dce66bd3cdb7b2fde0182b82b814e99f1b97543ac9f334c"} Oct 31 13:30:02.789060 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:02.789032 5693 scope.go:122] "RemoveContainer" containerID="16134cd9535caebe3dce66bd3cdb7b2fde0182b82b814e99f1b97543ac9f334c" Oct 31 13:30:02.790915 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:02.790613 5693 generic.go:356] "Generic (PLEG): container finished" podID="24816799-6034-48e6-9711-d9a342c0ef28" containerID="23f2878cc70fca9fc0b13c1cd257d4416bd38df5e16b41de03c9f3dad1df0934" exitCode=60 Oct 31 13:30:02.790915 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:02.790859 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890890-66f9d" event={"ID":"24816799-6034-48e6-9711-d9a342c0ef28","Type":"ContainerDied","Data":"23f2878cc70fca9fc0b13c1cd257d4416bd38df5e16b41de03c9f3dad1df0934"} Oct 31 13:30:02.791083 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:02.791039 5693 scope.go:122] "RemoveContainer" containerID="23f2878cc70fca9fc0b13c1cd257d4416bd38df5e16b41de03c9f3dad1df0934" Oct 31 13:30:02.793599 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:02.793573 5693 generic.go:356] "Generic (PLEG): container finished" podID="92712c0d-d49c-46fe-8bad-a023a997d5f2" containerID="d618ba08d9abe4c783fff0802233cbc217905ad58c7f1c41c08476c8ca069054" exitCode=60 Oct 31 13:30:02.793772 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:02.793693 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890890-5bssh" event={"ID":"92712c0d-d49c-46fe-8bad-a023a997d5f2","Type":"ContainerDied","Data":"d618ba08d9abe4c783fff0802233cbc217905ad58c7f1c41c08476c8ca069054"} Oct 31 13:30:02.794033 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:02.794011 5693 scope.go:122] "RemoveContainer" containerID="d618ba08d9abe4c783fff0802233cbc217905ad58c7f1c41c08476c8ca069054" Oct 31 13:30:02.798742 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:02.798717 5693 generic.go:356] "Generic (PLEG): container finished" podID="22242dc7-c483-4f14-8e16-5f5e5f1a7ad4" containerID="90277a9591b5cfe830ea4486efffaf7f1bc7aa8b08a3078657375aa9cb86ba02" exitCode=60 Oct 31 13:30:02.798848 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:02.798757 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890890-2k4sn" event={"ID":"22242dc7-c483-4f14-8e16-5f5e5f1a7ad4","Type":"ContainerDied","Data":"90277a9591b5cfe830ea4486efffaf7f1bc7aa8b08a3078657375aa9cb86ba02"} Oct 31 13:30:02.799088 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:02.799048 5693 scope.go:122] "RemoveContainer" containerID="90277a9591b5cfe830ea4486efffaf7f1bc7aa8b08a3078657375aa9cb86ba02" Oct 31 13:30:02.800698 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:02.800675 5693 generic.go:356] "Generic (PLEG): container finished" podID="c0fb78e4-7046-48f0-a737-4ca457074027" containerID="150abb014929880d45b86d822c5c9e2c7fd2dd6730e41f156449efb15f828f7e" exitCode=60 Oct 31 13:30:02.800798 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:02.800716 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890890-cnvdm" event={"ID":"c0fb78e4-7046-48f0-a737-4ca457074027","Type":"ContainerDied","Data":"150abb014929880d45b86d822c5c9e2c7fd2dd6730e41f156449efb15f828f7e"} Oct 31 13:30:02.801021 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:02.801003 5693 scope.go:122] "RemoveContainer" containerID="150abb014929880d45b86d822c5c9e2c7fd2dd6730e41f156449efb15f828f7e" Oct 31 13:30:02.804797 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:02.802661 5693 generic.go:356] "Generic (PLEG): container finished" podID="875da43d-e0a5-4fba-bd88-7758e8581e04" containerID="56e2c6f6e8265104bfe971c77a533dcd85f720ba52ef41a1611060d1696ec081" exitCode=60 Oct 31 13:30:02.804797 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:02.802690 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890890-rlpk4" event={"ID":"875da43d-e0a5-4fba-bd88-7758e8581e04","Type":"ContainerDied","Data":"56e2c6f6e8265104bfe971c77a533dcd85f720ba52ef41a1611060d1696ec081"} Oct 31 13:30:02.804797 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:02.802951 5693 scope.go:122] "RemoveContainer" containerID="56e2c6f6e8265104bfe971c77a533dcd85f720ba52ef41a1611060d1696ec081" Oct 31 13:30:03.052985 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:03.052939 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-a13836/test-model-a13836-kserve-5b7f69c4f5-4gl49"] Oct 31 13:30:03.067784 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:03.067741 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a13836/test-model-a13836-kserve-5b7f69c4f5-4gl49"] Oct 31 13:30:03.067964 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:03.067869 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a13836/test-model-a13836-kserve-5b7f69c4f5-4gl49" Oct 31 13:30:03.074587 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:03.074153 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a13836\"/\"openshift-service-ca.crt\"" Oct 31 13:30:03.074587 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:03.074320 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a13836\"/\"default-dockercfg-r96hc\"" Oct 31 13:30:03.074587 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:03.074396 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a13836\"/\"kube-root-ca.crt\"" Oct 31 13:30:03.074587 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:03.074439 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a13836\"/\"test-model-a13836-kserve-self-signed-certs\"" Oct 31 13:30:03.184621 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:03.184543 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d58e66c-aad7-488b-8315-3f357c4220c0-tmp-dir\") pod \"test-model-a13836-kserve-5b7f69c4f5-4gl49\" (UID: \"0d58e66c-aad7-488b-8315-3f357c4220c0\") " pod="ai-tenant-e2e-inf-b-a13836/test-model-a13836-kserve-5b7f69c4f5-4gl49" Oct 31 13:30:03.184847 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:03.184694 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d58e66c-aad7-488b-8315-3f357c4220c0-model-cache\") pod \"test-model-a13836-kserve-5b7f69c4f5-4gl49\" (UID: \"0d58e66c-aad7-488b-8315-3f357c4220c0\") " pod="ai-tenant-e2e-inf-b-a13836/test-model-a13836-kserve-5b7f69c4f5-4gl49" Oct 31 13:30:03.184847 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:03.184840 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d58e66c-aad7-488b-8315-3f357c4220c0-tls-certs\") pod \"test-model-a13836-kserve-5b7f69c4f5-4gl49\" (UID: \"0d58e66c-aad7-488b-8315-3f357c4220c0\") " pod="ai-tenant-e2e-inf-b-a13836/test-model-a13836-kserve-5b7f69c4f5-4gl49" Oct 31 13:30:03.184969 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:03.184871 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d58e66c-aad7-488b-8315-3f357c4220c0-home\") pod \"test-model-a13836-kserve-5b7f69c4f5-4gl49\" (UID: \"0d58e66c-aad7-488b-8315-3f357c4220c0\") " pod="ai-tenant-e2e-inf-b-a13836/test-model-a13836-kserve-5b7f69c4f5-4gl49" Oct 31 13:30:03.184969 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:03.184930 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vnzx\" (UniqueName: \"kubernetes.io/projected/0d58e66c-aad7-488b-8315-3f357c4220c0-kube-api-access-2vnzx\") pod \"test-model-a13836-kserve-5b7f69c4f5-4gl49\" (UID: \"0d58e66c-aad7-488b-8315-3f357c4220c0\") " pod="ai-tenant-e2e-inf-b-a13836/test-model-a13836-kserve-5b7f69c4f5-4gl49" Oct 31 13:30:03.185074 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:03.184986 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d58e66c-aad7-488b-8315-3f357c4220c0-dshm\") pod \"test-model-a13836-kserve-5b7f69c4f5-4gl49\" (UID: \"0d58e66c-aad7-488b-8315-3f357c4220c0\") " pod="ai-tenant-e2e-inf-b-a13836/test-model-a13836-kserve-5b7f69c4f5-4gl49" Oct 31 13:30:03.300664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:03.287012 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d58e66c-aad7-488b-8315-3f357c4220c0-tmp-dir\") pod \"test-model-a13836-kserve-5b7f69c4f5-4gl49\" (UID: \"0d58e66c-aad7-488b-8315-3f357c4220c0\") " pod="ai-tenant-e2e-inf-b-a13836/test-model-a13836-kserve-5b7f69c4f5-4gl49" Oct 31 13:30:03.301349 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:03.301315 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d58e66c-aad7-488b-8315-3f357c4220c0-model-cache\") pod \"test-model-a13836-kserve-5b7f69c4f5-4gl49\" (UID: \"0d58e66c-aad7-488b-8315-3f357c4220c0\") " pod="ai-tenant-e2e-inf-b-a13836/test-model-a13836-kserve-5b7f69c4f5-4gl49" Oct 31 13:30:03.301632 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:03.301615 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d58e66c-aad7-488b-8315-3f357c4220c0-tls-certs\") pod \"test-model-a13836-kserve-5b7f69c4f5-4gl49\" (UID: \"0d58e66c-aad7-488b-8315-3f357c4220c0\") " pod="ai-tenant-e2e-inf-b-a13836/test-model-a13836-kserve-5b7f69c4f5-4gl49" Oct 31 13:30:03.301817 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:03.301801 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d58e66c-aad7-488b-8315-3f357c4220c0-home\") pod \"test-model-a13836-kserve-5b7f69c4f5-4gl49\" (UID: \"0d58e66c-aad7-488b-8315-3f357c4220c0\") " pod="ai-tenant-e2e-inf-b-a13836/test-model-a13836-kserve-5b7f69c4f5-4gl49" Oct 31 13:30:03.301971 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:03.301955 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vnzx\" (UniqueName: \"kubernetes.io/projected/0d58e66c-aad7-488b-8315-3f357c4220c0-kube-api-access-2vnzx\") pod \"test-model-a13836-kserve-5b7f69c4f5-4gl49\" (UID: \"0d58e66c-aad7-488b-8315-3f357c4220c0\") " pod="ai-tenant-e2e-inf-b-a13836/test-model-a13836-kserve-5b7f69c4f5-4gl49" Oct 31 13:30:03.302102 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:03.302090 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d58e66c-aad7-488b-8315-3f357c4220c0-dshm\") pod \"test-model-a13836-kserve-5b7f69c4f5-4gl49\" (UID: \"0d58e66c-aad7-488b-8315-3f357c4220c0\") " pod="ai-tenant-e2e-inf-b-a13836/test-model-a13836-kserve-5b7f69c4f5-4gl49" Oct 31 13:30:03.308257 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:03.305444 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d58e66c-aad7-488b-8315-3f357c4220c0-home\") pod \"test-model-a13836-kserve-5b7f69c4f5-4gl49\" (UID: \"0d58e66c-aad7-488b-8315-3f357c4220c0\") " pod="ai-tenant-e2e-inf-b-a13836/test-model-a13836-kserve-5b7f69c4f5-4gl49" Oct 31 13:30:03.321136 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:03.290889 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d58e66c-aad7-488b-8315-3f357c4220c0-tmp-dir\") pod \"test-model-a13836-kserve-5b7f69c4f5-4gl49\" (UID: \"0d58e66c-aad7-488b-8315-3f357c4220c0\") " pod="ai-tenant-e2e-inf-b-a13836/test-model-a13836-kserve-5b7f69c4f5-4gl49" Oct 31 13:30:03.321362 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:03.305709 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d58e66c-aad7-488b-8315-3f357c4220c0-model-cache\") pod \"test-model-a13836-kserve-5b7f69c4f5-4gl49\" (UID: \"0d58e66c-aad7-488b-8315-3f357c4220c0\") " pod="ai-tenant-e2e-inf-b-a13836/test-model-a13836-kserve-5b7f69c4f5-4gl49" Oct 31 13:30:03.321480 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:03.316289 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d58e66c-aad7-488b-8315-3f357c4220c0-tls-certs\") pod \"test-model-a13836-kserve-5b7f69c4f5-4gl49\" (UID: \"0d58e66c-aad7-488b-8315-3f357c4220c0\") " pod="ai-tenant-e2e-inf-b-a13836/test-model-a13836-kserve-5b7f69c4f5-4gl49" Oct 31 13:30:03.321828 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:03.321015 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d58e66c-aad7-488b-8315-3f357c4220c0-dshm\") pod \"test-model-a13836-kserve-5b7f69c4f5-4gl49\" (UID: \"0d58e66c-aad7-488b-8315-3f357c4220c0\") " pod="ai-tenant-e2e-inf-b-a13836/test-model-a13836-kserve-5b7f69c4f5-4gl49" Oct 31 13:30:03.334892 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:03.328615 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vnzx\" (UniqueName: \"kubernetes.io/projected/0d58e66c-aad7-488b-8315-3f357c4220c0-kube-api-access-2vnzx\") pod \"test-model-a13836-kserve-5b7f69c4f5-4gl49\" (UID: \"0d58e66c-aad7-488b-8315-3f357c4220c0\") " pod="ai-tenant-e2e-inf-b-a13836/test-model-a13836-kserve-5b7f69c4f5-4gl49" Oct 31 13:30:03.403662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:03.396374 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a13836/test-model-a13836-kserve-5b7f69c4f5-4gl49" Oct 31 13:30:03.770656 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:30:03.770605 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d58e66c_aad7_488b_8315_3f357c4220c0.slice/crio-f46ab19702bf8817c27e9490f995707d861744ed125a0426cf2c2fee2e9319e5 WatchSource:0}: Error finding container f46ab19702bf8817c27e9490f995707d861744ed125a0426cf2c2fee2e9319e5: Status 404 returned error can't find the container with id f46ab19702bf8817c27e9490f995707d861744ed125a0426cf2c2fee2e9319e5 Oct 31 13:30:03.773406 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:03.773384 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a13836/test-model-a13836-kserve-5b7f69c4f5-4gl49"] Oct 31 13:30:03.815278 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:03.815241 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890890-5bssh" event={"ID":"92712c0d-d49c-46fe-8bad-a023a997d5f2","Type":"ContainerStarted","Data":"9b4f853f465d43f3cc96ea41ccb538d195c5c3c8f152e48d73922608673dd1be"} Oct 31 13:30:03.817690 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:03.817662 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890890-2k4sn" event={"ID":"22242dc7-c483-4f14-8e16-5f5e5f1a7ad4","Type":"ContainerStarted","Data":"191b9f550933f001d9616a0d32d50512bb12b0dbe7d4ca5852fc366026d94994"} Oct 31 13:30:03.820574 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:03.819989 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890890-cnvdm" event={"ID":"c0fb78e4-7046-48f0-a737-4ca457074027","Type":"ContainerStarted","Data":"21db924a4f462ae6f9685d1b48924dd06629485c48e594b4605df966bfe17e68"} Oct 31 13:30:03.821912 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:03.821890 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890890-rlpk4" event={"ID":"875da43d-e0a5-4fba-bd88-7758e8581e04","Type":"ContainerStarted","Data":"afae573979bf4a89d3ccac54d689cfea17ac3f2b286ca75c879c1d4e17a317b7"} Oct 31 13:30:03.823249 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:03.823224 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a13836/test-model-a13836-kserve-5b7f69c4f5-4gl49" event={"ID":"0d58e66c-aad7-488b-8315-3f357c4220c0","Type":"ContainerStarted","Data":"f46ab19702bf8817c27e9490f995707d861744ed125a0426cf2c2fee2e9319e5"} Oct 31 13:30:03.825110 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:03.825090 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890890-c6w7z" event={"ID":"e486c623-3983-406f-9f04-21a0d994c828","Type":"ContainerStarted","Data":"f4c7646be5184f7bbcf808d063a3d9c64ac6d3ab053d7a2fab55e74091c89d40"} Oct 31 13:30:03.827109 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:03.827088 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890890-66f9d" event={"ID":"24816799-6034-48e6-9711-d9a342c0ef28","Type":"ContainerStarted","Data":"9443c45589e49834db218d8caaf55691a6780c3bfcbc18191a98cdc8657d3a0c"} Oct 31 13:30:03.843689 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:03.843611 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890890-5bssh" podStartSLOduration=3.84359619 podStartE2EDuration="3.84359619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:03.836893158 +0000 UTC m=+3454.544479459" watchObservedRunningTime="2026-10-31 13:30:03.84359619 +0000 UTC m=+3454.551182485" Oct 31 13:30:03.866063 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:03.866012 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890890-rlpk4" podStartSLOduration=3.865998201 podStartE2EDuration="3.865998201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:03.863066599 +0000 UTC m=+3454.570652894" watchObservedRunningTime="2026-10-31 13:30:03.865998201 +0000 UTC m=+3454.573584508" Oct 31 13:30:03.889908 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:03.889848 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890890-66f9d" podStartSLOduration=3.88982853 podStartE2EDuration="3.88982853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:03.887507458 +0000 UTC m=+3454.595093762" watchObservedRunningTime="2026-10-31 13:30:03.88982853 +0000 UTC m=+3454.597414828" Oct 31 13:30:03.916732 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:03.916683 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890890-c6w7z" podStartSLOduration=3.916666762 podStartE2EDuration="3.916666762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:03.911047907 +0000 UTC m=+3454.618634205" watchObservedRunningTime="2026-10-31 13:30:03.916666762 +0000 UTC m=+3454.624253054" Oct 31 13:30:03.940589 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:03.940540 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890890-cnvdm" podStartSLOduration=3.940525008 podStartE2EDuration="3.940525008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:03.936685063 +0000 UTC m=+3454.644271350" watchObservedRunningTime="2026-10-31 13:30:03.940525008 +0000 UTC m=+3454.648111306" Oct 31 13:30:04.003692 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:03.998775 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890890-2k4sn" podStartSLOduration=3.998751067 podStartE2EDuration="3.998751067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:03.955137902 +0000 UTC m=+3454.662724199" watchObservedRunningTime="2026-10-31 13:30:03.998751067 +0000 UTC m=+3454.706337415" Oct 31 13:30:04.837435 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:04.837353 5693 generic.go:356] "Generic (PLEG): container finished" podID="875da43d-e0a5-4fba-bd88-7758e8581e04" containerID="afae573979bf4a89d3ccac54d689cfea17ac3f2b286ca75c879c1d4e17a317b7" exitCode=60 Oct 31 13:30:04.837968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:04.837437 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890890-rlpk4" event={"ID":"875da43d-e0a5-4fba-bd88-7758e8581e04","Type":"ContainerDied","Data":"afae573979bf4a89d3ccac54d689cfea17ac3f2b286ca75c879c1d4e17a317b7"} Oct 31 13:30:04.837968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:04.837499 5693 scope.go:122] "RemoveContainer" containerID="56e2c6f6e8265104bfe971c77a533dcd85f720ba52ef41a1611060d1696ec081" Oct 31 13:30:04.837968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:04.837839 5693 scope.go:122] "RemoveContainer" containerID="afae573979bf4a89d3ccac54d689cfea17ac3f2b286ca75c879c1d4e17a317b7" Oct 31 13:30:04.838318 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:30:04.838294 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890890-rlpk4_odh-ai-gateway-infra(875da43d-e0a5-4fba-bd88-7758e8581e04)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890890-rlpk4" podUID="875da43d-e0a5-4fba-bd88-7758e8581e04" Oct 31 13:30:04.839658 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:04.839615 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a13836/test-model-a13836-kserve-5b7f69c4f5-4gl49" event={"ID":"0d58e66c-aad7-488b-8315-3f357c4220c0","Type":"ContainerStarted","Data":"1491d613fcee0031ddda199c00a7626fb2060f71ea742bd8727ec324f6d10f62"} Oct 31 13:30:04.842700 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:04.842067 5693 generic.go:356] "Generic (PLEG): container finished" podID="e486c623-3983-406f-9f04-21a0d994c828" containerID="f4c7646be5184f7bbcf808d063a3d9c64ac6d3ab053d7a2fab55e74091c89d40" exitCode=60 Oct 31 13:30:04.842700 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:04.842155 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890890-c6w7z" event={"ID":"e486c623-3983-406f-9f04-21a0d994c828","Type":"ContainerDied","Data":"f4c7646be5184f7bbcf808d063a3d9c64ac6d3ab053d7a2fab55e74091c89d40"} Oct 31 13:30:04.842700 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:04.842467 5693 scope.go:122] "RemoveContainer" containerID="f4c7646be5184f7bbcf808d063a3d9c64ac6d3ab053d7a2fab55e74091c89d40" Oct 31 13:30:04.842946 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:30:04.842923 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890890-c6w7z_odh-ai-gateway-infra(e486c623-3983-406f-9f04-21a0d994c828)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890890-c6w7z" podUID="e486c623-3983-406f-9f04-21a0d994c828" Oct 31 13:30:04.844401 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:04.844380 5693 generic.go:356] "Generic (PLEG): container finished" podID="24816799-6034-48e6-9711-d9a342c0ef28" containerID="9443c45589e49834db218d8caaf55691a6780c3bfcbc18191a98cdc8657d3a0c" exitCode=60 Oct 31 13:30:04.844511 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:04.844460 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890890-66f9d" event={"ID":"24816799-6034-48e6-9711-d9a342c0ef28","Type":"ContainerDied","Data":"9443c45589e49834db218d8caaf55691a6780c3bfcbc18191a98cdc8657d3a0c"} Oct 31 13:30:04.844762 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:04.844746 5693 scope.go:122] "RemoveContainer" containerID="9443c45589e49834db218d8caaf55691a6780c3bfcbc18191a98cdc8657d3a0c" Oct 31 13:30:04.845115 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:30:04.845096 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-107955-29890890-66f9d_odh-ai-gateway-infra(24816799-6034-48e6-9711-d9a342c0ef28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890890-66f9d" podUID="24816799-6034-48e6-9711-d9a342c0ef28" Oct 31 13:30:04.846555 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:04.846523 5693 generic.go:356] "Generic (PLEG): container finished" podID="92712c0d-d49c-46fe-8bad-a023a997d5f2" containerID="9b4f853f465d43f3cc96ea41ccb538d195c5c3c8f152e48d73922608673dd1be" exitCode=60 Oct 31 13:30:04.846665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:04.846597 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890890-5bssh" event={"ID":"92712c0d-d49c-46fe-8bad-a023a997d5f2","Type":"ContainerDied","Data":"9b4f853f465d43f3cc96ea41ccb538d195c5c3c8f152e48d73922608673dd1be"} Oct 31 13:30:04.846984 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:04.846970 5693 scope.go:122] "RemoveContainer" containerID="9b4f853f465d43f3cc96ea41ccb538d195c5c3c8f152e48d73922608673dd1be" Oct 31 13:30:04.847282 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:30:04.847261 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890890-5bssh_odh-ai-gateway-infra(92712c0d-d49c-46fe-8bad-a023a997d5f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890890-5bssh" podUID="92712c0d-d49c-46fe-8bad-a023a997d5f2" Oct 31 13:30:04.849092 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:04.849071 5693 generic.go:356] "Generic (PLEG): container finished" podID="22242dc7-c483-4f14-8e16-5f5e5f1a7ad4" containerID="191b9f550933f001d9616a0d32d50512bb12b0dbe7d4ca5852fc366026d94994" exitCode=60 Oct 31 13:30:04.849218 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:04.849155 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890890-2k4sn" event={"ID":"22242dc7-c483-4f14-8e16-5f5e5f1a7ad4","Type":"ContainerDied","Data":"191b9f550933f001d9616a0d32d50512bb12b0dbe7d4ca5852fc366026d94994"} Oct 31 13:30:04.849368 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:04.849352 5693 scope.go:122] "RemoveContainer" containerID="191b9f550933f001d9616a0d32d50512bb12b0dbe7d4ca5852fc366026d94994" Oct 31 13:30:04.849611 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:30:04.849593 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890890-2k4sn_odh-ai-gateway-infra(22242dc7-c483-4f14-8e16-5f5e5f1a7ad4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890890-2k4sn" podUID="22242dc7-c483-4f14-8e16-5f5e5f1a7ad4" Oct 31 13:30:04.851159 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:04.851139 5693 generic.go:356] "Generic (PLEG): container finished" podID="c0fb78e4-7046-48f0-a737-4ca457074027" containerID="21db924a4f462ae6f9685d1b48924dd06629485c48e594b4605df966bfe17e68" exitCode=60 Oct 31 13:30:04.851266 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:04.851171 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890890-cnvdm" event={"ID":"c0fb78e4-7046-48f0-a737-4ca457074027","Type":"ContainerDied","Data":"21db924a4f462ae6f9685d1b48924dd06629485c48e594b4605df966bfe17e68"} Oct 31 13:30:04.851416 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:04.851398 5693 scope.go:122] "RemoveContainer" containerID="21db924a4f462ae6f9685d1b48924dd06629485c48e594b4605df966bfe17e68" Oct 31 13:30:04.851664 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:30:04.851628 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890890-cnvdm_odh-ai-gateway-infra(c0fb78e4-7046-48f0-a737-4ca457074027)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890890-cnvdm" podUID="c0fb78e4-7046-48f0-a737-4ca457074027" Oct 31 13:30:04.854600 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:04.854584 5693 scope.go:122] "RemoveContainer" containerID="16134cd9535caebe3dce66bd3cdb7b2fde0182b82b814e99f1b97543ac9f334c" Oct 31 13:30:04.872388 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:04.872361 5693 scope.go:122] "RemoveContainer" containerID="23f2878cc70fca9fc0b13c1cd257d4416bd38df5e16b41de03c9f3dad1df0934" Oct 31 13:30:04.887873 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:04.887792 5693 scope.go:122] "RemoveContainer" containerID="d618ba08d9abe4c783fff0802233cbc217905ad58c7f1c41c08476c8ca069054" Oct 31 13:30:04.973474 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:04.973435 5693 scope.go:122] "RemoveContainer" containerID="90277a9591b5cfe830ea4486efffaf7f1bc7aa8b08a3078657375aa9cb86ba02" Oct 31 13:30:04.991607 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:04.991577 5693 scope.go:122] "RemoveContainer" containerID="150abb014929880d45b86d822c5c9e2c7fd2dd6730e41f156449efb15f828f7e" Oct 31 13:30:05.017983 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:05.017929 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-a13836/test-model-a13836-kserve-5b7f69c4f5-4gl49" podStartSLOduration=3.017913514 podStartE2EDuration="3.017913514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:05.01538481 +0000 UTC m=+3455.722971111" watchObservedRunningTime="2026-10-31 13:30:05.017913514 +0000 UTC m=+3455.725499809" Oct 31 13:30:05.396512 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:05.396480 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-a13836/test-model-a13836-kserve-5b7f69c4f5-4gl49" Oct 31 13:30:05.863232 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:05.863140 5693 scope.go:122] "RemoveContainer" containerID="191b9f550933f001d9616a0d32d50512bb12b0dbe7d4ca5852fc366026d94994" Oct 31 13:30:05.863635 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:30:05.863579 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890890-2k4sn_odh-ai-gateway-infra(22242dc7-c483-4f14-8e16-5f5e5f1a7ad4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890890-2k4sn" podUID="22242dc7-c483-4f14-8e16-5f5e5f1a7ad4" Oct 31 13:30:05.865088 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:05.865061 5693 scope.go:122] "RemoveContainer" containerID="21db924a4f462ae6f9685d1b48924dd06629485c48e594b4605df966bfe17e68" Oct 31 13:30:05.865471 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:30:05.865442 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890890-cnvdm_odh-ai-gateway-infra(c0fb78e4-7046-48f0-a737-4ca457074027)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890890-cnvdm" podUID="c0fb78e4-7046-48f0-a737-4ca457074027" Oct 31 13:30:05.866862 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:05.866842 5693 scope.go:122] "RemoveContainer" containerID="afae573979bf4a89d3ccac54d689cfea17ac3f2b286ca75c879c1d4e17a317b7" Oct 31 13:30:05.867285 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:30:05.867258 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890890-rlpk4_odh-ai-gateway-infra(875da43d-e0a5-4fba-bd88-7758e8581e04)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890890-rlpk4" podUID="875da43d-e0a5-4fba-bd88-7758e8581e04" Oct 31 13:30:05.868724 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:05.868707 5693 scope.go:122] "RemoveContainer" containerID="f4c7646be5184f7bbcf808d063a3d9c64ac6d3ab053d7a2fab55e74091c89d40" Oct 31 13:30:05.869081 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:30:05.869057 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890890-c6w7z_odh-ai-gateway-infra(e486c623-3983-406f-9f04-21a0d994c828)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890890-c6w7z" podUID="e486c623-3983-406f-9f04-21a0d994c828" Oct 31 13:30:05.870480 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:05.870463 5693 scope.go:122] "RemoveContainer" containerID="9443c45589e49834db218d8caaf55691a6780c3bfcbc18191a98cdc8657d3a0c" Oct 31 13:30:05.870812 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:30:05.870792 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-107955-29890890-66f9d_odh-ai-gateway-infra(24816799-6034-48e6-9711-d9a342c0ef28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890890-66f9d" podUID="24816799-6034-48e6-9711-d9a342c0ef28" Oct 31 13:30:05.872308 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:05.872289 5693 scope.go:122] "RemoveContainer" containerID="9b4f853f465d43f3cc96ea41ccb538d195c5c3c8f152e48d73922608673dd1be" Oct 31 13:30:05.872561 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:30:05.872540 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890890-5bssh_odh-ai-gateway-infra(92712c0d-d49c-46fe-8bad-a023a997d5f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890890-5bssh" podUID="92712c0d-d49c-46fe-8bad-a023a997d5f2" Oct 31 13:30:08.966916 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:08.966875 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7"] Oct 31 13:30:13.414441 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:13.414403 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-a13836/test-model-a13836-kserve-5b7f69c4f5-4gl49" Oct 31 13:30:13.416306 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:13.416278 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-a13836/test-model-a13836-kserve-5b7f69c4f5-4gl49" Oct 31 13:30:13.972390 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:13.972348 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7" podUID="48fa6091-7341-4168-83da-93f772457206" containerName="payload-processing" containerID="cri-o://f0272db704c281abdbbecbfc522e345bc2a5cf1d7ef9dbeddc80c93fce3b31fd" gracePeriod=25 Oct 31 13:30:14.035996 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.035961 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-a13836/test-model-a13836-kserve-5b7f69c4f5-4gl49" Oct 31 13:30:14.413893 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.413864 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7" Oct 31 13:30:14.456281 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.455821 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890890-tcgwk"] Oct 31 13:30:14.457509 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.457479 5693 cpu_manager.go:414] "containerMap: removing container" podUID="48fa6091-7341-4168-83da-93f772457206" containerName="payload-processing" Oct 31 13:30:14.457509 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.457508 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48fa6091-7341-4168-83da-93f772457206" containerName="payload-processing" Oct 31 13:30:14.457904 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.457883 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="48fa6091-7341-4168-83da-93f772457206" containerName="payload-processing" Oct 31 13:30:14.463041 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.462994 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890890-tcgwk" Oct 31 13:30:14.518119 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.513734 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890890-m2s7m"] Oct 31 13:30:14.547955 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.547920 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890890-pnqpx"] Oct 31 13:30:14.549751 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.549574 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890890-m2s7m" Oct 31 13:30:14.555604 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.555576 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48fa6091-7341-4168-83da-93f772457206-kube-api-access-kkrlg\" (UniqueName: \"kubernetes.io/projected/48fa6091-7341-4168-83da-93f772457206-kube-api-access-kkrlg\") pod \"48fa6091-7341-4168-83da-93f772457206\" (UID: \"48fa6091-7341-4168-83da-93f772457206\") " Oct 31 13:30:14.555735 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.555692 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48fa6091-7341-4168-83da-93f772457206-provider-credentials\" (UniqueName: \"kubernetes.io/projected/48fa6091-7341-4168-83da-93f772457206-provider-credentials\") pod \"48fa6091-7341-4168-83da-93f772457206\" (UID: \"48fa6091-7341-4168-83da-93f772457206\") " Oct 31 13:30:14.555980 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.555961 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48fa6091-7341-4168-83da-93f772457206-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48fa6091-7341-4168-83da-93f772457206-routing-overlay-volume\") pod \"48fa6091-7341-4168-83da-93f772457206\" (UID: \"48fa6091-7341-4168-83da-93f772457206\") " Oct 31 13:30:14.556052 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.556039 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48fa6091-7341-4168-83da-93f772457206-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48fa6091-7341-4168-83da-93f772457206-plugins-config-volume\") pod \"48fa6091-7341-4168-83da-93f772457206\" (UID: \"48fa6091-7341-4168-83da-93f772457206\") " Oct 31 13:30:14.556398 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.556355 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48d1b4b9-adf7-48d9-bbae-6830a2331143-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890890-tcgwk\" (UID: \"48d1b4b9-adf7-48d9-bbae-6830a2331143\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890890-tcgwk" Oct 31 13:30:14.556757 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.556738 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfzhr\" (UniqueName: \"kubernetes.io/projected/48d1b4b9-adf7-48d9-bbae-6830a2331143-kube-api-access-jfzhr\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890890-tcgwk\" (UID: \"48d1b4b9-adf7-48d9-bbae-6830a2331143\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890890-tcgwk" Oct 31 13:30:14.558205 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.557679 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48fa6091-7341-4168-83da-93f772457206-plugins-config-volume" pod "48fa6091-7341-4168-83da-93f772457206" (UID: "48fa6091-7341-4168-83da-93f772457206"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:14.558205 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.557951 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48fa6091-7341-4168-83da-93f772457206-routing-overlay-volume" pod "48fa6091-7341-4168-83da-93f772457206" (UID: "48fa6091-7341-4168-83da-93f772457206"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:14.559558 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.559522 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890890-2ptgp"] Oct 31 13:30:14.559868 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.559851 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890890-pnqpx" Oct 31 13:30:14.560133 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.560099 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48fa6091-7341-4168-83da-93f772457206-kube-api-access-kkrlg" pod "48fa6091-7341-4168-83da-93f772457206" (UID: "48fa6091-7341-4168-83da-93f772457206"). InnerVolumeSpecName "kube-api-access-kkrlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:14.560941 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.560907 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48fa6091-7341-4168-83da-93f772457206-provider-credentials" pod "48fa6091-7341-4168-83da-93f772457206" (UID: "48fa6091-7341-4168-83da-93f772457206"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:14.572837 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.571849 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890890-bh5qm"] Oct 31 13:30:14.573827 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.573678 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890890-2ptgp" Oct 31 13:30:14.612070 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.612037 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890890-tcgwk"] Oct 31 13:30:14.612205 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.612095 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890890-m2s7m"] Oct 31 13:30:14.612205 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.612180 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-bh5qm" Oct 31 13:30:14.615039 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.615016 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890890-pnqpx"] Oct 31 13:30:14.623239 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.623179 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890890-2ptgp"] Oct 31 13:30:14.643687 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.642087 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890890-bh5qm"] Oct 31 13:30:14.665121 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.665085 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1055f537-d4e4-46fc-b684-1ec68a9bda17-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890890-m2s7m\" (UID: \"1055f537-d4e4-46fc-b684-1ec68a9bda17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890890-m2s7m" Oct 31 13:30:14.667882 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.667843 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1369557-2959-4d29-8756-bf9ddbd4b610-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890890-bh5qm\" (UID: \"e1369557-2959-4d29-8756-bf9ddbd4b610\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-bh5qm" Oct 31 13:30:14.668071 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.667942 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfzhr\" (UniqueName: \"kubernetes.io/projected/48d1b4b9-adf7-48d9-bbae-6830a2331143-kube-api-access-jfzhr\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890890-tcgwk\" (UID: \"48d1b4b9-adf7-48d9-bbae-6830a2331143\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890890-tcgwk" Oct 31 13:30:14.668071 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.667999 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41505b3-033f-4911-b467-0b9d29530279-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890890-pnqpx\" (UID: \"b41505b3-033f-4911-b467-0b9d29530279\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890890-pnqpx" Oct 31 13:30:14.668209 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.668104 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48cgh\" (UniqueName: \"kubernetes.io/projected/ba53a96b-92e6-4e45-9582-be73c7bf47bf-kube-api-access-48cgh\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890890-2ptgp\" (UID: \"ba53a96b-92e6-4e45-9582-be73c7bf47bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890890-2ptgp" Oct 31 13:30:14.668209 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.668163 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba53a96b-92e6-4e45-9582-be73c7bf47bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890890-2ptgp\" (UID: \"ba53a96b-92e6-4e45-9582-be73c7bf47bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890890-2ptgp" Oct 31 13:30:14.668373 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.668217 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5895r\" (UniqueName: \"kubernetes.io/projected/1055f537-d4e4-46fc-b684-1ec68a9bda17-kube-api-access-5895r\") pod \"maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890890-m2s7m\" (UID: \"1055f537-d4e4-46fc-b684-1ec68a9bda17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890890-m2s7m" Oct 31 13:30:14.668373 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.668289 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm7fr\" (UniqueName: \"kubernetes.io/projected/b41505b3-033f-4911-b467-0b9d29530279-kube-api-access-zm7fr\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890890-pnqpx\" (UID: \"b41505b3-033f-4911-b467-0b9d29530279\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890890-pnqpx" Oct 31 13:30:14.668373 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.668317 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48d1b4b9-adf7-48d9-bbae-6830a2331143-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890890-tcgwk\" (UID: \"48d1b4b9-adf7-48d9-bbae-6830a2331143\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890890-tcgwk" Oct 31 13:30:14.668373 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.668349 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pkts\" (UniqueName: \"kubernetes.io/projected/e1369557-2959-4d29-8756-bf9ddbd4b610-kube-api-access-7pkts\") pod \"maas-api-key-cleanup-29890890-bh5qm\" (UID: \"e1369557-2959-4d29-8756-bf9ddbd4b610\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-bh5qm" Oct 31 13:30:14.668619 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.668593 5693 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48fa6091-7341-4168-83da-93f772457206-routing-overlay-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:14.671150 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.671122 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48fa6091-7341-4168-83da-93f772457206-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:14.671276 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.671152 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkrlg\" (UniqueName: \"kubernetes.io/projected/48fa6091-7341-4168-83da-93f772457206-kube-api-access-kkrlg\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:14.671276 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.671168 5693 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/48fa6091-7341-4168-83da-93f772457206-provider-credentials\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:14.671276 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.669770 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48d1b4b9-adf7-48d9-bbae-6830a2331143-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890890-tcgwk\" (UID: \"48d1b4b9-adf7-48d9-bbae-6830a2331143\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890890-tcgwk" Oct 31 13:30:14.708789 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.705931 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfzhr\" (UniqueName: \"kubernetes.io/projected/48d1b4b9-adf7-48d9-bbae-6830a2331143-kube-api-access-jfzhr\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890890-tcgwk\" (UID: \"48d1b4b9-adf7-48d9-bbae-6830a2331143\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890890-tcgwk" Oct 31 13:30:14.773852 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.773821 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1055f537-d4e4-46fc-b684-1ec68a9bda17-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890890-m2s7m\" (UID: \"1055f537-d4e4-46fc-b684-1ec68a9bda17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890890-m2s7m" Oct 31 13:30:14.774144 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.774128 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1369557-2959-4d29-8756-bf9ddbd4b610-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890890-bh5qm\" (UID: \"e1369557-2959-4d29-8756-bf9ddbd4b610\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-bh5qm" Oct 31 13:30:14.774416 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.774382 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41505b3-033f-4911-b467-0b9d29530279-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890890-pnqpx\" (UID: \"b41505b3-033f-4911-b467-0b9d29530279\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890890-pnqpx" Oct 31 13:30:14.774595 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.774583 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48cgh\" (UniqueName: \"kubernetes.io/projected/ba53a96b-92e6-4e45-9582-be73c7bf47bf-kube-api-access-48cgh\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890890-2ptgp\" (UID: \"ba53a96b-92e6-4e45-9582-be73c7bf47bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890890-2ptgp" Oct 31 13:30:14.774749 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.774738 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba53a96b-92e6-4e45-9582-be73c7bf47bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890890-2ptgp\" (UID: \"ba53a96b-92e6-4e45-9582-be73c7bf47bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890890-2ptgp" Oct 31 13:30:14.774900 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.774873 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5895r\" (UniqueName: \"kubernetes.io/projected/1055f537-d4e4-46fc-b684-1ec68a9bda17-kube-api-access-5895r\") pod \"maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890890-m2s7m\" (UID: \"1055f537-d4e4-46fc-b684-1ec68a9bda17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890890-m2s7m" Oct 31 13:30:14.775064 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.775053 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm7fr\" (UniqueName: \"kubernetes.io/projected/b41505b3-033f-4911-b467-0b9d29530279-kube-api-access-zm7fr\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890890-pnqpx\" (UID: \"b41505b3-033f-4911-b467-0b9d29530279\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890890-pnqpx" Oct 31 13:30:14.775174 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.775163 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pkts\" (UniqueName: \"kubernetes.io/projected/e1369557-2959-4d29-8756-bf9ddbd4b610-kube-api-access-7pkts\") pod \"maas-api-key-cleanup-29890890-bh5qm\" (UID: \"e1369557-2959-4d29-8756-bf9ddbd4b610\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-bh5qm" Oct 31 13:30:14.775562 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.774916 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1055f537-d4e4-46fc-b684-1ec68a9bda17-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890890-m2s7m\" (UID: \"1055f537-d4e4-46fc-b684-1ec68a9bda17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890890-m2s7m" Oct 31 13:30:14.776633 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.776221 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba53a96b-92e6-4e45-9582-be73c7bf47bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890890-2ptgp\" (UID: \"ba53a96b-92e6-4e45-9582-be73c7bf47bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890890-2ptgp" Oct 31 13:30:14.778438 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.778389 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1369557-2959-4d29-8756-bf9ddbd4b610-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890890-bh5qm\" (UID: \"e1369557-2959-4d29-8756-bf9ddbd4b610\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-bh5qm" Oct 31 13:30:14.796432 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.787267 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41505b3-033f-4911-b467-0b9d29530279-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890890-pnqpx\" (UID: \"b41505b3-033f-4911-b467-0b9d29530279\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890890-pnqpx" Oct 31 13:30:14.796432 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.792506 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48cgh\" (UniqueName: \"kubernetes.io/projected/ba53a96b-92e6-4e45-9582-be73c7bf47bf-kube-api-access-48cgh\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890890-2ptgp\" (UID: \"ba53a96b-92e6-4e45-9582-be73c7bf47bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890890-2ptgp" Oct 31 13:30:14.814412 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.814379 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm7fr\" (UniqueName: \"kubernetes.io/projected/b41505b3-033f-4911-b467-0b9d29530279-kube-api-access-zm7fr\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890890-pnqpx\" (UID: \"b41505b3-033f-4911-b467-0b9d29530279\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890890-pnqpx" Oct 31 13:30:14.816715 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.816616 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5895r\" (UniqueName: \"kubernetes.io/projected/1055f537-d4e4-46fc-b684-1ec68a9bda17-kube-api-access-5895r\") pod \"maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890890-m2s7m\" (UID: \"1055f537-d4e4-46fc-b684-1ec68a9bda17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890890-m2s7m" Oct 31 13:30:14.817491 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.817436 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pkts\" (UniqueName: \"kubernetes.io/projected/e1369557-2959-4d29-8756-bf9ddbd4b610-kube-api-access-7pkts\") pod \"maas-api-key-cleanup-29890890-bh5qm\" (UID: \"e1369557-2959-4d29-8756-bf9ddbd4b610\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-bh5qm" Oct 31 13:30:14.851075 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.850989 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890890-tcgwk" Oct 31 13:30:14.945771 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.945737 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890890-m2s7m" Oct 31 13:30:14.981710 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:14.980946 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890890-pnqpx" Oct 31 13:30:15.021544 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:15.021283 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890890-2ptgp" Oct 31 13:30:15.031054 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:15.028959 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-bh5qm" Oct 31 13:30:15.175137 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:15.150822 5693 generic.go:356] "Generic (PLEG): container finished" podID="48fa6091-7341-4168-83da-93f772457206" containerID="f0272db704c281abdbbecbfc522e345bc2a5cf1d7ef9dbeddc80c93fce3b31fd" exitCode=0 Oct 31 13:30:15.175137 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:15.152099 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7" Oct 31 13:30:15.175137 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:15.169103 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7" event={"ID":"48fa6091-7341-4168-83da-93f772457206","Type":"ContainerDied","Data":"f0272db704c281abdbbecbfc522e345bc2a5cf1d7ef9dbeddc80c93fce3b31fd"} Oct 31 13:30:15.175137 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:15.169286 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7" event={"ID":"48fa6091-7341-4168-83da-93f772457206","Type":"ContainerDied","Data":"e208b228577d06bfbd23b9ee863ce34aaad831ba592b1c025caaa7f88d68c00d"} Oct 31 13:30:15.175137 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:15.169347 5693 scope.go:122] "RemoveContainer" containerID="f0272db704c281abdbbecbfc522e345bc2a5cf1d7ef9dbeddc80c93fce3b31fd" Oct 31 13:30:15.248864 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:15.248555 5693 scope.go:122] "RemoveContainer" containerID="f0272db704c281abdbbecbfc522e345bc2a5cf1d7ef9dbeddc80c93fce3b31fd" Oct 31 13:30:15.248953 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:30:15.248924 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0272db704c281abdbbecbfc522e345bc2a5cf1d7ef9dbeddc80c93fce3b31fd\": container with ID starting with f0272db704c281abdbbecbfc522e345bc2a5cf1d7ef9dbeddc80c93fce3b31fd not found: ID does not exist" containerID="f0272db704c281abdbbecbfc522e345bc2a5cf1d7ef9dbeddc80c93fce3b31fd" Oct 31 13:30:15.249015 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:15.248955 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0272db704c281abdbbecbfc522e345bc2a5cf1d7ef9dbeddc80c93fce3b31fd"} err="failed to get container status \"f0272db704c281abdbbecbfc522e345bc2a5cf1d7ef9dbeddc80c93fce3b31fd\": rpc error: code = NotFound desc = could not find container \"f0272db704c281abdbbecbfc522e345bc2a5cf1d7ef9dbeddc80c93fce3b31fd\": container with ID starting with f0272db704c281abdbbecbfc522e345bc2a5cf1d7ef9dbeddc80c93fce3b31fd not found: ID does not exist" Oct 31 13:30:15.262298 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:15.262241 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7"] Oct 31 13:30:15.262848 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:15.262825 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-784df9tf7"] Oct 31 13:30:15.417573 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:15.417535 5693 scope.go:122] "RemoveContainer" containerID="afae573979bf4a89d3ccac54d689cfea17ac3f2b286ca75c879c1d4e17a317b7" Oct 31 13:30:15.436691 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:15.436358 5693 scope.go:122] "RemoveContainer" containerID="191b9f550933f001d9616a0d32d50512bb12b0dbe7d4ca5852fc366026d94994" Oct 31 13:30:15.457417 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:30:15.457380 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48d1b4b9_adf7_48d9_bbae_6830a2331143.slice/crio-8c3325f4b7c810a3171c226ff1fd5b457db44e5fdd0b0b6cf750053219c3c992 WatchSource:0}: Error finding container 8c3325f4b7c810a3171c226ff1fd5b457db44e5fdd0b0b6cf750053219c3c992: Status 404 returned error can't find the container with id 8c3325f4b7c810a3171c226ff1fd5b457db44e5fdd0b0b6cf750053219c3c992 Oct 31 13:30:15.458895 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:15.458793 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48fa6091-7341-4168-83da-93f772457206" path="/var/lib/kubelet/pods/48fa6091-7341-4168-83da-93f772457206/volumes" Oct 31 13:30:15.469829 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:15.464337 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890890-tcgwk"] Oct 31 13:30:15.656533 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:15.656503 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890890-m2s7m"] Oct 31 13:30:15.716546 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:30:15.716447 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1055f537_d4e4_46fc_b684_1ec68a9bda17.slice/crio-4bcbec016a7e375399ca4e0f65afd6f80ef425241713baa9e137a4091595d5d8 WatchSource:0}: Error finding container 4bcbec016a7e375399ca4e0f65afd6f80ef425241713baa9e137a4091595d5d8: Status 404 returned error can't find the container with id 4bcbec016a7e375399ca4e0f65afd6f80ef425241713baa9e137a4091595d5d8 Oct 31 13:30:15.919999 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:15.919974 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890890-pnqpx"] Oct 31 13:30:15.926855 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:15.926825 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890890-2ptgp"] Oct 31 13:30:15.933098 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:15.933070 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890890-bh5qm"] Oct 31 13:30:16.165305 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:16.165274 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890890-2ptgp" event={"ID":"ba53a96b-92e6-4e45-9582-be73c7bf47bf","Type":"ContainerStarted","Data":"e602ae88ce44fd08f00089ec25d442c33aac39921d7b3900cc3ce5d63982c6e3"} Oct 31 13:30:16.166778 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:16.166750 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890890-m2s7m" event={"ID":"1055f537-d4e4-46fc-b684-1ec68a9bda17","Type":"ContainerStarted","Data":"4bcbec016a7e375399ca4e0f65afd6f80ef425241713baa9e137a4091595d5d8"} Oct 31 13:30:16.168968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:16.168946 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890890-rlpk4" event={"ID":"875da43d-e0a5-4fba-bd88-7758e8581e04","Type":"ContainerStarted","Data":"1b1835886302d7fa43ce0ed57980ce3324fb2afe86538e3648817eab5a14f562"} Oct 31 13:30:16.170744 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:16.170685 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890890-pnqpx" event={"ID":"b41505b3-033f-4911-b467-0b9d29530279","Type":"ContainerStarted","Data":"28baa8e42b80b33cea5a5db08abf49db0d9bdb3a25caecd9c434513408a2e459"} Oct 31 13:30:16.172057 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:16.172036 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890890-tcgwk" event={"ID":"48d1b4b9-adf7-48d9-bbae-6830a2331143","Type":"ContainerStarted","Data":"8c3325f4b7c810a3171c226ff1fd5b457db44e5fdd0b0b6cf750053219c3c992"} Oct 31 13:30:16.173374 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:16.173349 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-bh5qm" event={"ID":"e1369557-2959-4d29-8756-bf9ddbd4b610","Type":"ContainerStarted","Data":"5417cc7fcbf22e8e2b2ed8d3fbf0f2c7c66ef2e4f6cf564f981e6cc7771284b2"} Oct 31 13:30:16.405914 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:16.405230 5693 scope.go:122] "RemoveContainer" containerID="f4c7646be5184f7bbcf808d063a3d9c64ac6d3ab053d7a2fab55e74091c89d40" Oct 31 13:30:17.245809 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.245678 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890890-rlpk4"] Oct 31 13:30:17.250057 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.250022 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890890-2ptgp" event={"ID":"ba53a96b-92e6-4e45-9582-be73c7bf47bf","Type":"ContainerStarted","Data":"1e7054d3768d63b609c0e14848126b23bc8aebfd27e9a0514c47665ec78404e7"} Oct 31 13:30:17.251340 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.251186 5693 generic.go:356] "Generic (PLEG): container finished" podID="1055f537-d4e4-46fc-b684-1ec68a9bda17" containerID="8a4b55ecc14279bf920d36ef057799fd0fef7c3c263339d40a2d416f363ed1f6" exitCode=60 Oct 31 13:30:17.251340 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.251263 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890890-m2s7m" event={"ID":"1055f537-d4e4-46fc-b684-1ec68a9bda17","Type":"ContainerDied","Data":"8a4b55ecc14279bf920d36ef057799fd0fef7c3c263339d40a2d416f363ed1f6"} Oct 31 13:30:17.251595 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.251577 5693 scope.go:122] "RemoveContainer" containerID="8a4b55ecc14279bf920d36ef057799fd0fef7c3c263339d40a2d416f363ed1f6" Oct 31 13:30:17.253282 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.253259 5693 generic.go:356] "Generic (PLEG): container finished" podID="22242dc7-c483-4f14-8e16-5f5e5f1a7ad4" containerID="6dcbac897f19d368515991222b700aca930c7777a1bd0c79d703a8bcfd62c7b3" exitCode=60 Oct 31 13:30:17.253408 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.253328 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890890-2k4sn" event={"ID":"22242dc7-c483-4f14-8e16-5f5e5f1a7ad4","Type":"ContainerDied","Data":"6dcbac897f19d368515991222b700aca930c7777a1bd0c79d703a8bcfd62c7b3"} Oct 31 13:30:17.253472 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.253457 5693 scope.go:122] "RemoveContainer" containerID="191b9f550933f001d9616a0d32d50512bb12b0dbe7d4ca5852fc366026d94994" Oct 31 13:30:17.253865 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.253817 5693 scope.go:122] "RemoveContainer" containerID="6dcbac897f19d368515991222b700aca930c7777a1bd0c79d703a8bcfd62c7b3" Oct 31 13:30:17.256072 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:30:17.254123 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890890-2k4sn_odh-ai-gateway-infra(22242dc7-c483-4f14-8e16-5f5e5f1a7ad4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890890-2k4sn" podUID="22242dc7-c483-4f14-8e16-5f5e5f1a7ad4" Oct 31 13:30:17.257853 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.257832 5693 generic.go:356] "Generic (PLEG): container finished" podID="875da43d-e0a5-4fba-bd88-7758e8581e04" containerID="1b1835886302d7fa43ce0ed57980ce3324fb2afe86538e3648817eab5a14f562" exitCode=60 Oct 31 13:30:17.257946 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.257930 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890890-rlpk4" event={"ID":"875da43d-e0a5-4fba-bd88-7758e8581e04","Type":"ContainerDied","Data":"1b1835886302d7fa43ce0ed57980ce3324fb2afe86538e3648817eab5a14f562"} Oct 31 13:30:17.260630 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.260601 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890890-c6w7z" event={"ID":"e486c623-3983-406f-9f04-21a0d994c828","Type":"ContainerStarted","Data":"04b3295c80606292040b6afdc4f914b572803aa4e22a80e7b984f678fc9832da"} Oct 31 13:30:17.263322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.263298 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890890-pnqpx" event={"ID":"b41505b3-033f-4911-b467-0b9d29530279","Type":"ContainerStarted","Data":"22210c8daaeda4ff41fd8f204ce87c2226471f952c267dcc609cba58af58d0b7"} Oct 31 13:30:17.265588 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.265562 5693 generic.go:356] "Generic (PLEG): container finished" podID="48d1b4b9-adf7-48d9-bbae-6830a2331143" containerID="2c2d6518ed397b72a8679a18687cee7059723e6e4b5619c62037f48dae8fe409" exitCode=60 Oct 31 13:30:17.265747 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.265728 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890890-tcgwk" event={"ID":"48d1b4b9-adf7-48d9-bbae-6830a2331143","Type":"ContainerDied","Data":"2c2d6518ed397b72a8679a18687cee7059723e6e4b5619c62037f48dae8fe409"} Oct 31 13:30:17.267458 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.266868 5693 scope.go:122] "RemoveContainer" containerID="2c2d6518ed397b72a8679a18687cee7059723e6e4b5619c62037f48dae8fe409" Oct 31 13:30:17.268917 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.268891 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-bh5qm" event={"ID":"e1369557-2959-4d29-8756-bf9ddbd4b610","Type":"ContainerStarted","Data":"9e73b8312a30057dd48a47a2ff15ed6f847aeb6f840bc1c15c4bcf335af29196"} Oct 31 13:30:17.311556 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.309039 5693 scope.go:122] "RemoveContainer" containerID="afae573979bf4a89d3ccac54d689cfea17ac3f2b286ca75c879c1d4e17a317b7" Oct 31 13:30:17.311556 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.310756 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890890-2ptgp" podStartSLOduration=17.310737093 podStartE2EDuration="17.310737093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:17.309888575 +0000 UTC m=+3468.017474883" watchObservedRunningTime="2026-10-31 13:30:17.310737093 +0000 UTC m=+3468.018323388" Oct 31 13:30:17.409680 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.407176 5693 scope.go:122] "RemoveContainer" containerID="9443c45589e49834db218d8caaf55691a6780c3bfcbc18191a98cdc8657d3a0c" Oct 31 13:30:17.409680 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.408154 5693 scope.go:122] "RemoveContainer" containerID="9b4f853f465d43f3cc96ea41ccb538d195c5c3c8f152e48d73922608673dd1be" Oct 31 13:30:17.428681 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.422041 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890890-pnqpx" podStartSLOduration=17.422022026 podStartE2EDuration="17.422022026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:17.413652357 +0000 UTC m=+3468.121238648" watchObservedRunningTime="2026-10-31 13:30:17.422022026 +0000 UTC m=+3468.129608321" Oct 31 13:30:17.659830 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.659784 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-bh5qm" podStartSLOduration=17.65976296 podStartE2EDuration="17.65976296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:17.655404869 +0000 UTC m=+3468.362991165" watchObservedRunningTime="2026-10-31 13:30:17.65976296 +0000 UTC m=+3468.367349301" Oct 31 13:30:17.716834 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.716805 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890890-rlpk4" Oct 31 13:30:17.828479 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.826082 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890890-bbhs8"] Oct 31 13:30:17.828479 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.827762 5693 cpu_manager.go:414] "containerMap: removing container" podUID="875da43d-e0a5-4fba-bd88-7758e8581e04" containerName="cleanup" Oct 31 13:30:17.828479 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.827789 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="875da43d-e0a5-4fba-bd88-7758e8581e04" containerName="cleanup" Oct 31 13:30:17.828479 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.827813 5693 cpu_manager.go:414] "containerMap: removing container" podUID="875da43d-e0a5-4fba-bd88-7758e8581e04" containerName="cleanup" Oct 31 13:30:17.828479 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.827821 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="875da43d-e0a5-4fba-bd88-7758e8581e04" containerName="cleanup" Oct 31 13:30:17.830595 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.829213 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="875da43d-e0a5-4fba-bd88-7758e8581e04" containerName="cleanup" Oct 31 13:30:17.830595 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.829237 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="875da43d-e0a5-4fba-bd88-7758e8581e04" containerName="cleanup" Oct 31 13:30:17.834726 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.834704 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890890-bbhs8"] Oct 31 13:30:17.834862 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.834813 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890890-bbhs8" Oct 31 13:30:17.869107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.869071 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/875da43d-e0a5-4fba-bd88-7758e8581e04-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/875da43d-e0a5-4fba-bd88-7758e8581e04-maas-api-service-ca\") pod \"875da43d-e0a5-4fba-bd88-7758e8581e04\" (UID: \"875da43d-e0a5-4fba-bd88-7758e8581e04\") " Oct 31 13:30:17.869315 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.869283 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/875da43d-e0a5-4fba-bd88-7758e8581e04-kube-api-access-9tddl\" (UniqueName: \"kubernetes.io/projected/875da43d-e0a5-4fba-bd88-7758e8581e04-kube-api-access-9tddl\") pod \"875da43d-e0a5-4fba-bd88-7758e8581e04\" (UID: \"875da43d-e0a5-4fba-bd88-7758e8581e04\") " Oct 31 13:30:17.870698 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.869881 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxwww\" (UniqueName: \"kubernetes.io/projected/47872ba8-dd46-4d67-a513-e1081171e92e-kube-api-access-mxwww\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890890-bbhs8\" (UID: \"47872ba8-dd46-4d67-a513-e1081171e92e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890890-bbhs8" Oct 31 13:30:17.870698 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.870032 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/875da43d-e0a5-4fba-bd88-7758e8581e04-maas-api-service-ca" pod "875da43d-e0a5-4fba-bd88-7758e8581e04" (UID: "875da43d-e0a5-4fba-bd88-7758e8581e04"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:17.870698 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.870126 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47872ba8-dd46-4d67-a513-e1081171e92e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890890-bbhs8\" (UID: \"47872ba8-dd46-4d67-a513-e1081171e92e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890890-bbhs8" Oct 31 13:30:17.870698 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.870395 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/875da43d-e0a5-4fba-bd88-7758e8581e04-maas-api-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:17.873263 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.873236 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/875da43d-e0a5-4fba-bd88-7758e8581e04-kube-api-access-9tddl" pod "875da43d-e0a5-4fba-bd88-7758e8581e04" (UID: "875da43d-e0a5-4fba-bd88-7758e8581e04"). InnerVolumeSpecName "kube-api-access-9tddl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:17.971905 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.971872 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47872ba8-dd46-4d67-a513-e1081171e92e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890890-bbhs8\" (UID: \"47872ba8-dd46-4d67-a513-e1081171e92e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890890-bbhs8" Oct 31 13:30:17.972142 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.972123 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxwww\" (UniqueName: \"kubernetes.io/projected/47872ba8-dd46-4d67-a513-e1081171e92e-kube-api-access-mxwww\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890890-bbhs8\" (UID: \"47872ba8-dd46-4d67-a513-e1081171e92e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890890-bbhs8" Oct 31 13:30:17.972345 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.972328 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tddl\" (UniqueName: \"kubernetes.io/projected/875da43d-e0a5-4fba-bd88-7758e8581e04-kube-api-access-9tddl\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:17.972700 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:17.972673 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47872ba8-dd46-4d67-a513-e1081171e92e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890890-bbhs8\" (UID: \"47872ba8-dd46-4d67-a513-e1081171e92e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890890-bbhs8" Oct 31 13:30:18.076993 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:18.076902 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxwww\" (UniqueName: \"kubernetes.io/projected/47872ba8-dd46-4d67-a513-e1081171e92e-kube-api-access-mxwww\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890890-bbhs8\" (UID: \"47872ba8-dd46-4d67-a513-e1081171e92e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890890-bbhs8" Oct 31 13:30:18.211503 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:18.211467 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890890-bbhs8" Oct 31 13:30:18.341791 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:18.339472 5693 generic.go:356] "Generic (PLEG): container finished" podID="ba53a96b-92e6-4e45-9582-be73c7bf47bf" containerID="1e7054d3768d63b609c0e14848126b23bc8aebfd27e9a0514c47665ec78404e7" exitCode=60 Oct 31 13:30:18.341791 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:18.339590 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890890-2ptgp" event={"ID":"ba53a96b-92e6-4e45-9582-be73c7bf47bf","Type":"ContainerDied","Data":"1e7054d3768d63b609c0e14848126b23bc8aebfd27e9a0514c47665ec78404e7"} Oct 31 13:30:18.341791 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:18.340164 5693 scope.go:122] "RemoveContainer" containerID="1e7054d3768d63b609c0e14848126b23bc8aebfd27e9a0514c47665ec78404e7" Oct 31 13:30:18.377985 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:18.377955 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890890-m2s7m" event={"ID":"1055f537-d4e4-46fc-b684-1ec68a9bda17","Type":"ContainerStarted","Data":"2032fcf645d87d0ea12f55cb1530f303a7373d4adf9cae9f4ffba2a3266cd91d"} Oct 31 13:30:18.383932 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:18.383900 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890890-rlpk4" event={"ID":"875da43d-e0a5-4fba-bd88-7758e8581e04","Type":"ContainerDied","Data":"6797679d0bed8117a4b46bd50a44b9d064acbaf7234b8fcaf3f5a8de5e012c11"} Oct 31 13:30:18.384174 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:18.384141 5693 scope.go:122] "RemoveContainer" containerID="1b1835886302d7fa43ce0ed57980ce3324fb2afe86538e3648817eab5a14f562" Oct 31 13:30:18.384463 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:18.384445 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890890-rlpk4" Oct 31 13:30:18.432947 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:18.432507 5693 scope.go:122] "RemoveContainer" containerID="21db924a4f462ae6f9685d1b48924dd06629485c48e594b4605df966bfe17e68" Oct 31 13:30:18.471184 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:18.471139 5693 generic.go:356] "Generic (PLEG): container finished" podID="e486c623-3983-406f-9f04-21a0d994c828" containerID="04b3295c80606292040b6afdc4f914b572803aa4e22a80e7b984f678fc9832da" exitCode=60 Oct 31 13:30:18.488912 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:18.487692 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890890-c6w7z" event={"ID":"e486c623-3983-406f-9f04-21a0d994c828","Type":"ContainerDied","Data":"04b3295c80606292040b6afdc4f914b572803aa4e22a80e7b984f678fc9832da"} Oct 31 13:30:18.488912 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:18.488278 5693 scope.go:122] "RemoveContainer" containerID="04b3295c80606292040b6afdc4f914b572803aa4e22a80e7b984f678fc9832da" Oct 31 13:30:18.488912 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:30:18.488792 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890890-c6w7z_odh-ai-gateway-infra(e486c623-3983-406f-9f04-21a0d994c828)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890890-c6w7z" podUID="e486c623-3983-406f-9f04-21a0d994c828" Oct 31 13:30:18.502322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:18.496207 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890890-66f9d" event={"ID":"24816799-6034-48e6-9711-d9a342c0ef28","Type":"ContainerStarted","Data":"58c272e1a2c020d7b221ed0264824426f267be504e8357f63d996793c74db1e6"} Oct 31 13:30:18.502322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:18.500116 5693 generic.go:356] "Generic (PLEG): container finished" podID="b41505b3-033f-4911-b467-0b9d29530279" containerID="22210c8daaeda4ff41fd8f204ce87c2226471f952c267dcc609cba58af58d0b7" exitCode=60 Oct 31 13:30:18.502322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:18.500227 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890890-pnqpx" event={"ID":"b41505b3-033f-4911-b467-0b9d29530279","Type":"ContainerDied","Data":"22210c8daaeda4ff41fd8f204ce87c2226471f952c267dcc609cba58af58d0b7"} Oct 31 13:30:18.533405 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:18.530388 5693 scope.go:122] "RemoveContainer" containerID="22210c8daaeda4ff41fd8f204ce87c2226471f952c267dcc609cba58af58d0b7" Oct 31 13:30:18.555814 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:18.555029 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890890-tcgwk" event={"ID":"48d1b4b9-adf7-48d9-bbae-6830a2331143","Type":"ContainerStarted","Data":"5494f42e19e3bcf259e21cd2bb70f523cd8e2157c09a3abd8b0bb5b68899b531"} Oct 31 13:30:18.566026 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:18.565993 5693 generic.go:356] "Generic (PLEG): container finished" podID="e1369557-2959-4d29-8756-bf9ddbd4b610" containerID="9e73b8312a30057dd48a47a2ff15ed6f847aeb6f840bc1c15c4bcf335af29196" exitCode=60 Oct 31 13:30:18.566565 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:18.566005 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-bh5qm" event={"ID":"e1369557-2959-4d29-8756-bf9ddbd4b610","Type":"ContainerDied","Data":"9e73b8312a30057dd48a47a2ff15ed6f847aeb6f840bc1c15c4bcf335af29196"} Oct 31 13:30:18.568356 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:18.568335 5693 scope.go:122] "RemoveContainer" containerID="9e73b8312a30057dd48a47a2ff15ed6f847aeb6f840bc1c15c4bcf335af29196" Oct 31 13:30:18.623988 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:18.618763 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890890-c6w7z"] Oct 31 13:30:18.623988 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:18.618921 5693 scope.go:122] "RemoveContainer" containerID="f4c7646be5184f7bbcf808d063a3d9c64ac6d3ab053d7a2fab55e74091c89d40" Oct 31 13:30:18.634815 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:18.634756 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890890-m2s7m" podStartSLOduration=18.634736537 podStartE2EDuration="18.634736537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:18.43617424 +0000 UTC m=+3469.143760538" watchObservedRunningTime="2026-10-31 13:30:18.634736537 +0000 UTC m=+3469.342322835" Oct 31 13:30:18.648916 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:18.648883 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890890-2k4sn"] Oct 31 13:30:18.715889 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:18.715833 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890890-tcgwk" podStartSLOduration=18.715814796 podStartE2EDuration="18.715814796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:18.713011452 +0000 UTC m=+3469.420597748" watchObservedRunningTime="2026-10-31 13:30:18.715814796 +0000 UTC m=+3469.423401091" Oct 31 13:30:18.766139 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:30:18.766111 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:30:18.907887 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:18.907864 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890890-rlpk4"] Oct 31 13:30:18.917952 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:18.917923 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890890-rlpk4"] Oct 31 13:30:18.926498 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:18.926470 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890890-bbhs8"] Oct 31 13:30:19.155423 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.155396 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890890-2k4sn" Oct 31 13:30:19.191018 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.190990 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39960-29890890-n2j72"] Oct 31 13:30:19.192730 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.192628 5693 cpu_manager.go:414] "containerMap: removing container" podUID="22242dc7-c483-4f14-8e16-5f5e5f1a7ad4" containerName="cleanup" Oct 31 13:30:19.192915 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.192732 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22242dc7-c483-4f14-8e16-5f5e5f1a7ad4" containerName="cleanup" Oct 31 13:30:19.192915 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.192765 5693 cpu_manager.go:414] "containerMap: removing container" podUID="22242dc7-c483-4f14-8e16-5f5e5f1a7ad4" containerName="cleanup" Oct 31 13:30:19.192915 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.192773 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22242dc7-c483-4f14-8e16-5f5e5f1a7ad4" containerName="cleanup" Oct 31 13:30:19.192915 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.192794 5693 cpu_manager.go:414] "containerMap: removing container" podUID="875da43d-e0a5-4fba-bd88-7758e8581e04" containerName="cleanup" Oct 31 13:30:19.192915 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.192801 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="875da43d-e0a5-4fba-bd88-7758e8581e04" containerName="cleanup" Oct 31 13:30:19.193174 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.193149 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="22242dc7-c483-4f14-8e16-5f5e5f1a7ad4" containerName="cleanup" Oct 31 13:30:19.193174 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.193176 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="22242dc7-c483-4f14-8e16-5f5e5f1a7ad4" containerName="cleanup" Oct 31 13:30:19.193435 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.193189 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="875da43d-e0a5-4fba-bd88-7758e8581e04" containerName="cleanup" Oct 31 13:30:19.199340 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.197911 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39960-29890890-n2j72" Oct 31 13:30:19.207448 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.207420 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39960-29890890-n2j72"] Oct 31 13:30:19.270864 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.270831 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22242dc7-c483-4f14-8e16-5f5e5f1a7ad4-kube-api-access-phnq6\" (UniqueName: \"kubernetes.io/projected/22242dc7-c483-4f14-8e16-5f5e5f1a7ad4-kube-api-access-phnq6\") pod \"22242dc7-c483-4f14-8e16-5f5e5f1a7ad4\" (UID: \"22242dc7-c483-4f14-8e16-5f5e5f1a7ad4\") " Oct 31 13:30:19.271132 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.271113 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22242dc7-c483-4f14-8e16-5f5e5f1a7ad4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22242dc7-c483-4f14-8e16-5f5e5f1a7ad4-maas-api-service-ca\") pod \"22242dc7-c483-4f14-8e16-5f5e5f1a7ad4\" (UID: \"22242dc7-c483-4f14-8e16-5f5e5f1a7ad4\") " Oct 31 13:30:19.271461 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.271435 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22242dc7-c483-4f14-8e16-5f5e5f1a7ad4-maas-api-service-ca" pod "22242dc7-c483-4f14-8e16-5f5e5f1a7ad4" (UID: "22242dc7-c483-4f14-8e16-5f5e5f1a7ad4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:19.271862 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.271836 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ca294e6-ea27-4bc6-b710-60e6b27485ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a39960-29890890-n2j72\" (UID: \"5ca294e6-ea27-4bc6-b710-60e6b27485ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39960-29890890-n2j72" Oct 31 13:30:19.272335 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.272305 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr862\" (UniqueName: \"kubernetes.io/projected/5ca294e6-ea27-4bc6-b710-60e6b27485ee-kube-api-access-pr862\") pod \"maas-api-key-cleanup-e2e-inf-a-a39960-29890890-n2j72\" (UID: \"5ca294e6-ea27-4bc6-b710-60e6b27485ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39960-29890890-n2j72" Oct 31 13:30:19.272556 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.272538 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22242dc7-c483-4f14-8e16-5f5e5f1a7ad4-maas-api-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:19.278448 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.278418 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22242dc7-c483-4f14-8e16-5f5e5f1a7ad4-kube-api-access-phnq6" pod "22242dc7-c483-4f14-8e16-5f5e5f1a7ad4" (UID: "22242dc7-c483-4f14-8e16-5f5e5f1a7ad4"). InnerVolumeSpecName "kube-api-access-phnq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:19.374298 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.374267 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ca294e6-ea27-4bc6-b710-60e6b27485ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a39960-29890890-n2j72\" (UID: \"5ca294e6-ea27-4bc6-b710-60e6b27485ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39960-29890890-n2j72" Oct 31 13:30:19.374701 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.374539 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr862\" (UniqueName: \"kubernetes.io/projected/5ca294e6-ea27-4bc6-b710-60e6b27485ee-kube-api-access-pr862\") pod \"maas-api-key-cleanup-e2e-inf-a-a39960-29890890-n2j72\" (UID: \"5ca294e6-ea27-4bc6-b710-60e6b27485ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39960-29890890-n2j72" Oct 31 13:30:19.374748 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.374733 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phnq6\" (UniqueName: \"kubernetes.io/projected/22242dc7-c483-4f14-8e16-5f5e5f1a7ad4-kube-api-access-phnq6\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:19.374970 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.374953 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ca294e6-ea27-4bc6-b710-60e6b27485ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a39960-29890890-n2j72\" (UID: \"5ca294e6-ea27-4bc6-b710-60e6b27485ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39960-29890890-n2j72" Oct 31 13:30:19.390652 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.390607 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr862\" (UniqueName: \"kubernetes.io/projected/5ca294e6-ea27-4bc6-b710-60e6b27485ee-kube-api-access-pr862\") pod \"maas-api-key-cleanup-e2e-inf-a-a39960-29890890-n2j72\" (UID: \"5ca294e6-ea27-4bc6-b710-60e6b27485ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39960-29890890-n2j72" Oct 31 13:30:19.430045 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.430008 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="875da43d-e0a5-4fba-bd88-7758e8581e04" path="/var/lib/kubelet/pods/875da43d-e0a5-4fba-bd88-7758e8581e04/volumes" Oct 31 13:30:19.523187 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.523155 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39960-29890890-n2j72" Oct 31 13:30:19.590713 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.590673 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890890-2ptgp" event={"ID":"ba53a96b-92e6-4e45-9582-be73c7bf47bf","Type":"ContainerStarted","Data":"42f28bff87211e59fa63e8d7848a7111c930e782aa6931e81e95c07e48cafb57"} Oct 31 13:30:19.594195 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.594147 5693 generic.go:356] "Generic (PLEG): container finished" podID="1055f537-d4e4-46fc-b684-1ec68a9bda17" containerID="2032fcf645d87d0ea12f55cb1530f303a7373d4adf9cae9f4ffba2a3266cd91d" exitCode=60 Oct 31 13:30:19.594478 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.594222 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890890-m2s7m" event={"ID":"1055f537-d4e4-46fc-b684-1ec68a9bda17","Type":"ContainerDied","Data":"2032fcf645d87d0ea12f55cb1530f303a7373d4adf9cae9f4ffba2a3266cd91d"} Oct 31 13:30:19.594631 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.594533 5693 scope.go:122] "RemoveContainer" containerID="8a4b55ecc14279bf920d36ef057799fd0fef7c3c263339d40a2d416f363ed1f6" Oct 31 13:30:19.594631 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.594563 5693 scope.go:122] "RemoveContainer" containerID="2032fcf645d87d0ea12f55cb1530f303a7373d4adf9cae9f4ffba2a3266cd91d" Oct 31 13:30:19.598624 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:30:19.598597 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890890-m2s7m_odh-ai-gateway-infra(1055f537-d4e4-46fc-b684-1ec68a9bda17)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890890-m2s7m" podUID="1055f537-d4e4-46fc-b684-1ec68a9bda17" Oct 31 13:30:19.601285 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.601262 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890890-2k4sn" event={"ID":"22242dc7-c483-4f14-8e16-5f5e5f1a7ad4","Type":"ContainerDied","Data":"9d28cd3f69ac238d34f16c907124e7673a4dade4d682dfdc3d308ab0bda0fe22"} Oct 31 13:30:19.601389 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.601373 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890890-2k4sn" Oct 31 13:30:19.610606 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.609792 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890890-cnvdm" event={"ID":"c0fb78e4-7046-48f0-a737-4ca457074027","Type":"ContainerStarted","Data":"f5224ce9998b4a33c7eaaa3d339afe6a030d33db3440de0c06c00a1c6471e8d5"} Oct 31 13:30:19.613717 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.613539 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890890-bbhs8" event={"ID":"47872ba8-dd46-4d67-a513-e1081171e92e","Type":"ContainerStarted","Data":"f24fda5fc8f4a3cea5e077c7d3f3226b5aeb22d4d7ef64cf9aef764f50fc839a"} Oct 31 13:30:19.625875 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.623245 5693 scope.go:122] "RemoveContainer" containerID="6dcbac897f19d368515991222b700aca930c7777a1bd0c79d703a8bcfd62c7b3" Oct 31 13:30:19.626106 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.626085 5693 generic.go:356] "Generic (PLEG): container finished" podID="24816799-6034-48e6-9711-d9a342c0ef28" containerID="58c272e1a2c020d7b221ed0264824426f267be504e8357f63d996793c74db1e6" exitCode=60 Oct 31 13:30:19.626210 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.626150 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890890-66f9d" event={"ID":"24816799-6034-48e6-9711-d9a342c0ef28","Type":"ContainerDied","Data":"58c272e1a2c020d7b221ed0264824426f267be504e8357f63d996793c74db1e6"} Oct 31 13:30:19.627226 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.626995 5693 scope.go:122] "RemoveContainer" containerID="58c272e1a2c020d7b221ed0264824426f267be504e8357f63d996793c74db1e6" Oct 31 13:30:19.627825 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:30:19.627458 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-107955-29890890-66f9d_odh-ai-gateway-infra(24816799-6034-48e6-9711-d9a342c0ef28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890890-66f9d" podUID="24816799-6034-48e6-9711-d9a342c0ef28" Oct 31 13:30:19.629583 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.629196 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890890-pnqpx" event={"ID":"b41505b3-033f-4911-b467-0b9d29530279","Type":"ContainerStarted","Data":"52c295f72447153ed20fec75473d0d015be0d0d4657eac767c6833c6821e20de"} Oct 31 13:30:19.640261 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.640063 5693 generic.go:356] "Generic (PLEG): container finished" podID="48d1b4b9-adf7-48d9-bbae-6830a2331143" containerID="5494f42e19e3bcf259e21cd2bb70f523cd8e2157c09a3abd8b0bb5b68899b531" exitCode=60 Oct 31 13:30:19.640261 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.640141 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890890-tcgwk" event={"ID":"48d1b4b9-adf7-48d9-bbae-6830a2331143","Type":"ContainerDied","Data":"5494f42e19e3bcf259e21cd2bb70f523cd8e2157c09a3abd8b0bb5b68899b531"} Oct 31 13:30:19.642160 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.642139 5693 scope.go:122] "RemoveContainer" containerID="9443c45589e49834db218d8caaf55691a6780c3bfcbc18191a98cdc8657d3a0c" Oct 31 13:30:19.642271 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.642196 5693 scope.go:122] "RemoveContainer" containerID="5494f42e19e3bcf259e21cd2bb70f523cd8e2157c09a3abd8b0bb5b68899b531" Oct 31 13:30:19.642763 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:30:19.642713 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890890-tcgwk_odh-ai-gateway-infra(48d1b4b9-adf7-48d9-bbae-6830a2331143)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890890-tcgwk" podUID="48d1b4b9-adf7-48d9-bbae-6830a2331143" Oct 31 13:30:19.642945 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.642793 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-bh5qm" event={"ID":"e1369557-2959-4d29-8756-bf9ddbd4b610","Type":"ContainerStarted","Data":"6dfab6b8730763909d8b7e8ffb70e434218bb370a8fa37118504285caa587438"} Oct 31 13:30:19.650024 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.649941 5693 generic.go:356] "Generic (PLEG): container finished" podID="92712c0d-d49c-46fe-8bad-a023a997d5f2" containerID="dbed020cd01b9c70ca045073141e570300acfce17874e01c84815dc0759948bb" exitCode=60 Oct 31 13:30:19.650106 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.650016 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890890-5bssh" event={"ID":"92712c0d-d49c-46fe-8bad-a023a997d5f2","Type":"ContainerDied","Data":"dbed020cd01b9c70ca045073141e570300acfce17874e01c84815dc0759948bb"} Oct 31 13:30:19.650538 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.650518 5693 scope.go:122] "RemoveContainer" containerID="dbed020cd01b9c70ca045073141e570300acfce17874e01c84815dc0759948bb" Oct 31 13:30:19.652472 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:30:19.650983 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890890-5bssh_odh-ai-gateway-infra(92712c0d-d49c-46fe-8bad-a023a997d5f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890890-5bssh" podUID="92712c0d-d49c-46fe-8bad-a023a997d5f2" Oct 31 13:30:19.665286 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.665256 5693 scope.go:122] "RemoveContainer" containerID="2c2d6518ed397b72a8679a18687cee7059723e6e4b5619c62037f48dae8fe409" Oct 31 13:30:19.677183 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.677160 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890890-66f9d"] Oct 31 13:30:19.685224 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.685197 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890890-2k4sn"] Oct 31 13:30:19.693526 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.693497 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890890-2k4sn"] Oct 31 13:30:19.699337 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.699318 5693 scope.go:122] "RemoveContainer" containerID="9b4f853f465d43f3cc96ea41ccb538d195c5c3c8f152e48d73922608673dd1be" Oct 31 13:30:19.876832 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:19.876801 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39960-29890890-n2j72"] Oct 31 13:30:19.975993 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:30:19.975948 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ca294e6_ea27_4bc6_b710_60e6b27485ee.slice/crio-046fead63487eb1c78b51a961ebdf027e95217db62f76cb714bd98604b132566 WatchSource:0}: Error finding container 046fead63487eb1c78b51a961ebdf027e95217db62f76cb714bd98604b132566: Status 404 returned error can't find the container with id 046fead63487eb1c78b51a961ebdf027e95217db62f76cb714bd98604b132566 Oct 31 13:30:20.039270 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.039244 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890890-c6w7z" Oct 31 13:30:20.083865 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.081780 5693 patch_prober.go:29] interesting pod/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:30:20.083865 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.081835 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" podUID="efccc572-e9f9-4f4a-ab45-c7458b0e2e58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:20.101486 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.101444 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890890-cjwzd"] Oct 31 13:30:20.102556 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.102533 5693 cpu_manager.go:414] "containerMap: removing container" podUID="e486c623-3983-406f-9f04-21a0d994c828" containerName="cleanup" Oct 31 13:30:20.102556 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.102556 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e486c623-3983-406f-9f04-21a0d994c828" containerName="cleanup" Oct 31 13:30:20.102761 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.102690 5693 cpu_manager.go:414] "containerMap: removing container" podUID="22242dc7-c483-4f14-8e16-5f5e5f1a7ad4" containerName="cleanup" Oct 31 13:30:20.102761 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.102699 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22242dc7-c483-4f14-8e16-5f5e5f1a7ad4" containerName="cleanup" Oct 31 13:30:20.102761 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.102721 5693 cpu_manager.go:414] "containerMap: removing container" podUID="e486c623-3983-406f-9f04-21a0d994c828" containerName="cleanup" Oct 31 13:30:20.102761 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.102731 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e486c623-3983-406f-9f04-21a0d994c828" containerName="cleanup" Oct 31 13:30:20.103012 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.102969 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="e486c623-3983-406f-9f04-21a0d994c828" containerName="cleanup" Oct 31 13:30:20.103012 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.103006 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="22242dc7-c483-4f14-8e16-5f5e5f1a7ad4" containerName="cleanup" Oct 31 13:30:20.104667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.103021 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="e486c623-3983-406f-9f04-21a0d994c828" containerName="cleanup" Oct 31 13:30:20.107809 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.107788 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890890-cjwzd" Oct 31 13:30:20.120810 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.120779 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890890-cjwzd"] Oct 31 13:30:20.211922 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.211892 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e486c623-3983-406f-9f04-21a0d994c828-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e486c623-3983-406f-9f04-21a0d994c828-maas-api-service-ca\") pod \"e486c623-3983-406f-9f04-21a0d994c828\" (UID: \"e486c623-3983-406f-9f04-21a0d994c828\") " Oct 31 13:30:20.212090 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.211993 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e486c623-3983-406f-9f04-21a0d994c828-kube-api-access-z7hxg\" (UniqueName: \"kubernetes.io/projected/e486c623-3983-406f-9f04-21a0d994c828-kube-api-access-z7hxg\") pod \"e486c623-3983-406f-9f04-21a0d994c828\" (UID: \"e486c623-3983-406f-9f04-21a0d994c828\") " Oct 31 13:30:20.212564 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.212533 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e486c623-3983-406f-9f04-21a0d994c828-maas-api-service-ca" pod "e486c623-3983-406f-9f04-21a0d994c828" (UID: "e486c623-3983-406f-9f04-21a0d994c828"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:20.213040 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.213006 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwhqp\" (UniqueName: \"kubernetes.io/projected/c18e585d-a441-4ca7-b2b8-4342056c603f-kube-api-access-gwhqp\") pod \"maas-api-key-cleanup-e2e-extmod-a-5dc570-29890890-cjwzd\" (UID: \"c18e585d-a441-4ca7-b2b8-4342056c603f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890890-cjwzd" Oct 31 13:30:20.213193 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.213178 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c18e585d-a441-4ca7-b2b8-4342056c603f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-5dc570-29890890-cjwzd\" (UID: \"c18e585d-a441-4ca7-b2b8-4342056c603f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890890-cjwzd" Oct 31 13:30:20.213292 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.213260 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e486c623-3983-406f-9f04-21a0d994c828-maas-api-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:20.214932 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.214908 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e486c623-3983-406f-9f04-21a0d994c828-kube-api-access-z7hxg" pod "e486c623-3983-406f-9f04-21a0d994c828" (UID: "e486c623-3983-406f-9f04-21a0d994c828"). InnerVolumeSpecName "kube-api-access-z7hxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:20.315048 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.315015 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwhqp\" (UniqueName: \"kubernetes.io/projected/c18e585d-a441-4ca7-b2b8-4342056c603f-kube-api-access-gwhqp\") pod \"maas-api-key-cleanup-e2e-extmod-a-5dc570-29890890-cjwzd\" (UID: \"c18e585d-a441-4ca7-b2b8-4342056c603f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890890-cjwzd" Oct 31 13:30:20.315251 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.315141 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c18e585d-a441-4ca7-b2b8-4342056c603f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-5dc570-29890890-cjwzd\" (UID: \"c18e585d-a441-4ca7-b2b8-4342056c603f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890890-cjwzd" Oct 31 13:30:20.315385 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.315373 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7hxg\" (UniqueName: \"kubernetes.io/projected/e486c623-3983-406f-9f04-21a0d994c828-kube-api-access-z7hxg\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:20.315842 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.315820 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c18e585d-a441-4ca7-b2b8-4342056c603f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-5dc570-29890890-cjwzd\" (UID: \"c18e585d-a441-4ca7-b2b8-4342056c603f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890890-cjwzd" Oct 31 13:30:20.337082 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.337047 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwhqp\" (UniqueName: \"kubernetes.io/projected/c18e585d-a441-4ca7-b2b8-4342056c603f-kube-api-access-gwhqp\") pod \"maas-api-key-cleanup-e2e-extmod-a-5dc570-29890890-cjwzd\" (UID: \"c18e585d-a441-4ca7-b2b8-4342056c603f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890890-cjwzd" Oct 31 13:30:20.542169 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.542132 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890890-cjwzd" Oct 31 13:30:20.669776 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.669738 5693 generic.go:356] "Generic (PLEG): container finished" podID="ba53a96b-92e6-4e45-9582-be73c7bf47bf" containerID="42f28bff87211e59fa63e8d7848a7111c930e782aa6931e81e95c07e48cafb57" exitCode=60 Oct 31 13:30:20.670083 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.670058 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890890-2ptgp" event={"ID":"ba53a96b-92e6-4e45-9582-be73c7bf47bf","Type":"ContainerDied","Data":"42f28bff87211e59fa63e8d7848a7111c930e782aa6931e81e95c07e48cafb57"} Oct 31 13:30:20.670239 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.670127 5693 scope.go:122] "RemoveContainer" containerID="1e7054d3768d63b609c0e14848126b23bc8aebfd27e9a0514c47665ec78404e7" Oct 31 13:30:20.670370 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.670350 5693 scope.go:122] "RemoveContainer" containerID="42f28bff87211e59fa63e8d7848a7111c930e782aa6931e81e95c07e48cafb57" Oct 31 13:30:20.670891 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:30:20.670861 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890890-2ptgp_odh-ai-gateway-infra(ba53a96b-92e6-4e45-9582-be73c7bf47bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890890-2ptgp" podUID="ba53a96b-92e6-4e45-9582-be73c7bf47bf" Oct 31 13:30:20.674135 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.674102 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890890-cnvdm"] Oct 31 13:30:20.674448 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.674428 5693 scope.go:122] "RemoveContainer" containerID="2032fcf645d87d0ea12f55cb1530f303a7373d4adf9cae9f4ffba2a3266cd91d" Oct 31 13:30:20.674911 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:30:20.674884 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890890-m2s7m_odh-ai-gateway-infra(1055f537-d4e4-46fc-b684-1ec68a9bda17)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890890-m2s7m" podUID="1055f537-d4e4-46fc-b684-1ec68a9bda17" Oct 31 13:30:20.682672 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.681924 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39960-29890890-n2j72" event={"ID":"5ca294e6-ea27-4bc6-b710-60e6b27485ee","Type":"ContainerStarted","Data":"febcdb2bf418bc20bf940b4c4c801f09ba0b6109ccda3514b2be5f70b027480a"} Oct 31 13:30:20.682672 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.681980 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39960-29890890-n2j72" event={"ID":"5ca294e6-ea27-4bc6-b710-60e6b27485ee","Type":"ContainerStarted","Data":"046fead63487eb1c78b51a961ebdf027e95217db62f76cb714bd98604b132566"} Oct 31 13:30:20.685813 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.685501 5693 generic.go:356] "Generic (PLEG): container finished" podID="c0fb78e4-7046-48f0-a737-4ca457074027" containerID="f5224ce9998b4a33c7eaaa3d339afe6a030d33db3440de0c06c00a1c6471e8d5" exitCode=60 Oct 31 13:30:20.685813 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.685611 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890890-cnvdm" event={"ID":"c0fb78e4-7046-48f0-a737-4ca457074027","Type":"ContainerDied","Data":"f5224ce9998b4a33c7eaaa3d339afe6a030d33db3440de0c06c00a1c6471e8d5"} Oct 31 13:30:20.693809 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.693785 5693 scope.go:122] "RemoveContainer" containerID="21db924a4f462ae6f9685d1b48924dd06629485c48e594b4605df966bfe17e68" Oct 31 13:30:20.701480 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.700364 5693 generic.go:356] "Generic (PLEG): container finished" podID="47872ba8-dd46-4d67-a513-e1081171e92e" containerID="470b87afb15a020d1e7b808c47e4639ad511774aa75855ac20a18ef898fb7230" exitCode=60 Oct 31 13:30:20.701480 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.700448 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890890-bbhs8" event={"ID":"47872ba8-dd46-4d67-a513-e1081171e92e","Type":"ContainerDied","Data":"470b87afb15a020d1e7b808c47e4639ad511774aa75855ac20a18ef898fb7230"} Oct 31 13:30:20.701480 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.700905 5693 scope.go:122] "RemoveContainer" containerID="470b87afb15a020d1e7b808c47e4639ad511774aa75855ac20a18ef898fb7230" Oct 31 13:30:20.703357 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.703314 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890890-c6w7z" Oct 31 13:30:20.703510 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.703488 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890890-c6w7z" event={"ID":"e486c623-3983-406f-9f04-21a0d994c828","Type":"ContainerDied","Data":"88a02e1cbfa2e409d0b2e6c0f12beb778e92491c7abbe07173ad17ed520f80c6"} Oct 31 13:30:20.709069 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.708863 5693 generic.go:356] "Generic (PLEG): container finished" podID="b41505b3-033f-4911-b467-0b9d29530279" containerID="52c295f72447153ed20fec75473d0d015be0d0d4657eac767c6833c6821e20de" exitCode=60 Oct 31 13:30:20.709069 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.708937 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890890-pnqpx" event={"ID":"b41505b3-033f-4911-b467-0b9d29530279","Type":"ContainerDied","Data":"52c295f72447153ed20fec75473d0d015be0d0d4657eac767c6833c6821e20de"} Oct 31 13:30:20.709952 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.709840 5693 scope.go:122] "RemoveContainer" containerID="52c295f72447153ed20fec75473d0d015be0d0d4657eac767c6833c6821e20de" Oct 31 13:30:20.710725 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:30:20.710451 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890890-pnqpx_odh-ai-gateway-infra(b41505b3-033f-4911-b467-0b9d29530279)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890890-pnqpx" podUID="b41505b3-033f-4911-b467-0b9d29530279" Oct 31 13:30:20.728812 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.727929 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39960-29890890-n2j72" podStartSLOduration=20.727908585 podStartE2EDuration="20.727908585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:20.718004584 +0000 UTC m=+3471.425590888" watchObservedRunningTime="2026-10-31 13:30:20.727908585 +0000 UTC m=+3471.435494881" Oct 31 13:30:20.753612 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.753365 5693 scope.go:122] "RemoveContainer" containerID="04b3295c80606292040b6afdc4f914b572803aa4e22a80e7b984f678fc9832da" Oct 31 13:30:20.788085 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.786473 5693 scope.go:122] "RemoveContainer" containerID="5494f42e19e3bcf259e21cd2bb70f523cd8e2157c09a3abd8b0bb5b68899b531" Oct 31 13:30:20.788085 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:30:20.787004 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890890-tcgwk_odh-ai-gateway-infra(48d1b4b9-adf7-48d9-bbae-6830a2331143)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890890-tcgwk" podUID="48d1b4b9-adf7-48d9-bbae-6830a2331143" Oct 31 13:30:20.896666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.889705 5693 generic.go:356] "Generic (PLEG): container finished" podID="e1369557-2959-4d29-8756-bf9ddbd4b610" containerID="6dfab6b8730763909d8b7e8ffb70e434218bb370a8fa37118504285caa587438" exitCode=60 Oct 31 13:30:20.896666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.889880 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-bh5qm" event={"ID":"e1369557-2959-4d29-8756-bf9ddbd4b610","Type":"ContainerDied","Data":"6dfab6b8730763909d8b7e8ffb70e434218bb370a8fa37118504285caa587438"} Oct 31 13:30:20.896666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.890537 5693 scope.go:122] "RemoveContainer" containerID="6dfab6b8730763909d8b7e8ffb70e434218bb370a8fa37118504285caa587438" Oct 31 13:30:20.896666 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:30:20.891211 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890890-bh5qm_odh-ai-gateway-infra(e1369557-2959-4d29-8756-bf9ddbd4b610)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-bh5qm" podUID="e1369557-2959-4d29-8756-bf9ddbd4b610" Oct 31 13:30:20.899676 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.899396 5693 scope.go:122] "RemoveContainer" containerID="22210c8daaeda4ff41fd8f204ce87c2226471f952c267dcc609cba58af58d0b7" Oct 31 13:30:20.901503 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.901284 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890890-5bssh"] Oct 31 13:30:20.983862 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.983825 5693 scope.go:122] "RemoveContainer" containerID="9e73b8312a30057dd48a47a2ff15ed6f847aeb6f840bc1c15c4bcf335af29196" Oct 31 13:30:20.993970 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:20.993889 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890890-cjwzd"] Oct 31 13:30:21.010092 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.010039 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890890-c6w7z"] Oct 31 13:30:21.020717 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.019077 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890890-c6w7z"] Oct 31 13:30:21.024603 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.024569 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890890-cnvdm" Oct 31 13:30:21.172765 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.171286 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0fb78e4-7046-48f0-a737-4ca457074027-kube-api-access-9kcqv\" (UniqueName: \"kubernetes.io/projected/c0fb78e4-7046-48f0-a737-4ca457074027-kube-api-access-9kcqv\") pod \"c0fb78e4-7046-48f0-a737-4ca457074027\" (UID: \"c0fb78e4-7046-48f0-a737-4ca457074027\") " Oct 31 13:30:21.172765 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.171541 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0fb78e4-7046-48f0-a737-4ca457074027-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0fb78e4-7046-48f0-a737-4ca457074027-maas-api-service-ca\") pod \"c0fb78e4-7046-48f0-a737-4ca457074027\" (UID: \"c0fb78e4-7046-48f0-a737-4ca457074027\") " Oct 31 13:30:21.173687 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.173658 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0fb78e4-7046-48f0-a737-4ca457074027-maas-api-service-ca" pod "c0fb78e4-7046-48f0-a737-4ca457074027" (UID: "c0fb78e4-7046-48f0-a737-4ca457074027"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:21.197026 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.196995 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0fb78e4-7046-48f0-a737-4ca457074027-kube-api-access-9kcqv" pod "c0fb78e4-7046-48f0-a737-4ca457074027" (UID: "c0fb78e4-7046-48f0-a737-4ca457074027"). InnerVolumeSpecName "kube-api-access-9kcqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:21.274013 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.273976 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kcqv\" (UniqueName: \"kubernetes.io/projected/c0fb78e4-7046-48f0-a737-4ca457074027-kube-api-access-9kcqv\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:21.274013 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.274015 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0fb78e4-7046-48f0-a737-4ca457074027-maas-api-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:21.419230 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.419201 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22242dc7-c483-4f14-8e16-5f5e5f1a7ad4" path="/var/lib/kubelet/pods/22242dc7-c483-4f14-8e16-5f5e5f1a7ad4/volumes" Oct 31 13:30:21.419914 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.419891 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e486c623-3983-406f-9f04-21a0d994c828" path="/var/lib/kubelet/pods/e486c623-3983-406f-9f04-21a0d994c828/volumes" Oct 31 13:30:21.421556 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.421525 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890890-66f9d" Oct 31 13:30:21.427671 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.427619 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890890-5bssh" Oct 31 13:30:21.579895 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.579254 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24816799-6034-48e6-9711-d9a342c0ef28-kube-api-access-bvsd5\" (UniqueName: \"kubernetes.io/projected/24816799-6034-48e6-9711-d9a342c0ef28-kube-api-access-bvsd5\") pod \"24816799-6034-48e6-9711-d9a342c0ef28\" (UID: \"24816799-6034-48e6-9711-d9a342c0ef28\") " Oct 31 13:30:21.579895 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.579379 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24816799-6034-48e6-9711-d9a342c0ef28-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24816799-6034-48e6-9711-d9a342c0ef28-maas-api-service-ca\") pod \"24816799-6034-48e6-9711-d9a342c0ef28\" (UID: \"24816799-6034-48e6-9711-d9a342c0ef28\") " Oct 31 13:30:21.579895 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.579478 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92712c0d-d49c-46fe-8bad-a023a997d5f2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92712c0d-d49c-46fe-8bad-a023a997d5f2-maas-api-service-ca\") pod \"92712c0d-d49c-46fe-8bad-a023a997d5f2\" (UID: \"92712c0d-d49c-46fe-8bad-a023a997d5f2\") " Oct 31 13:30:21.579895 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.579832 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24816799-6034-48e6-9711-d9a342c0ef28-maas-api-service-ca" pod "24816799-6034-48e6-9711-d9a342c0ef28" (UID: "24816799-6034-48e6-9711-d9a342c0ef28"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:21.579895 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.579880 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92712c0d-d49c-46fe-8bad-a023a997d5f2-kube-api-access-98ldg\" (UniqueName: \"kubernetes.io/projected/92712c0d-d49c-46fe-8bad-a023a997d5f2-kube-api-access-98ldg\") pod \"92712c0d-d49c-46fe-8bad-a023a997d5f2\" (UID: \"92712c0d-d49c-46fe-8bad-a023a997d5f2\") " Oct 31 13:30:21.579895 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.579888 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92712c0d-d49c-46fe-8bad-a023a997d5f2-maas-api-service-ca" pod "92712c0d-d49c-46fe-8bad-a023a997d5f2" (UID: "92712c0d-d49c-46fe-8bad-a023a997d5f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:21.580902 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.580766 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24816799-6034-48e6-9711-d9a342c0ef28-maas-api-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:21.580902 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.580789 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92712c0d-d49c-46fe-8bad-a023a997d5f2-maas-api-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:21.582294 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.582255 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24816799-6034-48e6-9711-d9a342c0ef28-kube-api-access-bvsd5" pod "24816799-6034-48e6-9711-d9a342c0ef28" (UID: "24816799-6034-48e6-9711-d9a342c0ef28"). InnerVolumeSpecName "kube-api-access-bvsd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:21.582977 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.582942 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92712c0d-d49c-46fe-8bad-a023a997d5f2-kube-api-access-98ldg" pod "92712c0d-d49c-46fe-8bad-a023a997d5f2" (UID: "92712c0d-d49c-46fe-8bad-a023a997d5f2"). InnerVolumeSpecName "kube-api-access-98ldg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:21.683135 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.683092 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98ldg\" (UniqueName: \"kubernetes.io/projected/92712c0d-d49c-46fe-8bad-a023a997d5f2-kube-api-access-98ldg\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:21.683135 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.683129 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvsd5\" (UniqueName: \"kubernetes.io/projected/24816799-6034-48e6-9711-d9a342c0ef28-kube-api-access-bvsd5\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:21.921731 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.921499 5693 generic.go:356] "Generic (PLEG): container finished" podID="5ca294e6-ea27-4bc6-b710-60e6b27485ee" containerID="febcdb2bf418bc20bf940b4c4c801f09ba0b6109ccda3514b2be5f70b027480a" exitCode=60 Oct 31 13:30:21.921731 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.921602 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39960-29890890-n2j72" event={"ID":"5ca294e6-ea27-4bc6-b710-60e6b27485ee","Type":"ContainerDied","Data":"febcdb2bf418bc20bf940b4c4c801f09ba0b6109ccda3514b2be5f70b027480a"} Oct 31 13:30:21.929780 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.922244 5693 scope.go:122] "RemoveContainer" containerID="febcdb2bf418bc20bf940b4c4c801f09ba0b6109ccda3514b2be5f70b027480a" Oct 31 13:30:21.929780 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.929054 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890890-cnvdm" event={"ID":"c0fb78e4-7046-48f0-a737-4ca457074027","Type":"ContainerDied","Data":"bcd5ac5eb7829717059c16287b9b0503ead1ef52dea79c4dd0491391e15de109"} Oct 31 13:30:21.929780 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.929117 5693 scope.go:122] "RemoveContainer" containerID="f5224ce9998b4a33c7eaaa3d339afe6a030d33db3440de0c06c00a1c6471e8d5" Oct 31 13:30:21.929780 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.929236 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890890-cnvdm" Oct 31 13:30:21.937273 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.936533 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890890-bbhs8" event={"ID":"47872ba8-dd46-4d67-a513-e1081171e92e","Type":"ContainerStarted","Data":"0d20fc9514a52413ebf4fad2a8f8615137a6c6a1bd28658317c15db2438b00c4"} Oct 31 13:30:21.940540 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.940284 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890890-66f9d" Oct 31 13:30:21.940540 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.940286 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890890-66f9d" event={"ID":"24816799-6034-48e6-9711-d9a342c0ef28","Type":"ContainerDied","Data":"d2a6e0f938056fdbd5ab208cf58782cac36e9413823fe3b881df0cedf18445a7"} Oct 31 13:30:21.947389 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.947364 5693 scope.go:122] "RemoveContainer" containerID="52c295f72447153ed20fec75473d0d015be0d0d4657eac767c6833c6821e20de" Oct 31 13:30:21.948056 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.948035 5693 scope.go:122] "RemoveContainer" containerID="58c272e1a2c020d7b221ed0264824426f267be504e8357f63d996793c74db1e6" Oct 31 13:30:21.948274 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:30:21.948247 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890890-pnqpx_odh-ai-gateway-infra(b41505b3-033f-4911-b467-0b9d29530279)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890890-pnqpx" podUID="b41505b3-033f-4911-b467-0b9d29530279" Oct 31 13:30:21.951521 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.951506 5693 scope.go:122] "RemoveContainer" containerID="6dfab6b8730763909d8b7e8ffb70e434218bb370a8fa37118504285caa587438" Oct 31 13:30:21.951867 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:30:21.951846 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890890-bh5qm_odh-ai-gateway-infra(e1369557-2959-4d29-8756-bf9ddbd4b610)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-bh5qm" podUID="e1369557-2959-4d29-8756-bf9ddbd4b610" Oct 31 13:30:21.953830 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.953810 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890890-5bssh" Oct 31 13:30:21.953957 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.953861 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890890-5bssh" event={"ID":"92712c0d-d49c-46fe-8bad-a023a997d5f2","Type":"ContainerDied","Data":"7959cab03024ac7d1e508c21eebecdebe462b2ff3efcb0aa63551da9d2ad9a6b"} Oct 31 13:30:21.956755 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.956730 5693 scope.go:122] "RemoveContainer" containerID="42f28bff87211e59fa63e8d7848a7111c930e782aa6931e81e95c07e48cafb57" Oct 31 13:30:21.957143 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:30:21.957116 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890890-2ptgp_odh-ai-gateway-infra(ba53a96b-92e6-4e45-9582-be73c7bf47bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890890-2ptgp" podUID="ba53a96b-92e6-4e45-9582-be73c7bf47bf" Oct 31 13:30:21.961383 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.961359 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890890-cjwzd" event={"ID":"c18e585d-a441-4ca7-b2b8-4342056c603f","Type":"ContainerStarted","Data":"43f56608aff7c6489c92371f373996930e8598c59f7ffe249d6714df5d2607ac"} Oct 31 13:30:21.961501 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.961407 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890890-cjwzd" event={"ID":"c18e585d-a441-4ca7-b2b8-4342056c603f","Type":"ContainerStarted","Data":"d841ac784c7f3601b31814c205d0edc2c7ee13825f189de22770ab78a377c59e"} Oct 31 13:30:21.970780 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.970616 5693 scope.go:122] "RemoveContainer" containerID="dbed020cd01b9c70ca045073141e570300acfce17874e01c84815dc0759948bb" Oct 31 13:30:22.001179 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:21.992361 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890890-cnvdm"] Oct 31 13:30:22.001923 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:22.000471 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890890-cnvdm"] Oct 31 13:30:22.021630 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:22.021548 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890890-bbhs8" podStartSLOduration=22.021535194 podStartE2EDuration="22.021535194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:22.020414568 +0000 UTC m=+3472.728000862" watchObservedRunningTime="2026-10-31 13:30:22.021535194 +0000 UTC m=+3472.729121488" Oct 31 13:30:22.062175 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:22.062133 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890890-66f9d"] Oct 31 13:30:22.067474 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:22.067449 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890890-66f9d"] Oct 31 13:30:22.116006 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:22.115950 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890890-cjwzd" podStartSLOduration=22.115934093 podStartE2EDuration="22.115934093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:22.110293626 +0000 UTC m=+3472.817879929" watchObservedRunningTime="2026-10-31 13:30:22.115934093 +0000 UTC m=+3472.823520388" Oct 31 13:30:22.133180 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:22.133137 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890890-5bssh"] Oct 31 13:30:22.138134 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:22.138103 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890890-5bssh"] Oct 31 13:30:22.983947 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:22.983914 5693 generic.go:356] "Generic (PLEG): container finished" podID="c18e585d-a441-4ca7-b2b8-4342056c603f" containerID="43f56608aff7c6489c92371f373996930e8598c59f7ffe249d6714df5d2607ac" exitCode=60 Oct 31 13:30:22.984477 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:22.983955 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890890-cjwzd" event={"ID":"c18e585d-a441-4ca7-b2b8-4342056c603f","Type":"ContainerDied","Data":"43f56608aff7c6489c92371f373996930e8598c59f7ffe249d6714df5d2607ac"} Oct 31 13:30:22.984554 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:22.984476 5693 scope.go:122] "RemoveContainer" containerID="43f56608aff7c6489c92371f373996930e8598c59f7ffe249d6714df5d2607ac" Oct 31 13:30:22.991256 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:22.991221 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39960-29890890-n2j72" event={"ID":"5ca294e6-ea27-4bc6-b710-60e6b27485ee","Type":"ContainerStarted","Data":"c834966a14407e8075d1aabe91a45ba9eee98f14f4f37dcd12dc3c28f3861866"} Oct 31 13:30:22.994597 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:22.994567 5693 generic.go:356] "Generic (PLEG): container finished" podID="47872ba8-dd46-4d67-a513-e1081171e92e" containerID="0d20fc9514a52413ebf4fad2a8f8615137a6c6a1bd28658317c15db2438b00c4" exitCode=60 Oct 31 13:30:22.994799 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:22.994700 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890890-bbhs8" event={"ID":"47872ba8-dd46-4d67-a513-e1081171e92e","Type":"ContainerDied","Data":"0d20fc9514a52413ebf4fad2a8f8615137a6c6a1bd28658317c15db2438b00c4"} Oct 31 13:30:22.994799 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:22.994786 5693 scope.go:122] "RemoveContainer" containerID="470b87afb15a020d1e7b808c47e4639ad511774aa75855ac20a18ef898fb7230" Oct 31 13:30:22.995029 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:22.995006 5693 scope.go:122] "RemoveContainer" containerID="0d20fc9514a52413ebf4fad2a8f8615137a6c6a1bd28658317c15db2438b00c4" Oct 31 13:30:22.995726 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:30:22.995703 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890890-bbhs8_odh-ai-gateway-infra(47872ba8-dd46-4d67-a513-e1081171e92e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890890-bbhs8" podUID="47872ba8-dd46-4d67-a513-e1081171e92e" Oct 31 13:30:23.415672 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:23.415621 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24816799-6034-48e6-9711-d9a342c0ef28" path="/var/lib/kubelet/pods/24816799-6034-48e6-9711-d9a342c0ef28/volumes" Oct 31 13:30:23.416276 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:23.416259 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92712c0d-d49c-46fe-8bad-a023a997d5f2" path="/var/lib/kubelet/pods/92712c0d-d49c-46fe-8bad-a023a997d5f2/volumes" Oct 31 13:30:23.416731 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:23.416716 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0fb78e4-7046-48f0-a737-4ca457074027" path="/var/lib/kubelet/pods/c0fb78e4-7046-48f0-a737-4ca457074027/volumes" Oct 31 13:30:24.011389 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:24.011345 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890890-cjwzd" event={"ID":"c18e585d-a441-4ca7-b2b8-4342056c603f","Type":"ContainerStarted","Data":"5b28df3f900ae256440e7d83f386deed886bae12d45fa4cf8019bd599683ac3e"} Oct 31 13:30:24.014899 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:24.014866 5693 generic.go:356] "Generic (PLEG): container finished" podID="5ca294e6-ea27-4bc6-b710-60e6b27485ee" containerID="c834966a14407e8075d1aabe91a45ba9eee98f14f4f37dcd12dc3c28f3861866" exitCode=60 Oct 31 13:30:24.015348 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:24.014984 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39960-29890890-n2j72" event={"ID":"5ca294e6-ea27-4bc6-b710-60e6b27485ee","Type":"ContainerDied","Data":"c834966a14407e8075d1aabe91a45ba9eee98f14f4f37dcd12dc3c28f3861866"} Oct 31 13:30:24.015348 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:24.015134 5693 scope.go:122] "RemoveContainer" containerID="febcdb2bf418bc20bf940b4c4c801f09ba0b6109ccda3514b2be5f70b027480a" Oct 31 13:30:24.015348 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:24.015320 5693 scope.go:122] "RemoveContainer" containerID="c834966a14407e8075d1aabe91a45ba9eee98f14f4f37dcd12dc3c28f3861866" Oct 31 13:30:24.015818 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:30:24.015784 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a39960-29890890-n2j72_odh-ai-gateway-infra(5ca294e6-ea27-4bc6-b710-60e6b27485ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39960-29890890-n2j72" podUID="5ca294e6-ea27-4bc6-b710-60e6b27485ee" Oct 31 13:30:24.018832 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:24.018814 5693 scope.go:122] "RemoveContainer" containerID="0d20fc9514a52413ebf4fad2a8f8615137a6c6a1bd28658317c15db2438b00c4" Oct 31 13:30:24.019215 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:30:24.019196 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890890-bbhs8_odh-ai-gateway-infra(47872ba8-dd46-4d67-a513-e1081171e92e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890890-bbhs8" podUID="47872ba8-dd46-4d67-a513-e1081171e92e" Oct 31 13:30:25.038654 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:25.038544 5693 generic.go:356] "Generic (PLEG): container finished" podID="c18e585d-a441-4ca7-b2b8-4342056c603f" containerID="5b28df3f900ae256440e7d83f386deed886bae12d45fa4cf8019bd599683ac3e" exitCode=60 Oct 31 13:30:25.039202 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:25.038745 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890890-cjwzd" event={"ID":"c18e585d-a441-4ca7-b2b8-4342056c603f","Type":"ContainerDied","Data":"5b28df3f900ae256440e7d83f386deed886bae12d45fa4cf8019bd599683ac3e"} Oct 31 13:30:25.039202 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:25.038935 5693 scope.go:122] "RemoveContainer" containerID="43f56608aff7c6489c92371f373996930e8598c59f7ffe249d6714df5d2607ac" Oct 31 13:30:25.039534 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:25.039508 5693 scope.go:122] "RemoveContainer" containerID="5b28df3f900ae256440e7d83f386deed886bae12d45fa4cf8019bd599683ac3e" Oct 31 13:30:25.040552 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:30:25.040482 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-5dc570-29890890-cjwzd_odh-ai-gateway-infra(c18e585d-a441-4ca7-b2b8-4342056c603f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890890-cjwzd" podUID="c18e585d-a441-4ca7-b2b8-4342056c603f" Oct 31 13:30:25.042022 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:25.042002 5693 scope.go:122] "RemoveContainer" containerID="c834966a14407e8075d1aabe91a45ba9eee98f14f4f37dcd12dc3c28f3861866" Oct 31 13:30:25.042467 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:30:25.042429 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a39960-29890890-n2j72_odh-ai-gateway-infra(5ca294e6-ea27-4bc6-b710-60e6b27485ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39960-29890890-n2j72" podUID="5ca294e6-ea27-4bc6-b710-60e6b27485ee" Oct 31 13:30:25.044000 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:25.043972 5693 generic.go:356] "Generic (PLEG): container finished" podID="efccc572-e9f9-4f4a-ab45-c7458b0e2e58" containerID="15a949ec75dd1373b9bbb8b3ec5121bcd53fe94ec3e99f9369b78d8e66383412" exitCode=0 Oct 31 13:30:25.044111 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:25.044008 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" event={"ID":"efccc572-e9f9-4f4a-ab45-c7458b0e2e58","Type":"ContainerDied","Data":"15a949ec75dd1373b9bbb8b3ec5121bcd53fe94ec3e99f9369b78d8e66383412"} Oct 31 13:30:25.044414 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:25.044397 5693 scope.go:122] "RemoveContainer" containerID="15a949ec75dd1373b9bbb8b3ec5121bcd53fe94ec3e99f9369b78d8e66383412" Oct 31 13:30:25.047089 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:25.047050 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:30:25.047176 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:25.047143 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:30:25.047220 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:25.047190 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:30:26.055722 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:26.055686 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" event={"ID":"efccc572-e9f9-4f4a-ab45-c7458b0e2e58","Type":"ContainerStarted","Data":"53f8b1541faf3c232db0d1c3cb272695960884b719c385e5decd2325e2f290ae"} Oct 31 13:30:26.057945 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:26.057921 5693 scope.go:122] "RemoveContainer" containerID="5b28df3f900ae256440e7d83f386deed886bae12d45fa4cf8019bd599683ac3e" Oct 31 13:30:26.058279 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:30:26.058262 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-5dc570-29890890-cjwzd_odh-ai-gateway-infra(c18e585d-a441-4ca7-b2b8-4342056c603f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890890-cjwzd" podUID="c18e585d-a441-4ca7-b2b8-4342056c603f" Oct 31 13:30:26.519561 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:26.519528 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" Oct 31 13:30:26.519789 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:26.519718 5693 patch_prober.go:29] interesting pod/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Oct 31 13:30:26.519789 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:26.519761 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" podUID="efccc572-e9f9-4f4a-ab45-c7458b0e2e58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Oct 31 13:30:27.519489 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:27.519455 5693 patch_prober.go:29] interesting pod/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Oct 31 13:30:27.519942 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:27.519511 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" podUID="efccc572-e9f9-4f4a-ab45-c7458b0e2e58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Oct 31 13:30:28.521712 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:28.521655 5693 patch_prober.go:29] interesting pod/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Oct 31 13:30:28.522267 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:28.521749 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" podUID="efccc572-e9f9-4f4a-ab45-c7458b0e2e58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Oct 31 13:30:29.519805 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:29.519767 5693 patch_prober.go:29] interesting pod/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Oct 31 13:30:29.520003 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:29.519825 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" podUID="efccc572-e9f9-4f4a-ab45-c7458b0e2e58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Oct 31 13:30:30.401463 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:30.401429 5693 scope.go:122] "RemoveContainer" containerID="5494f42e19e3bcf259e21cd2bb70f523cd8e2157c09a3abd8b0bb5b68899b531" Oct 31 13:30:30.401918 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:30.401618 5693 scope.go:122] "RemoveContainer" containerID="42f28bff87211e59fa63e8d7848a7111c930e782aa6931e81e95c07e48cafb57" Oct 31 13:30:30.519566 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:30.519531 5693 patch_prober.go:29] interesting pod/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Oct 31 13:30:30.519728 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:30.519588 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" podUID="efccc572-e9f9-4f4a-ab45-c7458b0e2e58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Oct 31 13:30:31.194109 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:31.194072 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890890-2ptgp" event={"ID":"ba53a96b-92e6-4e45-9582-be73c7bf47bf","Type":"ContainerStarted","Data":"be8f6bdeb2bf6e7732fc481607ba98881051b810ec345476172ae915145e8120"} Oct 31 13:30:31.196557 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:31.196530 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890890-tcgwk" event={"ID":"48d1b4b9-adf7-48d9-bbae-6830a2331143","Type":"ContainerStarted","Data":"57bfeac365cadf99277364643373bc45d1db14c77a5a38d1417197e001f3354d"} Oct 31 13:30:31.524964 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:31.524937 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" Oct 31 13:30:31.525378 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:31.525344 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" Oct 31 13:30:31.526021 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:31.526001 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" Oct 31 13:30:32.258315 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:32.258283 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890890-2ptgp"] Oct 31 13:30:32.259722 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:32.259688 5693 generic.go:356] "Generic (PLEG): container finished" podID="ba53a96b-92e6-4e45-9582-be73c7bf47bf" containerID="be8f6bdeb2bf6e7732fc481607ba98881051b810ec345476172ae915145e8120" exitCode=60 Oct 31 13:30:32.259878 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:32.259779 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890890-2ptgp" event={"ID":"ba53a96b-92e6-4e45-9582-be73c7bf47bf","Type":"ContainerDied","Data":"be8f6bdeb2bf6e7732fc481607ba98881051b810ec345476172ae915145e8120"} Oct 31 13:30:32.259878 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:32.259872 5693 scope.go:122] "RemoveContainer" containerID="42f28bff87211e59fa63e8d7848a7111c930e782aa6931e81e95c07e48cafb57" Oct 31 13:30:32.264187 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:32.264108 5693 generic.go:356] "Generic (PLEG): container finished" podID="48d1b4b9-adf7-48d9-bbae-6830a2331143" containerID="57bfeac365cadf99277364643373bc45d1db14c77a5a38d1417197e001f3354d" exitCode=60 Oct 31 13:30:32.265582 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:32.265547 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890890-tcgwk" event={"ID":"48d1b4b9-adf7-48d9-bbae-6830a2331143","Type":"ContainerDied","Data":"57bfeac365cadf99277364643373bc45d1db14c77a5a38d1417197e001f3354d"} Oct 31 13:30:32.266604 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:32.266026 5693 scope.go:122] "RemoveContainer" containerID="57bfeac365cadf99277364643373bc45d1db14c77a5a38d1417197e001f3354d" Oct 31 13:30:32.266604 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:30:32.266416 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890890-tcgwk_odh-ai-gateway-infra(48d1b4b9-adf7-48d9-bbae-6830a2331143)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890890-tcgwk" podUID="48d1b4b9-adf7-48d9-bbae-6830a2331143" Oct 31 13:30:32.282280 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:32.282242 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890890-tcgwk"] Oct 31 13:30:32.287716 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:32.286822 5693 scope.go:122] "RemoveContainer" containerID="5494f42e19e3bcf259e21cd2bb70f523cd8e2157c09a3abd8b0bb5b68899b531" Oct 31 13:30:32.401821 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:32.401788 5693 scope.go:122] "RemoveContainer" containerID="6dfab6b8730763909d8b7e8ffb70e434218bb370a8fa37118504285caa587438" Oct 31 13:30:32.402103 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:32.402078 5693 scope.go:122] "RemoveContainer" containerID="52c295f72447153ed20fec75473d0d015be0d0d4657eac767c6833c6821e20de" Oct 31 13:30:32.403252 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:32.402355 5693 scope.go:122] "RemoveContainer" containerID="2032fcf645d87d0ea12f55cb1530f303a7373d4adf9cae9f4ffba2a3266cd91d" Oct 31 13:30:32.490840 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:32.490813 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890890-2ptgp" Oct 31 13:30:32.517836 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:32.517776 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba53a96b-92e6-4e45-9582-be73c7bf47bf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba53a96b-92e6-4e45-9582-be73c7bf47bf-maas-api-service-ca\") pod \"ba53a96b-92e6-4e45-9582-be73c7bf47bf\" (UID: \"ba53a96b-92e6-4e45-9582-be73c7bf47bf\") " Oct 31 13:30:32.518065 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:32.518052 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba53a96b-92e6-4e45-9582-be73c7bf47bf-kube-api-access-48cgh\" (UniqueName: \"kubernetes.io/projected/ba53a96b-92e6-4e45-9582-be73c7bf47bf-kube-api-access-48cgh\") pod \"ba53a96b-92e6-4e45-9582-be73c7bf47bf\" (UID: \"ba53a96b-92e6-4e45-9582-be73c7bf47bf\") " Oct 31 13:30:32.518132 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:32.518093 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba53a96b-92e6-4e45-9582-be73c7bf47bf-maas-api-service-ca" pod "ba53a96b-92e6-4e45-9582-be73c7bf47bf" (UID: "ba53a96b-92e6-4e45-9582-be73c7bf47bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:32.519265 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:32.519236 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba53a96b-92e6-4e45-9582-be73c7bf47bf-maas-api-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:32.520552 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:32.520529 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba53a96b-92e6-4e45-9582-be73c7bf47bf-kube-api-access-48cgh" pod "ba53a96b-92e6-4e45-9582-be73c7bf47bf" (UID: "ba53a96b-92e6-4e45-9582-be73c7bf47bf"). InnerVolumeSpecName "kube-api-access-48cgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:32.620806 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:32.620774 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48cgh\" (UniqueName: \"kubernetes.io/projected/ba53a96b-92e6-4e45-9582-be73c7bf47bf-kube-api-access-48cgh\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:33.274813 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:33.274772 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890890-pnqpx" event={"ID":"b41505b3-033f-4911-b467-0b9d29530279","Type":"ContainerStarted","Data":"8d41384ef2f159e6f1817cfc88f50d2fedfb461f233a1ac1b57c3a78e1c1a52e"} Oct 31 13:30:33.279376 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:33.279345 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-bh5qm" event={"ID":"e1369557-2959-4d29-8756-bf9ddbd4b610","Type":"ContainerStarted","Data":"544a9e305891d068b057eb7443f3fa8b4b4fb027d64d59905c04c52e132a9bf7"} Oct 31 13:30:33.281144 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:33.281118 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890890-2ptgp" event={"ID":"ba53a96b-92e6-4e45-9582-be73c7bf47bf","Type":"ContainerDied","Data":"e602ae88ce44fd08f00089ec25d442c33aac39921d7b3900cc3ce5d63982c6e3"} Oct 31 13:30:33.281257 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:33.281180 5693 scope.go:122] "RemoveContainer" containerID="be8f6bdeb2bf6e7732fc481607ba98881051b810ec345476172ae915145e8120" Oct 31 13:30:33.281257 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:33.281131 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890890-2ptgp" Oct 31 13:30:33.283243 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:33.283217 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890890-m2s7m" event={"ID":"1055f537-d4e4-46fc-b684-1ec68a9bda17","Type":"ContainerStarted","Data":"3ca196984e6ad671457251921767ec9e45fb578cbf7369f3c15ef33cb413b9d9"} Oct 31 13:30:33.331329 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:33.331295 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890890-2ptgp"] Oct 31 13:30:33.455626 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:33.455317 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890890-2ptgp"] Oct 31 13:30:33.548554 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:33.548474 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890890-tcgwk" Oct 31 13:30:33.646790 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:33.646760 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48d1b4b9-adf7-48d9-bbae-6830a2331143-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48d1b4b9-adf7-48d9-bbae-6830a2331143-maas-api-service-ca\") pod \"48d1b4b9-adf7-48d9-bbae-6830a2331143\" (UID: \"48d1b4b9-adf7-48d9-bbae-6830a2331143\") " Oct 31 13:30:33.647353 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:33.647054 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48d1b4b9-adf7-48d9-bbae-6830a2331143-kube-api-access-jfzhr\" (UniqueName: \"kubernetes.io/projected/48d1b4b9-adf7-48d9-bbae-6830a2331143-kube-api-access-jfzhr\") pod \"48d1b4b9-adf7-48d9-bbae-6830a2331143\" (UID: \"48d1b4b9-adf7-48d9-bbae-6830a2331143\") " Oct 31 13:30:33.647353 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:33.647138 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48d1b4b9-adf7-48d9-bbae-6830a2331143-maas-api-service-ca" pod "48d1b4b9-adf7-48d9-bbae-6830a2331143" (UID: "48d1b4b9-adf7-48d9-bbae-6830a2331143"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:33.648260 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:33.648234 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48d1b4b9-adf7-48d9-bbae-6830a2331143-maas-api-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:33.649812 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:33.649783 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48d1b4b9-adf7-48d9-bbae-6830a2331143-kube-api-access-jfzhr" pod "48d1b4b9-adf7-48d9-bbae-6830a2331143" (UID: "48d1b4b9-adf7-48d9-bbae-6830a2331143"). InnerVolumeSpecName "kube-api-access-jfzhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:33.750269 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:33.750233 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfzhr\" (UniqueName: \"kubernetes.io/projected/48d1b4b9-adf7-48d9-bbae-6830a2331143-kube-api-access-jfzhr\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:34.335523 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:34.335421 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890890-pnqpx"] Oct 31 13:30:34.370662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:34.370568 5693 generic.go:356] "Generic (PLEG): container finished" podID="1055f537-d4e4-46fc-b684-1ec68a9bda17" containerID="3ca196984e6ad671457251921767ec9e45fb578cbf7369f3c15ef33cb413b9d9" exitCode=60 Oct 31 13:30:34.370844 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:34.370815 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890890-m2s7m" event={"ID":"1055f537-d4e4-46fc-b684-1ec68a9bda17","Type":"ContainerDied","Data":"3ca196984e6ad671457251921767ec9e45fb578cbf7369f3c15ef33cb413b9d9"} Oct 31 13:30:34.370901 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:34.370889 5693 scope.go:122] "RemoveContainer" containerID="2032fcf645d87d0ea12f55cb1530f303a7373d4adf9cae9f4ffba2a3266cd91d" Oct 31 13:30:34.379674 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:34.371412 5693 scope.go:122] "RemoveContainer" containerID="3ca196984e6ad671457251921767ec9e45fb578cbf7369f3c15ef33cb413b9d9" Oct 31 13:30:34.379674 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:30:34.371844 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890890-m2s7m_odh-ai-gateway-infra(1055f537-d4e4-46fc-b684-1ec68a9bda17)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890890-m2s7m" podUID="1055f537-d4e4-46fc-b684-1ec68a9bda17" Oct 31 13:30:34.394709 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:34.392870 5693 generic.go:356] "Generic (PLEG): container finished" podID="b41505b3-033f-4911-b467-0b9d29530279" containerID="8d41384ef2f159e6f1817cfc88f50d2fedfb461f233a1ac1b57c3a78e1c1a52e" exitCode=60 Oct 31 13:30:34.394709 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:34.392965 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890890-pnqpx" event={"ID":"b41505b3-033f-4911-b467-0b9d29530279","Type":"ContainerDied","Data":"8d41384ef2f159e6f1817cfc88f50d2fedfb461f233a1ac1b57c3a78e1c1a52e"} Oct 31 13:30:34.425663 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:34.408078 5693 scope.go:122] "RemoveContainer" containerID="0d20fc9514a52413ebf4fad2a8f8615137a6c6a1bd28658317c15db2438b00c4" Oct 31 13:30:34.446070 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:34.446038 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890890-tcgwk" Oct 31 13:30:34.457350 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:34.457154 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890890-tcgwk" event={"ID":"48d1b4b9-adf7-48d9-bbae-6830a2331143","Type":"ContainerDied","Data":"8c3325f4b7c810a3171c226ff1fd5b457db44e5fdd0b0b6cf750053219c3c992"} Oct 31 13:30:34.486595 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:34.484363 5693 generic.go:356] "Generic (PLEG): container finished" podID="e1369557-2959-4d29-8756-bf9ddbd4b610" containerID="544a9e305891d068b057eb7443f3fa8b4b4fb027d64d59905c04c52e132a9bf7" exitCode=60 Oct 31 13:30:34.486595 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:34.484566 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-bh5qm" event={"ID":"e1369557-2959-4d29-8756-bf9ddbd4b610","Type":"ContainerDied","Data":"544a9e305891d068b057eb7443f3fa8b4b4fb027d64d59905c04c52e132a9bf7"} Oct 31 13:30:34.486595 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:34.485672 5693 scope.go:122] "RemoveContainer" containerID="544a9e305891d068b057eb7443f3fa8b4b4fb027d64d59905c04c52e132a9bf7" Oct 31 13:30:34.487282 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:30:34.486891 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890890-bh5qm_odh-ai-gateway-infra(e1369557-2959-4d29-8756-bf9ddbd4b610)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-bh5qm" podUID="e1369557-2959-4d29-8756-bf9ddbd4b610" Oct 31 13:30:34.535895 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:34.532866 5693 scope.go:122] "RemoveContainer" containerID="52c295f72447153ed20fec75473d0d015be0d0d4657eac767c6833c6821e20de" Oct 31 13:30:34.543725 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:34.543491 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890890-m2s7m"] Oct 31 13:30:34.575980 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:34.575947 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890890-bh5qm"] Oct 31 13:30:34.628674 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:34.627313 5693 scope.go:122] "RemoveContainer" containerID="57bfeac365cadf99277364643373bc45d1db14c77a5a38d1417197e001f3354d" Oct 31 13:30:34.699427 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:34.681873 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890890-tcgwk"] Oct 31 13:30:34.699427 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:34.684830 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890890-tcgwk"] Oct 31 13:30:34.701233 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:34.700204 5693 scope.go:122] "RemoveContainer" containerID="6dfab6b8730763909d8b7e8ffb70e434218bb370a8fa37118504285caa587438" Oct 31 13:30:34.857361 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:34.857298 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890890-pnqpx" Oct 31 13:30:34.892009 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:34.891942 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b41505b3-033f-4911-b467-0b9d29530279-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41505b3-033f-4911-b467-0b9d29530279-maas-api-service-ca\") pod \"b41505b3-033f-4911-b467-0b9d29530279\" (UID: \"b41505b3-033f-4911-b467-0b9d29530279\") " Oct 31 13:30:34.892566 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:34.892454 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b41505b3-033f-4911-b467-0b9d29530279-kube-api-access-zm7fr\" (UniqueName: \"kubernetes.io/projected/b41505b3-033f-4911-b467-0b9d29530279-kube-api-access-zm7fr\") pod \"b41505b3-033f-4911-b467-0b9d29530279\" (UID: \"b41505b3-033f-4911-b467-0b9d29530279\") " Oct 31 13:30:34.895100 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:34.895056 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b41505b3-033f-4911-b467-0b9d29530279-maas-api-service-ca" pod "b41505b3-033f-4911-b467-0b9d29530279" (UID: "b41505b3-033f-4911-b467-0b9d29530279"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:34.898526 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:34.898496 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b41505b3-033f-4911-b467-0b9d29530279-kube-api-access-zm7fr" pod "b41505b3-033f-4911-b467-0b9d29530279" (UID: "b41505b3-033f-4911-b467-0b9d29530279"). InnerVolumeSpecName "kube-api-access-zm7fr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:34.995838 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:34.995811 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41505b3-033f-4911-b467-0b9d29530279-maas-api-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:34.995838 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:34.995836 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm7fr\" (UniqueName: \"kubernetes.io/projected/b41505b3-033f-4911-b467-0b9d29530279-kube-api-access-zm7fr\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:35.414310 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:35.414275 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48d1b4b9-adf7-48d9-bbae-6830a2331143" path="/var/lib/kubelet/pods/48d1b4b9-adf7-48d9-bbae-6830a2331143/volumes" Oct 31 13:30:35.414915 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:35.414896 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba53a96b-92e6-4e45-9582-be73c7bf47bf" path="/var/lib/kubelet/pods/ba53a96b-92e6-4e45-9582-be73c7bf47bf/volumes" Oct 31 13:30:35.498528 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:35.498495 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890890-bbhs8" event={"ID":"47872ba8-dd46-4d67-a513-e1081171e92e","Type":"ContainerStarted","Data":"bdf7193fa44c0d758a1443316c7aec4fb83cb78c14b28ca165d2e54f45bf3fa7"} Oct 31 13:30:35.500160 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:35.500137 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890890-pnqpx" Oct 31 13:30:35.500286 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:35.500133 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890890-pnqpx" event={"ID":"b41505b3-033f-4911-b467-0b9d29530279","Type":"ContainerDied","Data":"28baa8e42b80b33cea5a5db08abf49db0d9bdb3a25caecd9c434513408a2e459"} Oct 31 13:30:35.500286 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:35.500259 5693 scope.go:122] "RemoveContainer" containerID="8d41384ef2f159e6f1817cfc88f50d2fedfb461f233a1ac1b57c3a78e1c1a52e" Oct 31 13:30:35.710430 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:35.710176 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890890-pnqpx"] Oct 31 13:30:35.716252 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:35.716224 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890890-pnqpx"] Oct 31 13:30:35.862277 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:35.862253 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890890-m2s7m" Oct 31 13:30:35.866571 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:35.866550 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-bh5qm" Oct 31 13:30:36.039528 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:36.039430 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1369557-2959-4d29-8756-bf9ddbd4b610-kube-api-access-7pkts\" (UniqueName: \"kubernetes.io/projected/e1369557-2959-4d29-8756-bf9ddbd4b610-kube-api-access-7pkts\") pod \"e1369557-2959-4d29-8756-bf9ddbd4b610\" (UID: \"e1369557-2959-4d29-8756-bf9ddbd4b610\") " Oct 31 13:30:36.039727 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:36.039547 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1369557-2959-4d29-8756-bf9ddbd4b610-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1369557-2959-4d29-8756-bf9ddbd4b610-maas-api-service-ca\") pod \"e1369557-2959-4d29-8756-bf9ddbd4b610\" (UID: \"e1369557-2959-4d29-8756-bf9ddbd4b610\") " Oct 31 13:30:36.039727 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:36.039703 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1055f537-d4e4-46fc-b684-1ec68a9bda17-kube-api-access-5895r\" (UniqueName: \"kubernetes.io/projected/1055f537-d4e4-46fc-b684-1ec68a9bda17-kube-api-access-5895r\") pod \"1055f537-d4e4-46fc-b684-1ec68a9bda17\" (UID: \"1055f537-d4e4-46fc-b684-1ec68a9bda17\") " Oct 31 13:30:36.039875 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:36.039742 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1055f537-d4e4-46fc-b684-1ec68a9bda17-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1055f537-d4e4-46fc-b684-1ec68a9bda17-maas-api-service-ca\") pod \"1055f537-d4e4-46fc-b684-1ec68a9bda17\" (UID: \"1055f537-d4e4-46fc-b684-1ec68a9bda17\") " Oct 31 13:30:36.040202 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:36.040172 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1369557-2959-4d29-8756-bf9ddbd4b610-maas-api-service-ca" pod "e1369557-2959-4d29-8756-bf9ddbd4b610" (UID: "e1369557-2959-4d29-8756-bf9ddbd4b610"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:36.040536 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:36.040517 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1369557-2959-4d29-8756-bf9ddbd4b610-maas-api-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:36.040768 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:36.040737 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1055f537-d4e4-46fc-b684-1ec68a9bda17-maas-api-service-ca" pod "1055f537-d4e4-46fc-b684-1ec68a9bda17" (UID: "1055f537-d4e4-46fc-b684-1ec68a9bda17"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:36.042610 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:36.042590 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1369557-2959-4d29-8756-bf9ddbd4b610-kube-api-access-7pkts" pod "e1369557-2959-4d29-8756-bf9ddbd4b610" (UID: "e1369557-2959-4d29-8756-bf9ddbd4b610"). InnerVolumeSpecName "kube-api-access-7pkts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:36.042947 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:36.042924 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1055f537-d4e4-46fc-b684-1ec68a9bda17-kube-api-access-5895r" pod "1055f537-d4e4-46fc-b684-1ec68a9bda17" (UID: "1055f537-d4e4-46fc-b684-1ec68a9bda17"). InnerVolumeSpecName "kube-api-access-5895r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:36.141649 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:36.141598 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pkts\" (UniqueName: \"kubernetes.io/projected/e1369557-2959-4d29-8756-bf9ddbd4b610-kube-api-access-7pkts\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:36.141649 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:36.141630 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5895r\" (UniqueName: \"kubernetes.io/projected/1055f537-d4e4-46fc-b684-1ec68a9bda17-kube-api-access-5895r\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:36.141649 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:36.141654 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1055f537-d4e4-46fc-b684-1ec68a9bda17-maas-api-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:36.401718 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:36.401682 5693 scope.go:122] "RemoveContainer" containerID="c834966a14407e8075d1aabe91a45ba9eee98f14f4f37dcd12dc3c28f3861866" Oct 31 13:30:36.518176 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:36.518147 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890890-m2s7m" Oct 31 13:30:36.518176 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:36.518171 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890890-m2s7m" event={"ID":"1055f537-d4e4-46fc-b684-1ec68a9bda17","Type":"ContainerDied","Data":"4bcbec016a7e375399ca4e0f65afd6f80ef425241713baa9e137a4091595d5d8"} Oct 31 13:30:36.518466 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:36.518231 5693 scope.go:122] "RemoveContainer" containerID="3ca196984e6ad671457251921767ec9e45fb578cbf7369f3c15ef33cb413b9d9" Oct 31 13:30:36.520847 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:36.520817 5693 generic.go:356] "Generic (PLEG): container finished" podID="47872ba8-dd46-4d67-a513-e1081171e92e" containerID="bdf7193fa44c0d758a1443316c7aec4fb83cb78c14b28ca165d2e54f45bf3fa7" exitCode=60 Oct 31 13:30:36.520958 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:36.520874 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890890-bbhs8" event={"ID":"47872ba8-dd46-4d67-a513-e1081171e92e","Type":"ContainerDied","Data":"bdf7193fa44c0d758a1443316c7aec4fb83cb78c14b28ca165d2e54f45bf3fa7"} Oct 31 13:30:36.521569 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:36.521380 5693 scope.go:122] "RemoveContainer" containerID="bdf7193fa44c0d758a1443316c7aec4fb83cb78c14b28ca165d2e54f45bf3fa7" Oct 31 13:30:36.521870 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:30:36.521845 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890890-bbhs8_odh-ai-gateway-infra(47872ba8-dd46-4d67-a513-e1081171e92e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890890-bbhs8" podUID="47872ba8-dd46-4d67-a513-e1081171e92e" Oct 31 13:30:36.524045 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:36.524019 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-bh5qm" event={"ID":"e1369557-2959-4d29-8756-bf9ddbd4b610","Type":"ContainerDied","Data":"5417cc7fcbf22e8e2b2ed8d3fbf0f2c7c66ef2e4f6cf564f981e6cc7771284b2"} Oct 31 13:30:36.524162 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:36.524058 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-bh5qm" Oct 31 13:30:36.534128 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:36.534107 5693 scope.go:122] "RemoveContainer" containerID="0d20fc9514a52413ebf4fad2a8f8615137a6c6a1bd28658317c15db2438b00c4" Oct 31 13:30:36.550495 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:36.549679 5693 scope.go:122] "RemoveContainer" containerID="544a9e305891d068b057eb7443f3fa8b4b4fb027d64d59905c04c52e132a9bf7" Oct 31 13:30:36.567180 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:36.567148 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890890-bbhs8"] Oct 31 13:30:36.592289 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:36.592260 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890890-m2s7m"] Oct 31 13:30:36.604353 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:36.604323 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c2c98b-29890890-m2s7m"] Oct 31 13:30:36.620706 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:36.620673 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890890-bh5qm"] Oct 31 13:30:36.625957 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:36.625933 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890890-bh5qm"] Oct 31 13:30:37.427383 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:37.427328 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1055f537-d4e4-46fc-b684-1ec68a9bda17" path="/var/lib/kubelet/pods/1055f537-d4e4-46fc-b684-1ec68a9bda17/volumes" Oct 31 13:30:37.428195 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:37.428155 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b41505b3-033f-4911-b467-0b9d29530279" path="/var/lib/kubelet/pods/b41505b3-033f-4911-b467-0b9d29530279/volumes" Oct 31 13:30:37.428885 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:37.428865 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1369557-2959-4d29-8756-bf9ddbd4b610" path="/var/lib/kubelet/pods/e1369557-2959-4d29-8756-bf9ddbd4b610/volumes" Oct 31 13:30:37.540023 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:37.539987 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39960-29890890-n2j72" event={"ID":"5ca294e6-ea27-4bc6-b710-60e6b27485ee","Type":"ContainerStarted","Data":"cba5e1778292caf59da70d11217b5bc41ab64cac23de9fda392338830b1a747a"} Oct 31 13:30:37.790578 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:37.790547 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890890-bbhs8" Oct 31 13:30:37.888055 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:37.888013 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47872ba8-dd46-4d67-a513-e1081171e92e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47872ba8-dd46-4d67-a513-e1081171e92e-maas-api-service-ca\") pod \"47872ba8-dd46-4d67-a513-e1081171e92e\" (UID: \"47872ba8-dd46-4d67-a513-e1081171e92e\") " Oct 31 13:30:37.888257 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:37.888167 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47872ba8-dd46-4d67-a513-e1081171e92e-kube-api-access-mxwww\" (UniqueName: \"kubernetes.io/projected/47872ba8-dd46-4d67-a513-e1081171e92e-kube-api-access-mxwww\") pod \"47872ba8-dd46-4d67-a513-e1081171e92e\" (UID: \"47872ba8-dd46-4d67-a513-e1081171e92e\") " Oct 31 13:30:37.888524 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:37.888494 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47872ba8-dd46-4d67-a513-e1081171e92e-maas-api-service-ca" pod "47872ba8-dd46-4d67-a513-e1081171e92e" (UID: "47872ba8-dd46-4d67-a513-e1081171e92e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:37.888830 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:37.888813 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47872ba8-dd46-4d67-a513-e1081171e92e-maas-api-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:37.890542 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:37.890522 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47872ba8-dd46-4d67-a513-e1081171e92e-kube-api-access-mxwww" pod "47872ba8-dd46-4d67-a513-e1081171e92e" (UID: "47872ba8-dd46-4d67-a513-e1081171e92e"). InnerVolumeSpecName "kube-api-access-mxwww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:38.037371 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:38.025764 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxwww\" (UniqueName: \"kubernetes.io/projected/47872ba8-dd46-4d67-a513-e1081171e92e-kube-api-access-mxwww\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:38.401197 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:38.401169 5693 scope.go:122] "RemoveContainer" containerID="5b28df3f900ae256440e7d83f386deed886bae12d45fa4cf8019bd599683ac3e" Oct 31 13:30:38.592664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:38.592584 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39960-29890890-n2j72"] Oct 31 13:30:38.671972 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:38.654125 5693 generic.go:356] "Generic (PLEG): container finished" podID="5ca294e6-ea27-4bc6-b710-60e6b27485ee" containerID="cba5e1778292caf59da70d11217b5bc41ab64cac23de9fda392338830b1a747a" exitCode=60 Oct 31 13:30:38.719947 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:38.719919 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890890-bbhs8" Oct 31 13:30:38.837668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:38.832741 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39960-29890890-n2j72" event={"ID":"5ca294e6-ea27-4bc6-b710-60e6b27485ee","Type":"ContainerDied","Data":"cba5e1778292caf59da70d11217b5bc41ab64cac23de9fda392338830b1a747a"} Oct 31 13:30:38.837668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:38.832798 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890890-bbhs8" event={"ID":"47872ba8-dd46-4d67-a513-e1081171e92e","Type":"ContainerDied","Data":"f24fda5fc8f4a3cea5e077c7d3f3226b5aeb22d4d7ef64cf9aef764f50fc839a"} Oct 31 13:30:38.837668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:38.832832 5693 scope.go:122] "RemoveContainer" containerID="c834966a14407e8075d1aabe91a45ba9eee98f14f4f37dcd12dc3c28f3861866" Oct 31 13:30:38.922846 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:38.922820 5693 scope.go:122] "RemoveContainer" containerID="bdf7193fa44c0d758a1443316c7aec4fb83cb78c14b28ca165d2e54f45bf3fa7" Oct 31 13:30:38.962635 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:38.962600 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890890-bbhs8"] Oct 31 13:30:38.975585 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:38.975556 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890890-bbhs8"] Oct 31 13:30:39.109380 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.109353 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39960-29890890-n2j72" Oct 31 13:30:39.153938 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.153904 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs"] Oct 31 13:30:39.155022 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.154999 5693 cpu_manager.go:414] "containerMap: removing container" podUID="48d1b4b9-adf7-48d9-bbae-6830a2331143" containerName="cleanup" Oct 31 13:30:39.155022 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155021 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48d1b4b9-adf7-48d9-bbae-6830a2331143" containerName="cleanup" Oct 31 13:30:39.155201 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155038 5693 cpu_manager.go:414] "containerMap: removing container" podUID="47872ba8-dd46-4d67-a513-e1081171e92e" containerName="cleanup" Oct 31 13:30:39.155201 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155045 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47872ba8-dd46-4d67-a513-e1081171e92e" containerName="cleanup" Oct 31 13:30:39.155201 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155076 5693 cpu_manager.go:414] "containerMap: removing container" podUID="1055f537-d4e4-46fc-b684-1ec68a9bda17" containerName="cleanup" Oct 31 13:30:39.155201 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155084 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1055f537-d4e4-46fc-b684-1ec68a9bda17" containerName="cleanup" Oct 31 13:30:39.155201 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155098 5693 cpu_manager.go:414] "containerMap: removing container" podUID="b41505b3-033f-4911-b467-0b9d29530279" containerName="cleanup" Oct 31 13:30:39.155201 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155105 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b41505b3-033f-4911-b467-0b9d29530279" containerName="cleanup" Oct 31 13:30:39.155201 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155121 5693 cpu_manager.go:414] "containerMap: removing container" podUID="47872ba8-dd46-4d67-a513-e1081171e92e" containerName="cleanup" Oct 31 13:30:39.155201 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155127 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47872ba8-dd46-4d67-a513-e1081171e92e" containerName="cleanup" Oct 31 13:30:39.155201 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155155 5693 cpu_manager.go:414] "containerMap: removing container" podUID="24816799-6034-48e6-9711-d9a342c0ef28" containerName="cleanup" Oct 31 13:30:39.155201 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155163 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24816799-6034-48e6-9711-d9a342c0ef28" containerName="cleanup" Oct 31 13:30:39.155201 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155174 5693 cpu_manager.go:414] "containerMap: removing container" podUID="5ca294e6-ea27-4bc6-b710-60e6b27485ee" containerName="cleanup" Oct 31 13:30:39.155201 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155181 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ca294e6-ea27-4bc6-b710-60e6b27485ee" containerName="cleanup" Oct 31 13:30:39.155201 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155196 5693 cpu_manager.go:414] "containerMap: removing container" podUID="1055f537-d4e4-46fc-b684-1ec68a9bda17" containerName="cleanup" Oct 31 13:30:39.155201 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155202 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1055f537-d4e4-46fc-b684-1ec68a9bda17" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155219 5693 cpu_manager.go:414] "containerMap: removing container" podUID="92712c0d-d49c-46fe-8bad-a023a997d5f2" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155227 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92712c0d-d49c-46fe-8bad-a023a997d5f2" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155239 5693 cpu_manager.go:414] "containerMap: removing container" podUID="c0fb78e4-7046-48f0-a737-4ca457074027" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155246 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0fb78e4-7046-48f0-a737-4ca457074027" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155255 5693 cpu_manager.go:414] "containerMap: removing container" podUID="5ca294e6-ea27-4bc6-b710-60e6b27485ee" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155261 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ca294e6-ea27-4bc6-b710-60e6b27485ee" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155271 5693 cpu_manager.go:414] "containerMap: removing container" podUID="e1369557-2959-4d29-8756-bf9ddbd4b610" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155277 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1369557-2959-4d29-8756-bf9ddbd4b610" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155291 5693 cpu_manager.go:414] "containerMap: removing container" podUID="48d1b4b9-adf7-48d9-bbae-6830a2331143" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155297 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48d1b4b9-adf7-48d9-bbae-6830a2331143" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155307 5693 cpu_manager.go:414] "containerMap: removing container" podUID="ba53a96b-92e6-4e45-9582-be73c7bf47bf" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155314 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba53a96b-92e6-4e45-9582-be73c7bf47bf" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155324 5693 cpu_manager.go:414] "containerMap: removing container" podUID="1055f537-d4e4-46fc-b684-1ec68a9bda17" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155330 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1055f537-d4e4-46fc-b684-1ec68a9bda17" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155339 5693 cpu_manager.go:414] "containerMap: removing container" podUID="e1369557-2959-4d29-8756-bf9ddbd4b610" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155346 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1369557-2959-4d29-8756-bf9ddbd4b610" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155363 5693 cpu_manager.go:414] "containerMap: removing container" podUID="e1369557-2959-4d29-8756-bf9ddbd4b610" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155370 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1369557-2959-4d29-8756-bf9ddbd4b610" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155381 5693 cpu_manager.go:414] "containerMap: removing container" podUID="ba53a96b-92e6-4e45-9582-be73c7bf47bf" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155388 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba53a96b-92e6-4e45-9582-be73c7bf47bf" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155398 5693 cpu_manager.go:414] "containerMap: removing container" podUID="ba53a96b-92e6-4e45-9582-be73c7bf47bf" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155404 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba53a96b-92e6-4e45-9582-be73c7bf47bf" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155419 5693 cpu_manager.go:414] "containerMap: removing container" podUID="47872ba8-dd46-4d67-a513-e1081171e92e" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155426 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47872ba8-dd46-4d67-a513-e1081171e92e" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155437 5693 cpu_manager.go:414] "containerMap: removing container" podUID="92712c0d-d49c-46fe-8bad-a023a997d5f2" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155444 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92712c0d-d49c-46fe-8bad-a023a997d5f2" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155468 5693 cpu_manager.go:414] "containerMap: removing container" podUID="b41505b3-033f-4911-b467-0b9d29530279" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155476 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b41505b3-033f-4911-b467-0b9d29530279" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155488 5693 cpu_manager.go:414] "containerMap: removing container" podUID="b41505b3-033f-4911-b467-0b9d29530279" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155496 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b41505b3-033f-4911-b467-0b9d29530279" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155514 5693 cpu_manager.go:414] "containerMap: removing container" podUID="e486c623-3983-406f-9f04-21a0d994c828" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155522 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e486c623-3983-406f-9f04-21a0d994c828" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155540 5693 cpu_manager.go:414] "containerMap: removing container" podUID="c0fb78e4-7046-48f0-a737-4ca457074027" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155547 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0fb78e4-7046-48f0-a737-4ca457074027" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155560 5693 cpu_manager.go:414] "containerMap: removing container" podUID="c0fb78e4-7046-48f0-a737-4ca457074027" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155567 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0fb78e4-7046-48f0-a737-4ca457074027" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155583 5693 cpu_manager.go:414] "containerMap: removing container" podUID="24816799-6034-48e6-9711-d9a342c0ef28" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155591 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24816799-6034-48e6-9711-d9a342c0ef28" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155833 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="b41505b3-033f-4911-b467-0b9d29530279" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155850 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="b41505b3-033f-4911-b467-0b9d29530279" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155865 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="e486c623-3983-406f-9f04-21a0d994c828" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155876 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="92712c0d-d49c-46fe-8bad-a023a997d5f2" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155885 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0fb78e4-7046-48f0-a737-4ca457074027" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155893 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0fb78e4-7046-48f0-a737-4ca457074027" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155903 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="24816799-6034-48e6-9711-d9a342c0ef28" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155913 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="48d1b4b9-adf7-48d9-bbae-6830a2331143" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155921 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="47872ba8-dd46-4d67-a513-e1081171e92e" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155934 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="1055f537-d4e4-46fc-b684-1ec68a9bda17" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155943 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="47872ba8-dd46-4d67-a513-e1081171e92e" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155954 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="24816799-6034-48e6-9711-d9a342c0ef28" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155963 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ca294e6-ea27-4bc6-b710-60e6b27485ee" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155974 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="92712c0d-d49c-46fe-8bad-a023a997d5f2" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155982 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ca294e6-ea27-4bc6-b710-60e6b27485ee" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.155989 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1369557-2959-4d29-8756-bf9ddbd4b610" containerName="cleanup" Oct 31 13:30:39.155968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.156000 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba53a96b-92e6-4e45-9582-be73c7bf47bf" containerName="cleanup" Oct 31 13:30:39.158742 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.156010 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="1055f537-d4e4-46fc-b684-1ec68a9bda17" containerName="cleanup" Oct 31 13:30:39.158742 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.156018 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1369557-2959-4d29-8756-bf9ddbd4b610" containerName="cleanup" Oct 31 13:30:39.158742 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.156027 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1369557-2959-4d29-8756-bf9ddbd4b610" containerName="cleanup" Oct 31 13:30:39.158742 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.156037 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba53a96b-92e6-4e45-9582-be73c7bf47bf" containerName="cleanup" Oct 31 13:30:39.158742 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.156049 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba53a96b-92e6-4e45-9582-be73c7bf47bf" containerName="cleanup" Oct 31 13:30:39.158742 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.156056 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="48d1b4b9-adf7-48d9-bbae-6830a2331143" containerName="cleanup" Oct 31 13:30:39.158742 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.156064 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="92712c0d-d49c-46fe-8bad-a023a997d5f2" containerName="cleanup" Oct 31 13:30:39.160949 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.160928 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" Oct 31 13:30:39.173005 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.172939 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-3cddb3-openshift-default-dockercfg-9cn7x\"" Oct 31 13:30:39.185404 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.184994 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs"] Oct 31 13:30:39.204472 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.204437 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ca294e6-ea27-4bc6-b710-60e6b27485ee-kube-api-access-pr862\" (UniqueName: \"kubernetes.io/projected/5ca294e6-ea27-4bc6-b710-60e6b27485ee-kube-api-access-pr862\") pod \"5ca294e6-ea27-4bc6-b710-60e6b27485ee\" (UID: \"5ca294e6-ea27-4bc6-b710-60e6b27485ee\") " Oct 31 13:30:39.204768 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.204746 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ca294e6-ea27-4bc6-b710-60e6b27485ee-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ca294e6-ea27-4bc6-b710-60e6b27485ee-maas-api-service-ca\") pod \"5ca294e6-ea27-4bc6-b710-60e6b27485ee\" (UID: \"5ca294e6-ea27-4bc6-b710-60e6b27485ee\") " Oct 31 13:30:39.205096 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.205064 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ca294e6-ea27-4bc6-b710-60e6b27485ee-maas-api-service-ca" pod "5ca294e6-ea27-4bc6-b710-60e6b27485ee" (UID: "5ca294e6-ea27-4bc6-b710-60e6b27485ee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:39.205310 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.205291 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d137459e-9920-4c27-a910-0216cc6a7168-workload-socket\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs\" (UID: \"d137459e-9920-4c27-a910-0216cc6a7168\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" Oct 31 13:30:39.205387 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.205344 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d137459e-9920-4c27-a910-0216cc6a7168-credential-socket\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs\" (UID: \"d137459e-9920-4c27-a910-0216cc6a7168\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" Oct 31 13:30:39.205512 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.205489 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d137459e-9920-4c27-a910-0216cc6a7168-workload-certs\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs\" (UID: \"d137459e-9920-4c27-a910-0216cc6a7168\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" Oct 31 13:30:39.205578 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.205560 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d137459e-9920-4c27-a910-0216cc6a7168-istio-podinfo\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs\" (UID: \"d137459e-9920-4c27-a910-0216cc6a7168\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" Oct 31 13:30:39.205663 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.205629 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4mr9\" (UniqueName: \"kubernetes.io/projected/d137459e-9920-4c27-a910-0216cc6a7168-kube-api-access-f4mr9\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs\" (UID: \"d137459e-9920-4c27-a910-0216cc6a7168\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" Oct 31 13:30:39.205739 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.205717 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d137459e-9920-4c27-a910-0216cc6a7168-istio-data\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs\" (UID: \"d137459e-9920-4c27-a910-0216cc6a7168\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" Oct 31 13:30:39.205830 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.205816 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d137459e-9920-4c27-a910-0216cc6a7168-istiod-ca-cert\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs\" (UID: \"d137459e-9920-4c27-a910-0216cc6a7168\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" Oct 31 13:30:39.205946 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.205931 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d137459e-9920-4c27-a910-0216cc6a7168-istio-token\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs\" (UID: \"d137459e-9920-4c27-a910-0216cc6a7168\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" Oct 31 13:30:39.206509 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.206039 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d137459e-9920-4c27-a910-0216cc6a7168-istio-envoy\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs\" (UID: \"d137459e-9920-4c27-a910-0216cc6a7168\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" Oct 31 13:30:39.206509 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.206213 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ca294e6-ea27-4bc6-b710-60e6b27485ee-maas-api-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:39.206920 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.206899 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ca294e6-ea27-4bc6-b710-60e6b27485ee-kube-api-access-pr862" pod "5ca294e6-ea27-4bc6-b710-60e6b27485ee" (UID: "5ca294e6-ea27-4bc6-b710-60e6b27485ee"). InnerVolumeSpecName "kube-api-access-pr862". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:39.310904 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.308464 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4mr9\" (UniqueName: \"kubernetes.io/projected/d137459e-9920-4c27-a910-0216cc6a7168-kube-api-access-f4mr9\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs\" (UID: \"d137459e-9920-4c27-a910-0216cc6a7168\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" Oct 31 13:30:39.310904 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.308521 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d137459e-9920-4c27-a910-0216cc6a7168-istio-data\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs\" (UID: \"d137459e-9920-4c27-a910-0216cc6a7168\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" Oct 31 13:30:39.310904 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.308612 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d137459e-9920-4c27-a910-0216cc6a7168-istiod-ca-cert\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs\" (UID: \"d137459e-9920-4c27-a910-0216cc6a7168\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" Oct 31 13:30:39.310904 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.308762 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d137459e-9920-4c27-a910-0216cc6a7168-istio-token\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs\" (UID: \"d137459e-9920-4c27-a910-0216cc6a7168\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" Oct 31 13:30:39.310904 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.308860 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d137459e-9920-4c27-a910-0216cc6a7168-istio-envoy\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs\" (UID: \"d137459e-9920-4c27-a910-0216cc6a7168\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" Oct 31 13:30:39.310904 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.309059 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d137459e-9920-4c27-a910-0216cc6a7168-workload-socket\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs\" (UID: \"d137459e-9920-4c27-a910-0216cc6a7168\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" Oct 31 13:30:39.310904 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.309114 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d137459e-9920-4c27-a910-0216cc6a7168-credential-socket\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs\" (UID: \"d137459e-9920-4c27-a910-0216cc6a7168\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" Oct 31 13:30:39.310904 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.309196 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d137459e-9920-4c27-a910-0216cc6a7168-workload-certs\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs\" (UID: \"d137459e-9920-4c27-a910-0216cc6a7168\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" Oct 31 13:30:39.310904 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.309274 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d137459e-9920-4c27-a910-0216cc6a7168-istio-podinfo\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs\" (UID: \"d137459e-9920-4c27-a910-0216cc6a7168\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" Oct 31 13:30:39.310904 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.309431 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr862\" (UniqueName: \"kubernetes.io/projected/5ca294e6-ea27-4bc6-b710-60e6b27485ee-kube-api-access-pr862\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:39.316425 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.316342 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d137459e-9920-4c27-a910-0216cc6a7168-istiod-ca-cert\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs\" (UID: \"d137459e-9920-4c27-a910-0216cc6a7168\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" Oct 31 13:30:39.319359 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.319305 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d137459e-9920-4c27-a910-0216cc6a7168-workload-certs\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs\" (UID: \"d137459e-9920-4c27-a910-0216cc6a7168\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" Oct 31 13:30:39.320519 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.320135 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d137459e-9920-4c27-a910-0216cc6a7168-credential-socket\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs\" (UID: \"d137459e-9920-4c27-a910-0216cc6a7168\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" Oct 31 13:30:39.320519 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.320375 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d137459e-9920-4c27-a910-0216cc6a7168-istio-data\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs\" (UID: \"d137459e-9920-4c27-a910-0216cc6a7168\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" Oct 31 13:30:39.324672 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.321142 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d137459e-9920-4c27-a910-0216cc6a7168-workload-socket\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs\" (UID: \"d137459e-9920-4c27-a910-0216cc6a7168\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" Oct 31 13:30:39.327096 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.327073 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d137459e-9920-4c27-a910-0216cc6a7168-istio-envoy\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs\" (UID: \"d137459e-9920-4c27-a910-0216cc6a7168\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" Oct 31 13:30:39.327096 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.327086 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d137459e-9920-4c27-a910-0216cc6a7168-istio-podinfo\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs\" (UID: \"d137459e-9920-4c27-a910-0216cc6a7168\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" Oct 31 13:30:39.339950 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.339921 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d137459e-9920-4c27-a910-0216cc6a7168-istio-token\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs\" (UID: \"d137459e-9920-4c27-a910-0216cc6a7168\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" Oct 31 13:30:39.341000 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.340969 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4mr9\" (UniqueName: \"kubernetes.io/projected/d137459e-9920-4c27-a910-0216cc6a7168-kube-api-access-f4mr9\") pod \"e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs\" (UID: \"d137459e-9920-4c27-a910-0216cc6a7168\") " pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" Oct 31 13:30:39.413544 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.413510 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47872ba8-dd46-4d67-a513-e1081171e92e" path="/var/lib/kubelet/pods/47872ba8-dd46-4d67-a513-e1081171e92e/volumes" Oct 31 13:30:39.482182 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.482104 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" Oct 31 13:30:39.731509 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.731484 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39960-29890890-n2j72" Oct 31 13:30:39.731952 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.731480 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39960-29890890-n2j72" event={"ID":"5ca294e6-ea27-4bc6-b710-60e6b27485ee","Type":"ContainerDied","Data":"046fead63487eb1c78b51a961ebdf027e95217db62f76cb714bd98604b132566"} Oct 31 13:30:39.731952 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.731630 5693 scope.go:122] "RemoveContainer" containerID="cba5e1778292caf59da70d11217b5bc41ab64cac23de9fda392338830b1a747a" Oct 31 13:30:39.736154 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.736123 5693 generic.go:356] "Generic (PLEG): container finished" podID="c18e585d-a441-4ca7-b2b8-4342056c603f" containerID="27f64bd5c87da8cd0a87120038de3b07f8ead9ba5596118701d35ebf5afdb216" exitCode=60 Oct 31 13:30:39.736270 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.736197 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890890-cjwzd" event={"ID":"c18e585d-a441-4ca7-b2b8-4342056c603f","Type":"ContainerDied","Data":"27f64bd5c87da8cd0a87120038de3b07f8ead9ba5596118701d35ebf5afdb216"} Oct 31 13:30:39.736820 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.736803 5693 scope.go:122] "RemoveContainer" containerID="27f64bd5c87da8cd0a87120038de3b07f8ead9ba5596118701d35ebf5afdb216" Oct 31 13:30:39.737189 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:30:39.737163 5693 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-5dc570-29890890-cjwzd_odh-ai-gateway-infra(c18e585d-a441-4ca7-b2b8-4342056c603f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890890-cjwzd" podUID="c18e585d-a441-4ca7-b2b8-4342056c603f" Oct 31 13:30:39.746057 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.746038 5693 scope.go:122] "RemoveContainer" containerID="5b28df3f900ae256440e7d83f386deed886bae12d45fa4cf8019bd599683ac3e" Oct 31 13:30:39.757550 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.757524 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39960-29890890-n2j72"] Oct 31 13:30:39.771222 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.771189 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39960-29890890-n2j72"] Oct 31 13:30:39.829595 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.829569 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs"] Oct 31 13:30:39.829983 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:30:39.829957 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd137459e_9920_4c27_a910_0216cc6a7168.slice/crio-7ef47b380dd6dfbcc6fb95269446aee029b284cef76b0e8f838637141b5ad432 WatchSource:0}: Error finding container 7ef47b380dd6dfbcc6fb95269446aee029b284cef76b0e8f838637141b5ad432: Status 404 returned error can't find the container with id 7ef47b380dd6dfbcc6fb95269446aee029b284cef76b0e8f838637141b5ad432 Oct 31 13:30:39.833054 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.833019 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:30:39.833150 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.833086 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:30:39.833150 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:39.833115 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:30:40.750246 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:40.750212 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" event={"ID":"d137459e-9920-4c27-a910-0216cc6a7168","Type":"ContainerStarted","Data":"8eac644d96d259e42257cee93694e01758a3f6c3885083b74871f9f8ea701d42"} Oct 31 13:30:40.750799 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:40.750266 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" event={"ID":"d137459e-9920-4c27-a910-0216cc6a7168","Type":"ContainerStarted","Data":"7ef47b380dd6dfbcc6fb95269446aee029b284cef76b0e8f838637141b5ad432"} Oct 31 13:30:40.786890 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:40.786842 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" podStartSLOduration=2.786828878 podStartE2EDuration="2.786828878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:40.783208461 +0000 UTC m=+3491.490794757" watchObservedRunningTime="2026-10-31 13:30:40.786828878 +0000 UTC m=+3491.494415173" Oct 31 13:30:40.825811 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:40.825780 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890890-cjwzd"] Oct 31 13:30:41.119697 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:41.119669 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890890-cjwzd" Oct 31 13:30:41.280558 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:41.280520 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c18e585d-a441-4ca7-b2b8-4342056c603f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c18e585d-a441-4ca7-b2b8-4342056c603f-maas-api-service-ca\") pod \"c18e585d-a441-4ca7-b2b8-4342056c603f\" (UID: \"c18e585d-a441-4ca7-b2b8-4342056c603f\") " Oct 31 13:30:41.280783 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:41.280694 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c18e585d-a441-4ca7-b2b8-4342056c603f-kube-api-access-gwhqp\" (UniqueName: \"kubernetes.io/projected/c18e585d-a441-4ca7-b2b8-4342056c603f-kube-api-access-gwhqp\") pod \"c18e585d-a441-4ca7-b2b8-4342056c603f\" (UID: \"c18e585d-a441-4ca7-b2b8-4342056c603f\") " Oct 31 13:30:41.282283 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:41.282242 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c18e585d-a441-4ca7-b2b8-4342056c603f-maas-api-service-ca" pod "c18e585d-a441-4ca7-b2b8-4342056c603f" (UID: "c18e585d-a441-4ca7-b2b8-4342056c603f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:41.285111 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:41.285084 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c18e585d-a441-4ca7-b2b8-4342056c603f-kube-api-access-gwhqp" pod "c18e585d-a441-4ca7-b2b8-4342056c603f" (UID: "c18e585d-a441-4ca7-b2b8-4342056c603f"). InnerVolumeSpecName "kube-api-access-gwhqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:41.383561 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:41.383484 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c18e585d-a441-4ca7-b2b8-4342056c603f-maas-api-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:41.383561 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:41.383517 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwhqp\" (UniqueName: \"kubernetes.io/projected/c18e585d-a441-4ca7-b2b8-4342056c603f-kube-api-access-gwhqp\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:41.416808 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:41.416769 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ca294e6-ea27-4bc6-b710-60e6b27485ee" path="/var/lib/kubelet/pods/5ca294e6-ea27-4bc6-b710-60e6b27485ee/volumes" Oct 31 13:30:41.483316 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:41.483281 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" Oct 31 13:30:41.499756 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:41.492479 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" Oct 31 13:30:41.762114 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:41.762063 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890890-cjwzd" Oct 31 13:30:41.762114 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:41.762091 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890890-cjwzd" event={"ID":"c18e585d-a441-4ca7-b2b8-4342056c603f","Type":"ContainerDied","Data":"d841ac784c7f3601b31814c205d0edc2c7ee13825f189de22770ab78a377c59e"} Oct 31 13:30:41.762733 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:41.762158 5693 scope.go:122] "RemoveContainer" containerID="27f64bd5c87da8cd0a87120038de3b07f8ead9ba5596118701d35ebf5afdb216" Oct 31 13:30:41.762733 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:41.762680 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" Oct 31 13:30:41.763939 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:41.763921 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" Oct 31 13:30:41.963235 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:41.963198 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890890-cjwzd"] Oct 31 13:30:41.971529 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:41.971496 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890890-cjwzd"] Oct 31 13:30:43.501272 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:30:43.501237 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c18e585d-a441-4ca7-b2b8-4342056c603f" path="/var/lib/kubelet/pods/c18e585d-a441-4ca7-b2b8-4342056c603f/volumes" Oct 31 13:30:48.690311 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:30:48.690274 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:31:08.721098 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:08.721004 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx"] Oct 31 13:31:08.721737 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:08.721300 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" podUID="76570ffc-6a6b-4f4d-b8f0-0db5cde967c6" containerName="istio-proxy" containerID="cri-o://0e95c2e859ab2c48f3918e71e9bd286dba34af97ef1aa9c93f7738fff7d99044" gracePeriod=30 Oct 31 13:31:10.708747 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:10.708706 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61110dd0-xpjvk"] Oct 31 13:31:10.709723 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:10.709702 5693 cpu_manager.go:414] "containerMap: removing container" podUID="c18e585d-a441-4ca7-b2b8-4342056c603f" containerName="cleanup" Oct 31 13:31:10.709723 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:10.709720 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c18e585d-a441-4ca7-b2b8-4342056c603f" containerName="cleanup" Oct 31 13:31:10.709943 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:10.709746 5693 cpu_manager.go:414] "containerMap: removing container" podUID="c18e585d-a441-4ca7-b2b8-4342056c603f" containerName="cleanup" Oct 31 13:31:10.709943 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:10.709751 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c18e585d-a441-4ca7-b2b8-4342056c603f" containerName="cleanup" Oct 31 13:31:10.709943 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:10.709762 5693 cpu_manager.go:414] "containerMap: removing container" podUID="5ca294e6-ea27-4bc6-b710-60e6b27485ee" containerName="cleanup" Oct 31 13:31:10.709943 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:10.709766 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ca294e6-ea27-4bc6-b710-60e6b27485ee" containerName="cleanup" Oct 31 13:31:10.709943 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:10.709785 5693 cpu_manager.go:414] "containerMap: removing container" podUID="48d1b4b9-adf7-48d9-bbae-6830a2331143" containerName="cleanup" Oct 31 13:31:10.709943 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:10.709789 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48d1b4b9-adf7-48d9-bbae-6830a2331143" containerName="cleanup" Oct 31 13:31:10.709943 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:10.709802 5693 cpu_manager.go:414] "containerMap: removing container" podUID="92712c0d-d49c-46fe-8bad-a023a997d5f2" containerName="cleanup" Oct 31 13:31:10.709943 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:10.709806 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92712c0d-d49c-46fe-8bad-a023a997d5f2" containerName="cleanup" Oct 31 13:31:10.709943 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:10.709829 5693 cpu_manager.go:414] "containerMap: removing container" podUID="24816799-6034-48e6-9711-d9a342c0ef28" containerName="cleanup" Oct 31 13:31:10.709943 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:10.709833 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24816799-6034-48e6-9711-d9a342c0ef28" containerName="cleanup" Oct 31 13:31:10.710555 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:10.709981 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="b41505b3-033f-4911-b467-0b9d29530279" containerName="cleanup" Oct 31 13:31:10.710555 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:10.709994 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0fb78e4-7046-48f0-a737-4ca457074027" containerName="cleanup" Oct 31 13:31:10.710555 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:10.710001 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="c18e585d-a441-4ca7-b2b8-4342056c603f" containerName="cleanup" Oct 31 13:31:10.710555 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:10.710006 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="24816799-6034-48e6-9711-d9a342c0ef28" containerName="cleanup" Oct 31 13:31:10.710555 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:10.710012 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="48d1b4b9-adf7-48d9-bbae-6830a2331143" containerName="cleanup" Oct 31 13:31:10.710555 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:10.710021 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="1055f537-d4e4-46fc-b684-1ec68a9bda17" containerName="cleanup" Oct 31 13:31:10.710555 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:10.710026 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="c18e585d-a441-4ca7-b2b8-4342056c603f" containerName="cleanup" Oct 31 13:31:10.710555 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:10.710033 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="47872ba8-dd46-4d67-a513-e1081171e92e" containerName="cleanup" Oct 31 13:31:10.710555 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:10.710041 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="c18e585d-a441-4ca7-b2b8-4342056c603f" containerName="cleanup" Oct 31 13:31:10.710555 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:10.710048 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ca294e6-ea27-4bc6-b710-60e6b27485ee" containerName="cleanup" Oct 31 13:31:10.714213 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:10.714192 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61110dd0-xpjvk" Oct 31 13:31:10.721817 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:10.721787 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:31:10.722078 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:10.722057 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:31:10.723516 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:10.723486 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e9833c8-a068-4613-983b-de79cae7c50e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61110dd0-xpjvk\" (UID: \"4e9833c8-a068-4613-983b-de79cae7c50e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61110dd0-xpjvk" Oct 31 13:31:10.723660 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:10.723544 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg5ck\" (UniqueName: \"kubernetes.io/projected/4e9833c8-a068-4613-983b-de79cae7c50e-kube-api-access-vg5ck\") pod \"maas-api-revoke-sub-61110dd0-xpjvk\" (UID: \"4e9833c8-a068-4613-983b-de79cae7c50e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61110dd0-xpjvk" Oct 31 13:31:10.723760 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:10.723742 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e9833c8-a068-4613-983b-de79cae7c50e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61110dd0-xpjvk\" (UID: \"4e9833c8-a068-4613-983b-de79cae7c50e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61110dd0-xpjvk" Oct 31 13:31:10.725235 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:10.725214 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61110dd0-xpjvk"] Oct 31 13:31:10.825555 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:10.825521 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e9833c8-a068-4613-983b-de79cae7c50e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61110dd0-xpjvk\" (UID: \"4e9833c8-a068-4613-983b-de79cae7c50e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61110dd0-xpjvk" Oct 31 13:31:10.825785 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:10.825740 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e9833c8-a068-4613-983b-de79cae7c50e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61110dd0-xpjvk\" (UID: \"4e9833c8-a068-4613-983b-de79cae7c50e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61110dd0-xpjvk" Oct 31 13:31:10.825852 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:10.825793 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg5ck\" (UniqueName: \"kubernetes.io/projected/4e9833c8-a068-4613-983b-de79cae7c50e-kube-api-access-vg5ck\") pod \"maas-api-revoke-sub-61110dd0-xpjvk\" (UID: \"4e9833c8-a068-4613-983b-de79cae7c50e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61110dd0-xpjvk" Oct 31 13:31:10.826570 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:10.826545 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e9833c8-a068-4613-983b-de79cae7c50e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61110dd0-xpjvk\" (UID: \"4e9833c8-a068-4613-983b-de79cae7c50e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61110dd0-xpjvk" Oct 31 13:31:10.828678 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:10.828610 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e9833c8-a068-4613-983b-de79cae7c50e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61110dd0-xpjvk\" (UID: \"4e9833c8-a068-4613-983b-de79cae7c50e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61110dd0-xpjvk" Oct 31 13:31:10.850822 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:10.850762 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg5ck\" (UniqueName: \"kubernetes.io/projected/4e9833c8-a068-4613-983b-de79cae7c50e-kube-api-access-vg5ck\") pod \"maas-api-revoke-sub-61110dd0-xpjvk\" (UID: \"4e9833c8-a068-4613-983b-de79cae7c50e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61110dd0-xpjvk" Oct 31 13:31:10.884036 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:10.884003 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp"] Oct 31 13:31:10.884224 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:10.884204 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" podUID="95c1be59-5195-45f3-a004-820466d70099" containerName="istio-proxy" containerID="cri-o://a9cadc7cbdc2479d2c4971964da2417c7a8e0249068f4a966f678047d2457e2f" gracePeriod=30 Oct 31 13:31:11.039207 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:11.039112 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61110dd0-xpjvk" Oct 31 13:31:11.402557 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:31:11.402524 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e9833c8_a068_4613_983b_de79cae7c50e.slice/crio-31be20f8959b3d5f25f8c57b5d3e8c06f67a7f0425f5be5525f3f6cac0900cfb WatchSource:0}: Error finding container 31be20f8959b3d5f25f8c57b5d3e8c06f67a7f0425f5be5525f3f6cac0900cfb: Status 404 returned error can't find the container with id 31be20f8959b3d5f25f8c57b5d3e8c06f67a7f0425f5be5525f3f6cac0900cfb Oct 31 13:31:11.414458 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:11.414431 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61110dd0-xpjvk"] Oct 31 13:31:12.251304 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:12.251260 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61110dd0-xpjvk" event={"ID":"4e9833c8-a068-4613-983b-de79cae7c50e","Type":"ContainerStarted","Data":"3ac9f31da84d6a9c3f0b3ea588a14b5c9a79f7ba158b3c2dfc01428c2159080b"} Oct 31 13:31:12.251752 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:12.251355 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61110dd0-xpjvk" event={"ID":"4e9833c8-a068-4613-983b-de79cae7c50e","Type":"ContainerStarted","Data":"31be20f8959b3d5f25f8c57b5d3e8c06f67a7f0425f5be5525f3f6cac0900cfb"} Oct 31 13:31:12.279087 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:12.279037 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61110dd0-xpjvk" podStartSLOduration=2.279022914 podStartE2EDuration="2.279022914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:31:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:31:12.278306883 +0000 UTC m=+3522.985893179" watchObservedRunningTime="2026-10-31 13:31:12.279022914 +0000 UTC m=+3522.986609208" Oct 31 13:31:13.262773 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:13.262740 5693 generic.go:356] "Generic (PLEG): container finished" podID="4e9833c8-a068-4613-983b-de79cae7c50e" containerID="3ac9f31da84d6a9c3f0b3ea588a14b5c9a79f7ba158b3c2dfc01428c2159080b" exitCode=0 Oct 31 13:31:13.263261 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:13.262787 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61110dd0-xpjvk" event={"ID":"4e9833c8-a068-4613-983b-de79cae7c50e","Type":"ContainerDied","Data":"3ac9f31da84d6a9c3f0b3ea588a14b5c9a79f7ba158b3c2dfc01428c2159080b"} Oct 31 13:31:13.620763 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:13.620671 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv"] Oct 31 13:31:13.620932 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:13.620878 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" podUID="92d4bf25-3aad-42dc-989f-88c4d68dd188" containerName="istio-proxy" containerID="cri-o://e19fa931f2ee8d9f63c64b1975f3a3734222762613789ecd2a3f1c992042f8d6" gracePeriod=30 Oct 31 13:31:14.121459 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.121434 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" Oct 31 13:31:14.159969 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.159929 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-09db67-568998f649-d2ntj"] Oct 31 13:31:14.161495 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.161470 5693 cpu_manager.go:414] "containerMap: removing container" podUID="76570ffc-6a6b-4f4d-b8f0-0db5cde967c6" containerName="istio-proxy" Oct 31 13:31:14.161557 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.161501 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76570ffc-6a6b-4f4d-b8f0-0db5cde967c6" containerName="istio-proxy" Oct 31 13:31:14.161595 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.161589 5693 cpu_manager.go:414] "containerMap: removing container" podUID="c18e585d-a441-4ca7-b2b8-4342056c603f" containerName="cleanup" Oct 31 13:31:14.161652 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.161598 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c18e585d-a441-4ca7-b2b8-4342056c603f" containerName="cleanup" Oct 31 13:31:14.161859 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.161843 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="76570ffc-6a6b-4f4d-b8f0-0db5cde967c6" containerName="istio-proxy" Oct 31 13:31:14.161895 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.161881 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="76570ffc-6a6b-4f4d-b8f0-0db5cde967c6" containerName="istio-proxy" Oct 31 13:31:14.168919 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.168892 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3cddb3-6b9b59f449-7cfsr"] Oct 31 13:31:14.170402 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.170374 5693 cpu_manager.go:414] "containerMap: removing container" podUID="76570ffc-6a6b-4f4d-b8f0-0db5cde967c6" containerName="istio-proxy" Oct 31 13:31:14.170402 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.170398 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76570ffc-6a6b-4f4d-b8f0-0db5cde967c6" containerName="istio-proxy" Oct 31 13:31:14.170875 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.170851 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-09db67-568998f649-d2ntj" Oct 31 13:31:14.176789 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.176627 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-09db67-568998f649-d2ntj"] Oct 31 13:31:14.176976 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.176767 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3cddb3-6b9b59f449-7cfsr" Oct 31 13:31:14.178697 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.178621 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-09db67\"" Oct 31 13:31:14.179402 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.179352 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-09db67-dockercfg-xw8jw\"" Oct 31 13:31:14.180067 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.180045 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3cddb3-6b9b59f449-7cfsr"] Oct 31 13:31:14.180488 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.180468 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-3cddb3\"" Oct 31 13:31:14.180622 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.180602 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-3cddb3-dockercfg-4xlrb\"" Oct 31 13:31:14.251206 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.251117 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-istio-podinfo\") pod \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\" (UID: \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\") " Oct 31 13:31:14.251206 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.251163 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-credential-socket\") pod \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\" (UID: \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\") " Oct 31 13:31:14.251478 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.251236 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-istio-token\" (UniqueName: \"kubernetes.io/projected/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-istio-token\") pod \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\" (UID: \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\") " Oct 31 13:31:14.251478 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.251328 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-istio-data\") pod \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\" (UID: \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\") " Oct 31 13:31:14.251478 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.251364 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-workload-socket\") pod \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\" (UID: \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\") " Oct 31 13:31:14.251478 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.251439 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-istio-envoy\") pod \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\" (UID: \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\") " Oct 31 13:31:14.251742 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.251496 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-istiod-ca-cert\") pod \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\" (UID: \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\") " Oct 31 13:31:14.251742 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.251544 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-credential-socket" pod "76570ffc-6a6b-4f4d-b8f0-0db5cde967c6" (UID: "76570ffc-6a6b-4f4d-b8f0-0db5cde967c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:14.251869 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.251796 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-workload-certs\") pod \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\" (UID: \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\") " Oct 31 13:31:14.251927 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.251867 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-kube-api-access-pg9ws\" (UniqueName: \"kubernetes.io/projected/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-kube-api-access-pg9ws\") pod \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\" (UID: \"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6\") " Oct 31 13:31:14.251992 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.251798 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-workload-socket" pod "76570ffc-6a6b-4f4d-b8f0-0db5cde967c6" (UID: "76570ffc-6a6b-4f4d-b8f0-0db5cde967c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:14.252095 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.252071 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-istio-data" pod "76570ffc-6a6b-4f4d-b8f0-0db5cde967c6" (UID: "76570ffc-6a6b-4f4d-b8f0-0db5cde967c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:14.252162 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.252088 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-istiod-ca-cert" pod "76570ffc-6a6b-4f4d-b8f0-0db5cde967c6" (UID: "76570ffc-6a6b-4f4d-b8f0-0db5cde967c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:31:14.252162 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.252100 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-workload-certs" pod "76570ffc-6a6b-4f4d-b8f0-0db5cde967c6" (UID: "76570ffc-6a6b-4f4d-b8f0-0db5cde967c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:14.252416 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.252387 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gtfh\" (UniqueName: \"kubernetes.io/projected/d92607b7-b885-43e2-9694-7bcb94d4f6b5-kube-api-access-5gtfh\") pod \"payload-processing-e2e-inf-b-3cddb3-6b9b59f449-7cfsr\" (UID: \"d92607b7-b885-43e2-9694-7bcb94d4f6b5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3cddb3-6b9b59f449-7cfsr" Oct 31 13:31:14.252498 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.252486 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kjbp\" (UniqueName: \"kubernetes.io/projected/05078422-1fc4-4bf1-ab82-673c654c7b0a-kube-api-access-5kjbp\") pod \"payload-processing-e2e-inf-a-09db67-568998f649-d2ntj\" (UID: \"05078422-1fc4-4bf1-ab82-673c654c7b0a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-09db67-568998f649-d2ntj" Oct 31 13:31:14.252823 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.252731 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05078422-1fc4-4bf1-ab82-673c654c7b0a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-09db67-568998f649-d2ntj\" (UID: \"05078422-1fc4-4bf1-ab82-673c654c7b0a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-09db67-568998f649-d2ntj" Oct 31 13:31:14.253105 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.253077 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d92607b7-b885-43e2-9694-7bcb94d4f6b5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3cddb3-6b9b59f449-7cfsr\" (UID: \"d92607b7-b885-43e2-9694-7bcb94d4f6b5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3cddb3-6b9b59f449-7cfsr" Oct 31 13:31:14.253320 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.253299 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:14.253407 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.253328 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:14.253407 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.253375 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:14.253407 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.253392 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:14.253575 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.253406 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:14.254128 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.254105 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-istio-podinfo" pod "76570ffc-6a6b-4f4d-b8f0-0db5cde967c6" (UID: "76570ffc-6a6b-4f4d-b8f0-0db5cde967c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:31:14.254263 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.254242 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-istio-token" pod "76570ffc-6a6b-4f4d-b8f0-0db5cde967c6" (UID: "76570ffc-6a6b-4f4d-b8f0-0db5cde967c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:31:14.254497 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.254476 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-kube-api-access-pg9ws" pod "76570ffc-6a6b-4f4d-b8f0-0db5cde967c6" (UID: "76570ffc-6a6b-4f4d-b8f0-0db5cde967c6"). InnerVolumeSpecName "kube-api-access-pg9ws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:31:14.254674 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.254622 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-istio-envoy" pod "76570ffc-6a6b-4f4d-b8f0-0db5cde967c6" (UID: "76570ffc-6a6b-4f4d-b8f0-0db5cde967c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:14.277518 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.277468 5693 generic.go:356] "Generic (PLEG): container finished" podID="76570ffc-6a6b-4f4d-b8f0-0db5cde967c6" containerID="0e95c2e859ab2c48f3918e71e9bd286dba34af97ef1aa9c93f7738fff7d99044" exitCode=0 Oct 31 13:31:14.278059 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.278028 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" event={"ID":"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6","Type":"ContainerDied","Data":"0e95c2e859ab2c48f3918e71e9bd286dba34af97ef1aa9c93f7738fff7d99044"} Oct 31 13:31:14.278191 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.278061 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" Oct 31 13:31:14.278191 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.278086 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx" event={"ID":"76570ffc-6a6b-4f4d-b8f0-0db5cde967c6","Type":"ContainerDied","Data":"d84bae351fa140b4104960bc3a9c449179583aefac6fe9bc7251a4f3c19fe2d3"} Oct 31 13:31:14.278191 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.278115 5693 scope.go:122] "RemoveContainer" containerID="0e95c2e859ab2c48f3918e71e9bd286dba34af97ef1aa9c93f7738fff7d99044" Oct 31 13:31:14.383673 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.382750 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx"] Oct 31 13:31:14.394764 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.390689 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-bktnx"] Oct 31 13:31:14.394764 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.391616 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d92607b7-b885-43e2-9694-7bcb94d4f6b5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3cddb3-6b9b59f449-7cfsr\" (UID: \"d92607b7-b885-43e2-9694-7bcb94d4f6b5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3cddb3-6b9b59f449-7cfsr" Oct 31 13:31:14.394764 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.391852 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gtfh\" (UniqueName: \"kubernetes.io/projected/d92607b7-b885-43e2-9694-7bcb94d4f6b5-kube-api-access-5gtfh\") pod \"payload-processing-e2e-inf-b-3cddb3-6b9b59f449-7cfsr\" (UID: \"d92607b7-b885-43e2-9694-7bcb94d4f6b5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3cddb3-6b9b59f449-7cfsr" Oct 31 13:31:14.394764 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.391937 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kjbp\" (UniqueName: \"kubernetes.io/projected/05078422-1fc4-4bf1-ab82-673c654c7b0a-kube-api-access-5kjbp\") pod \"payload-processing-e2e-inf-a-09db67-568998f649-d2ntj\" (UID: \"05078422-1fc4-4bf1-ab82-673c654c7b0a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-09db67-568998f649-d2ntj" Oct 31 13:31:14.394764 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.392034 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05078422-1fc4-4bf1-ab82-673c654c7b0a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-09db67-568998f649-d2ntj\" (UID: \"05078422-1fc4-4bf1-ab82-673c654c7b0a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-09db67-568998f649-d2ntj" Oct 31 13:31:14.394764 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.392335 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg9ws\" (UniqueName: \"kubernetes.io/projected/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-kube-api-access-pg9ws\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:14.394764 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.392353 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:14.394764 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.392367 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:14.394764 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.392384 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:14.421135 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.421088 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d92607b7-b885-43e2-9694-7bcb94d4f6b5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3cddb3-6b9b59f449-7cfsr\" (UID: \"d92607b7-b885-43e2-9694-7bcb94d4f6b5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3cddb3-6b9b59f449-7cfsr" Oct 31 13:31:14.423366 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.423321 5693 scope.go:122] "RemoveContainer" containerID="f6f09ceee1023a3138095e8e21c5e30b90f34d0b64a389504ff470751a3e7e1c" Oct 31 13:31:14.430869 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.430679 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05078422-1fc4-4bf1-ab82-673c654c7b0a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-09db67-568998f649-d2ntj\" (UID: \"05078422-1fc4-4bf1-ab82-673c654c7b0a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-09db67-568998f649-d2ntj" Oct 31 13:31:14.442990 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.442864 5693 scope.go:122] "RemoveContainer" containerID="0e95c2e859ab2c48f3918e71e9bd286dba34af97ef1aa9c93f7738fff7d99044" Oct 31 13:31:14.443383 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:31:14.443354 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e95c2e859ab2c48f3918e71e9bd286dba34af97ef1aa9c93f7738fff7d99044\": container with ID starting with 0e95c2e859ab2c48f3918e71e9bd286dba34af97ef1aa9c93f7738fff7d99044 not found: ID does not exist" containerID="0e95c2e859ab2c48f3918e71e9bd286dba34af97ef1aa9c93f7738fff7d99044" Oct 31 13:31:14.443514 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.443397 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e95c2e859ab2c48f3918e71e9bd286dba34af97ef1aa9c93f7738fff7d99044"} err="failed to get container status \"0e95c2e859ab2c48f3918e71e9bd286dba34af97ef1aa9c93f7738fff7d99044\": rpc error: code = NotFound desc = could not find container \"0e95c2e859ab2c48f3918e71e9bd286dba34af97ef1aa9c93f7738fff7d99044\": container with ID starting with 0e95c2e859ab2c48f3918e71e9bd286dba34af97ef1aa9c93f7738fff7d99044 not found: ID does not exist" Oct 31 13:31:14.443514 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.443422 5693 scope.go:122] "RemoveContainer" containerID="f6f09ceee1023a3138095e8e21c5e30b90f34d0b64a389504ff470751a3e7e1c" Oct 31 13:31:14.443627 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.443510 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kjbp\" (UniqueName: \"kubernetes.io/projected/05078422-1fc4-4bf1-ab82-673c654c7b0a-kube-api-access-5kjbp\") pod \"payload-processing-e2e-inf-a-09db67-568998f649-d2ntj\" (UID: \"05078422-1fc4-4bf1-ab82-673c654c7b0a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-09db67-568998f649-d2ntj" Oct 31 13:31:14.443627 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.443540 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gtfh\" (UniqueName: \"kubernetes.io/projected/d92607b7-b885-43e2-9694-7bcb94d4f6b5-kube-api-access-5gtfh\") pod \"payload-processing-e2e-inf-b-3cddb3-6b9b59f449-7cfsr\" (UID: \"d92607b7-b885-43e2-9694-7bcb94d4f6b5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3cddb3-6b9b59f449-7cfsr" Oct 31 13:31:14.444089 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:31:14.444067 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6f09ceee1023a3138095e8e21c5e30b90f34d0b64a389504ff470751a3e7e1c\": container with ID starting with f6f09ceee1023a3138095e8e21c5e30b90f34d0b64a389504ff470751a3e7e1c not found: ID does not exist" containerID="f6f09ceee1023a3138095e8e21c5e30b90f34d0b64a389504ff470751a3e7e1c" Oct 31 13:31:14.444168 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.444095 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6f09ceee1023a3138095e8e21c5e30b90f34d0b64a389504ff470751a3e7e1c"} err="failed to get container status \"f6f09ceee1023a3138095e8e21c5e30b90f34d0b64a389504ff470751a3e7e1c\": rpc error: code = NotFound desc = could not find container \"f6f09ceee1023a3138095e8e21c5e30b90f34d0b64a389504ff470751a3e7e1c\": container with ID starting with f6f09ceee1023a3138095e8e21c5e30b90f34d0b64a389504ff470751a3e7e1c not found: ID does not exist" Oct 31 13:31:14.501931 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.498704 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-09db67-568998f649-d2ntj" Oct 31 13:31:14.528072 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.517328 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3cddb3-6b9b59f449-7cfsr" Oct 31 13:31:14.628545 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.628518 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61110dd0-xpjvk" Oct 31 13:31:14.677599 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.677570 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d0ce4a0-l5m59"] Oct 31 13:31:14.678609 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.678587 5693 cpu_manager.go:414] "containerMap: removing container" podUID="4e9833c8-a068-4613-983b-de79cae7c50e" containerName="revoke-keys" Oct 31 13:31:14.678609 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.678608 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e9833c8-a068-4613-983b-de79cae7c50e" containerName="revoke-keys" Oct 31 13:31:14.678862 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.678823 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e9833c8-a068-4613-983b-de79cae7c50e" containerName="revoke-keys" Oct 31 13:31:14.682979 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.682957 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0ce4a0-l5m59" Oct 31 13:31:14.698958 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.698930 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d0ce4a0-l5m59"] Oct 31 13:31:14.715051 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.715023 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e9833c8-a068-4613-983b-de79cae7c50e-kube-api-access-vg5ck\" (UniqueName: \"kubernetes.io/projected/4e9833c8-a068-4613-983b-de79cae7c50e-kube-api-access-vg5ck\") pod \"4e9833c8-a068-4613-983b-de79cae7c50e\" (UID: \"4e9833c8-a068-4613-983b-de79cae7c50e\") " Oct 31 13:31:14.715665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.715609 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e9833c8-a068-4613-983b-de79cae7c50e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e9833c8-a068-4613-983b-de79cae7c50e-maas-api-xks-ca\") pod \"4e9833c8-a068-4613-983b-de79cae7c50e\" (UID: \"4e9833c8-a068-4613-983b-de79cae7c50e\") " Oct 31 13:31:14.716017 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.715996 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e9833c8-a068-4613-983b-de79cae7c50e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e9833c8-a068-4613-983b-de79cae7c50e-maas-api-openshift-ca\") pod \"4e9833c8-a068-4613-983b-de79cae7c50e\" (UID: \"4e9833c8-a068-4613-983b-de79cae7c50e\") " Oct 31 13:31:14.717294 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.717271 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e9833c8-a068-4613-983b-de79cae7c50e-maas-api-openshift-ca" pod "4e9833c8-a068-4613-983b-de79cae7c50e" (UID: "4e9833c8-a068-4613-983b-de79cae7c50e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:31:14.724247 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.724224 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e9833c8-a068-4613-983b-de79cae7c50e-kube-api-access-vg5ck" pod "4e9833c8-a068-4613-983b-de79cae7c50e" (UID: "4e9833c8-a068-4613-983b-de79cae7c50e"). InnerVolumeSpecName "kube-api-access-vg5ck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:31:14.724344 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.724263 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e9833c8-a068-4613-983b-de79cae7c50e-maas-api-xks-ca" pod "4e9833c8-a068-4613-983b-de79cae7c50e" (UID: "4e9833c8-a068-4613-983b-de79cae7c50e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:31:14.834255 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.820370 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d0ce4a0-l5m59\" (UID: \"4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0ce4a0-l5m59" Oct 31 13:31:14.834255 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.820603 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d0ce4a0-l5m59\" (UID: \"4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0ce4a0-l5m59" Oct 31 13:31:14.834255 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.829160 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2j5j\" (UniqueName: \"kubernetes.io/projected/4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f-kube-api-access-g2j5j\") pod \"maas-api-revoke-sub-0d0ce4a0-l5m59\" (UID: \"4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0ce4a0-l5m59" Oct 31 13:31:14.834255 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.829494 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e9833c8-a068-4613-983b-de79cae7c50e-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:14.834255 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.829515 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e9833c8-a068-4613-983b-de79cae7c50e-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:14.834255 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.829529 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg5ck\" (UniqueName: \"kubernetes.io/projected/4e9833c8-a068-4613-983b-de79cae7c50e-kube-api-access-vg5ck\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:14.887028 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.886990 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-09db67-568998f649-d2ntj"] Oct 31 13:31:14.887532 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:31:14.887504 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05078422_1fc4_4bf1_ab82_673c654c7b0a.slice/crio-c770e29b68e4d829a2f99fda507fcff6787fc08d6fa11861a3d6fff5fd5ab621 WatchSource:0}: Error finding container c770e29b68e4d829a2f99fda507fcff6787fc08d6fa11861a3d6fff5fd5ab621: Status 404 returned error can't find the container with id c770e29b68e4d829a2f99fda507fcff6787fc08d6fa11861a3d6fff5fd5ab621 Oct 31 13:31:14.931940 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.931904 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2j5j\" (UniqueName: \"kubernetes.io/projected/4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f-kube-api-access-g2j5j\") pod \"maas-api-revoke-sub-0d0ce4a0-l5m59\" (UID: \"4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0ce4a0-l5m59" Oct 31 13:31:14.932184 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.932093 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d0ce4a0-l5m59\" (UID: \"4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0ce4a0-l5m59" Oct 31 13:31:14.932379 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.932247 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d0ce4a0-l5m59\" (UID: \"4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0ce4a0-l5m59" Oct 31 13:31:14.933699 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.933611 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d0ce4a0-l5m59\" (UID: \"4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0ce4a0-l5m59" Oct 31 13:31:14.936690 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.936617 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d0ce4a0-l5m59\" (UID: \"4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0ce4a0-l5m59" Oct 31 13:31:14.938469 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.938442 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3cddb3-6b9b59f449-7cfsr"] Oct 31 13:31:14.946024 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:14.945830 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2j5j\" (UniqueName: \"kubernetes.io/projected/4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f-kube-api-access-g2j5j\") pod \"maas-api-revoke-sub-0d0ce4a0-l5m59\" (UID: \"4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0ce4a0-l5m59" Oct 31 13:31:15.017688 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:15.017659 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0ce4a0-l5m59" Oct 31 13:31:15.290417 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:15.290375 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3cddb3-6b9b59f449-7cfsr" event={"ID":"d92607b7-b885-43e2-9694-7bcb94d4f6b5","Type":"ContainerStarted","Data":"c1a915dbcedb5f1aa3720a47b8409db69fe2a6b9e8b1502b760e0dd8fdc4015d"} Oct 31 13:31:15.291053 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:15.290455 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-3cddb3-6b9b59f449-7cfsr" Oct 31 13:31:15.291053 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:15.290473 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3cddb3-6b9b59f449-7cfsr" event={"ID":"d92607b7-b885-43e2-9694-7bcb94d4f6b5","Type":"ContainerStarted","Data":"4c553ac9d2b645f24b2c2f8dbfc6fa73d12ea4f926e51e1636dc836af8688744"} Oct 31 13:31:15.291998 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:15.291977 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61110dd0-xpjvk" Oct 31 13:31:15.292129 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:15.292015 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61110dd0-xpjvk" event={"ID":"4e9833c8-a068-4613-983b-de79cae7c50e","Type":"ContainerDied","Data":"31be20f8959b3d5f25f8c57b5d3e8c06f67a7f0425f5be5525f3f6cac0900cfb"} Oct 31 13:31:15.292129 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:15.292054 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31be20f8959b3d5f25f8c57b5d3e8c06f67a7f0425f5be5525f3f6cac0900cfb" Oct 31 13:31:15.294173 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:15.294132 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-09db67-568998f649-d2ntj" event={"ID":"05078422-1fc4-4bf1-ab82-673c654c7b0a","Type":"ContainerStarted","Data":"6a4017a99987b2be0fd7820ee3309640494b0062d34b6a41034473d1392d4040"} Oct 31 13:31:15.294332 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:15.294200 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-09db67-568998f649-d2ntj" event={"ID":"05078422-1fc4-4bf1-ab82-673c654c7b0a","Type":"ContainerStarted","Data":"c770e29b68e4d829a2f99fda507fcff6787fc08d6fa11861a3d6fff5fd5ab621"} Oct 31 13:31:15.294398 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:15.294364 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-09db67-568998f649-d2ntj" Oct 31 13:31:15.320329 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:31:15.318681 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d3d89e5_7c07_4a62_85d1_9fb51c6e1a3f.slice/crio-7cf985291d51f7f1ef78ad01a92997e568b2c8da6f78a9954c314c34f4852946 WatchSource:0}: Error finding container 7cf985291d51f7f1ef78ad01a92997e568b2c8da6f78a9954c314c34f4852946: Status 404 returned error can't find the container with id 7cf985291d51f7f1ef78ad01a92997e568b2c8da6f78a9954c314c34f4852946 Oct 31 13:31:15.320329 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:15.319362 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d0ce4a0-l5m59"] Oct 31 13:31:15.324328 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:15.324277 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-3cddb3-6b9b59f449-7cfsr" podStartSLOduration=30.324260312 podStartE2EDuration="30.324260312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:31:15.315070639 +0000 UTC m=+3526.022656934" watchObservedRunningTime="2026-10-31 13:31:15.324260312 +0000 UTC m=+3526.031846608" Oct 31 13:31:15.349630 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:15.349582 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-09db67-568998f649-d2ntj" podStartSLOduration=33.349562582 podStartE2EDuration="33.349562582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:31:15.345893514 +0000 UTC m=+3526.053479812" watchObservedRunningTime="2026-10-31 13:31:15.349562582 +0000 UTC m=+3526.057148882" Oct 31 13:31:15.413920 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:15.413878 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76570ffc-6a6b-4f4d-b8f0-0db5cde967c6" path="/var/lib/kubelet/pods/76570ffc-6a6b-4f4d-b8f0-0db5cde967c6/volumes" Oct 31 13:31:15.414338 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:15.414319 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q"] Oct 31 13:31:15.414565 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:15.414537 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" podUID="9e899259-b760-4d69-9576-aca3d72409c8" containerName="istio-proxy" containerID="cri-o://4a598835f59fe61969c5a52abe888aca1a182fb5397dc051c271da43d46bc52f" gracePeriod=30 Oct 31 13:31:16.207924 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.207898 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" Oct 31 13:31:16.244451 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.244421 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq"] Oct 31 13:31:16.245422 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.245404 5693 cpu_manager.go:414] "containerMap: removing container" podUID="95c1be59-5195-45f3-a004-820466d70099" containerName="istio-proxy" Oct 31 13:31:16.245422 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.245422 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95c1be59-5195-45f3-a004-820466d70099" containerName="istio-proxy" Oct 31 13:31:16.245593 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.245583 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="95c1be59-5195-45f3-a004-820466d70099" containerName="istio-proxy" Oct 31 13:31:16.250121 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.250102 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" Oct 31 13:31:16.285332 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.285292 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq"] Oct 31 13:31:16.301132 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.301096 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95c1be59-5195-45f3-a004-820466d70099-istio-token\" (UniqueName: \"kubernetes.io/projected/95c1be59-5195-45f3-a004-820466d70099-istio-token\") pod \"95c1be59-5195-45f3-a004-820466d70099\" (UID: \"95c1be59-5195-45f3-a004-820466d70099\") " Oct 31 13:31:16.301715 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.301247 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95c1be59-5195-45f3-a004-820466d70099-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c1be59-5195-45f3-a004-820466d70099-credential-socket\") pod \"95c1be59-5195-45f3-a004-820466d70099\" (UID: \"95c1be59-5195-45f3-a004-820466d70099\") " Oct 31 13:31:16.301715 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.301357 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95c1be59-5195-45f3-a004-820466d70099-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95c1be59-5195-45f3-a004-820466d70099-workload-certs\") pod \"95c1be59-5195-45f3-a004-820466d70099\" (UID: \"95c1be59-5195-45f3-a004-820466d70099\") " Oct 31 13:31:16.301715 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.301389 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95c1be59-5195-45f3-a004-820466d70099-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c1be59-5195-45f3-a004-820466d70099-workload-socket\") pod \"95c1be59-5195-45f3-a004-820466d70099\" (UID: \"95c1be59-5195-45f3-a004-820466d70099\") " Oct 31 13:31:16.301715 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.301500 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95c1be59-5195-45f3-a004-820466d70099-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95c1be59-5195-45f3-a004-820466d70099-istio-podinfo\") pod \"95c1be59-5195-45f3-a004-820466d70099\" (UID: \"95c1be59-5195-45f3-a004-820466d70099\") " Oct 31 13:31:16.301715 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.301538 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95c1be59-5195-45f3-a004-820466d70099-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95c1be59-5195-45f3-a004-820466d70099-istio-envoy\") pod \"95c1be59-5195-45f3-a004-820466d70099\" (UID: \"95c1be59-5195-45f3-a004-820466d70099\") " Oct 31 13:31:16.301715 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.301698 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95c1be59-5195-45f3-a004-820466d70099-kube-api-access-j5977\" (UniqueName: \"kubernetes.io/projected/95c1be59-5195-45f3-a004-820466d70099-kube-api-access-j5977\") pod \"95c1be59-5195-45f3-a004-820466d70099\" (UID: \"95c1be59-5195-45f3-a004-820466d70099\") " Oct 31 13:31:16.302158 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.301751 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95c1be59-5195-45f3-a004-820466d70099-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95c1be59-5195-45f3-a004-820466d70099-istiod-ca-cert\") pod \"95c1be59-5195-45f3-a004-820466d70099\" (UID: \"95c1be59-5195-45f3-a004-820466d70099\") " Oct 31 13:31:16.302158 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.301904 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95c1be59-5195-45f3-a004-820466d70099-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95c1be59-5195-45f3-a004-820466d70099-istio-data\") pod \"95c1be59-5195-45f3-a004-820466d70099\" (UID: \"95c1be59-5195-45f3-a004-820466d70099\") " Oct 31 13:31:16.303350 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.303320 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c1be59-5195-45f3-a004-820466d70099-workload-socket" pod "95c1be59-5195-45f3-a004-820466d70099" (UID: "95c1be59-5195-45f3-a004-820466d70099"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:16.303633 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.303609 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c1be59-5195-45f3-a004-820466d70099-workload-certs" pod "95c1be59-5195-45f3-a004-820466d70099" (UID: "95c1be59-5195-45f3-a004-820466d70099"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:16.303756 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.303679 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c1be59-5195-45f3-a004-820466d70099-credential-socket" pod "95c1be59-5195-45f3-a004-820466d70099" (UID: "95c1be59-5195-45f3-a004-820466d70099"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:16.303756 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.303725 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95c1be59-5195-45f3-a004-820466d70099-istiod-ca-cert" pod "95c1be59-5195-45f3-a004-820466d70099" (UID: "95c1be59-5195-45f3-a004-820466d70099"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:31:16.304010 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.303976 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c1be59-5195-45f3-a004-820466d70099-istio-data" pod "95c1be59-5195-45f3-a004-820466d70099" (UID: "95c1be59-5195-45f3-a004-820466d70099"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:16.304255 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.304221 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dd14658-e45a-468a-8691-97601b44e956-istio-envoy\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq\" (UID: \"9dd14658-e45a-468a-8691-97601b44e956\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" Oct 31 13:31:16.304388 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.304297 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvc5l\" (UniqueName: \"kubernetes.io/projected/9dd14658-e45a-468a-8691-97601b44e956-kube-api-access-qvc5l\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq\" (UID: \"9dd14658-e45a-468a-8691-97601b44e956\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" Oct 31 13:31:16.304388 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.304374 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dd14658-e45a-468a-8691-97601b44e956-credential-socket\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq\" (UID: \"9dd14658-e45a-468a-8691-97601b44e956\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" Oct 31 13:31:16.304632 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.304602 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dd14658-e45a-468a-8691-97601b44e956-istio-token\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq\" (UID: \"9dd14658-e45a-468a-8691-97601b44e956\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" Oct 31 13:31:16.304793 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.304777 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dd14658-e45a-468a-8691-97601b44e956-istiod-ca-cert\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq\" (UID: \"9dd14658-e45a-468a-8691-97601b44e956\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" Oct 31 13:31:16.304898 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.304885 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dd14658-e45a-468a-8691-97601b44e956-workload-socket\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq\" (UID: \"9dd14658-e45a-468a-8691-97601b44e956\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" Oct 31 13:31:16.305035 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.305019 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dd14658-e45a-468a-8691-97601b44e956-istio-podinfo\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq\" (UID: \"9dd14658-e45a-468a-8691-97601b44e956\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" Oct 31 13:31:16.305469 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.305366 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dd14658-e45a-468a-8691-97601b44e956-workload-certs\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq\" (UID: \"9dd14658-e45a-468a-8691-97601b44e956\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" Oct 31 13:31:16.305469 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.305443 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dd14658-e45a-468a-8691-97601b44e956-istio-data\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq\" (UID: \"9dd14658-e45a-468a-8691-97601b44e956\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" Oct 31 13:31:16.305743 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.305629 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c1be59-5195-45f3-a004-820466d70099-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:16.305743 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.305667 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95c1be59-5195-45f3-a004-820466d70099-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:16.305743 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.305682 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c1be59-5195-45f3-a004-820466d70099-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:16.305743 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.305696 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95c1be59-5195-45f3-a004-820466d70099-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:16.305743 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.305712 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95c1be59-5195-45f3-a004-820466d70099-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:16.310871 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.310059 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95c1be59-5195-45f3-a004-820466d70099-istio-podinfo" pod "95c1be59-5195-45f3-a004-820466d70099" (UID: "95c1be59-5195-45f3-a004-820466d70099"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:31:16.310871 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.310239 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95c1be59-5195-45f3-a004-820466d70099-kube-api-access-j5977" pod "95c1be59-5195-45f3-a004-820466d70099" (UID: "95c1be59-5195-45f3-a004-820466d70099"). InnerVolumeSpecName "kube-api-access-j5977". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:31:16.311396 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.311363 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95c1be59-5195-45f3-a004-820466d70099-istio-token" pod "95c1be59-5195-45f3-a004-820466d70099" (UID: "95c1be59-5195-45f3-a004-820466d70099"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:31:16.312018 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.311996 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c1be59-5195-45f3-a004-820466d70099-istio-envoy" pod "95c1be59-5195-45f3-a004-820466d70099" (UID: "95c1be59-5195-45f3-a004-820466d70099"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:16.317322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.317301 5693 generic.go:356] "Generic (PLEG): container finished" podID="95c1be59-5195-45f3-a004-820466d70099" containerID="a9cadc7cbdc2479d2c4971964da2417c7a8e0249068f4a966f678047d2457e2f" exitCode=0 Oct 31 13:31:16.317419 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.317403 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" Oct 31 13:31:16.317498 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.317436 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" event={"ID":"95c1be59-5195-45f3-a004-820466d70099","Type":"ContainerDied","Data":"a9cadc7cbdc2479d2c4971964da2417c7a8e0249068f4a966f678047d2457e2f"} Oct 31 13:31:16.317559 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.317506 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp" event={"ID":"95c1be59-5195-45f3-a004-820466d70099","Type":"ContainerDied","Data":"1a5dc0f3e42dbc4baf4359c5e011d1a7ce97af1163d88337f3496ad876ed6655"} Oct 31 13:31:16.317559 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.317539 5693 scope.go:122] "RemoveContainer" containerID="a9cadc7cbdc2479d2c4971964da2417c7a8e0249068f4a966f678047d2457e2f" Oct 31 13:31:16.324473 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.323179 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0ce4a0-l5m59" event={"ID":"4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f","Type":"ContainerStarted","Data":"20c347ba957c868d19521d8a6bb37cd0edc85b5fc4b5d5d52a891fb91a59138a"} Oct 31 13:31:16.324473 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.323240 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0ce4a0-l5m59" event={"ID":"4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f","Type":"ContainerStarted","Data":"7cf985291d51f7f1ef78ad01a92997e568b2c8da6f78a9954c314c34f4852946"} Oct 31 13:31:16.361380 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.361327 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0ce4a0-l5m59" podStartSLOduration=6.361312584 podStartE2EDuration="6.361312584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:31:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:31:16.355250949 +0000 UTC m=+3527.062837245" watchObservedRunningTime="2026-10-31 13:31:16.361312584 +0000 UTC m=+3527.068898878" Oct 31 13:31:16.371299 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.371264 5693 scope.go:122] "RemoveContainer" containerID="a9cadc7cbdc2479d2c4971964da2417c7a8e0249068f4a966f678047d2457e2f" Oct 31 13:31:16.371735 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:31:16.371711 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9cadc7cbdc2479d2c4971964da2417c7a8e0249068f4a966f678047d2457e2f\": container with ID starting with a9cadc7cbdc2479d2c4971964da2417c7a8e0249068f4a966f678047d2457e2f not found: ID does not exist" containerID="a9cadc7cbdc2479d2c4971964da2417c7a8e0249068f4a966f678047d2457e2f" Oct 31 13:31:16.371792 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.371743 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9cadc7cbdc2479d2c4971964da2417c7a8e0249068f4a966f678047d2457e2f"} err="failed to get container status \"a9cadc7cbdc2479d2c4971964da2417c7a8e0249068f4a966f678047d2457e2f\": rpc error: code = NotFound desc = could not find container \"a9cadc7cbdc2479d2c4971964da2417c7a8e0249068f4a966f678047d2457e2f\": container with ID starting with a9cadc7cbdc2479d2c4971964da2417c7a8e0249068f4a966f678047d2457e2f not found: ID does not exist" Oct 31 13:31:16.376747 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.376720 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp"] Oct 31 13:31:16.383353 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.383326 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-86jnp"] Oct 31 13:31:16.407343 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.407048 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dd14658-e45a-468a-8691-97601b44e956-workload-certs\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq\" (UID: \"9dd14658-e45a-468a-8691-97601b44e956\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" Oct 31 13:31:16.407460 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.407375 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dd14658-e45a-468a-8691-97601b44e956-istio-data\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq\" (UID: \"9dd14658-e45a-468a-8691-97601b44e956\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" Oct 31 13:31:16.407460 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.407439 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dd14658-e45a-468a-8691-97601b44e956-istio-envoy\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq\" (UID: \"9dd14658-e45a-468a-8691-97601b44e956\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" Oct 31 13:31:16.407541 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.407497 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvc5l\" (UniqueName: \"kubernetes.io/projected/9dd14658-e45a-468a-8691-97601b44e956-kube-api-access-qvc5l\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq\" (UID: \"9dd14658-e45a-468a-8691-97601b44e956\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" Oct 31 13:31:16.407586 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.407535 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dd14658-e45a-468a-8691-97601b44e956-workload-certs\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq\" (UID: \"9dd14658-e45a-468a-8691-97601b44e956\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" Oct 31 13:31:16.407696 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.407677 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dd14658-e45a-468a-8691-97601b44e956-istio-data\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq\" (UID: \"9dd14658-e45a-468a-8691-97601b44e956\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" Oct 31 13:31:16.407779 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.407698 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dd14658-e45a-468a-8691-97601b44e956-credential-socket\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq\" (UID: \"9dd14658-e45a-468a-8691-97601b44e956\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" Oct 31 13:31:16.407866 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.407844 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dd14658-e45a-468a-8691-97601b44e956-istio-token\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq\" (UID: \"9dd14658-e45a-468a-8691-97601b44e956\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" Oct 31 13:31:16.407941 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.407923 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dd14658-e45a-468a-8691-97601b44e956-istiod-ca-cert\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq\" (UID: \"9dd14658-e45a-468a-8691-97601b44e956\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" Oct 31 13:31:16.408128 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.407965 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dd14658-e45a-468a-8691-97601b44e956-credential-socket\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq\" (UID: \"9dd14658-e45a-468a-8691-97601b44e956\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" Oct 31 13:31:16.408128 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.408060 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dd14658-e45a-468a-8691-97601b44e956-workload-socket\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq\" (UID: \"9dd14658-e45a-468a-8691-97601b44e956\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" Oct 31 13:31:16.408338 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.408154 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dd14658-e45a-468a-8691-97601b44e956-istio-podinfo\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq\" (UID: \"9dd14658-e45a-468a-8691-97601b44e956\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" Oct 31 13:31:16.408338 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.408305 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95c1be59-5195-45f3-a004-820466d70099-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:16.408338 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.408325 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95c1be59-5195-45f3-a004-820466d70099-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:16.408532 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.408334 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dd14658-e45a-468a-8691-97601b44e956-workload-socket\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq\" (UID: \"9dd14658-e45a-468a-8691-97601b44e956\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" Oct 31 13:31:16.408532 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.408339 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5977\" (UniqueName: \"kubernetes.io/projected/95c1be59-5195-45f3-a004-820466d70099-kube-api-access-j5977\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:16.408532 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.408383 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95c1be59-5195-45f3-a004-820466d70099-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:16.408874 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.408851 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dd14658-e45a-468a-8691-97601b44e956-istiod-ca-cert\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq\" (UID: \"9dd14658-e45a-468a-8691-97601b44e956\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" Oct 31 13:31:16.410354 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.410331 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dd14658-e45a-468a-8691-97601b44e956-istio-envoy\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq\" (UID: \"9dd14658-e45a-468a-8691-97601b44e956\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" Oct 31 13:31:16.411035 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.411014 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dd14658-e45a-468a-8691-97601b44e956-istio-podinfo\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq\" (UID: \"9dd14658-e45a-468a-8691-97601b44e956\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" Oct 31 13:31:16.420674 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.420585 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dd14658-e45a-468a-8691-97601b44e956-istio-token\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq\" (UID: \"9dd14658-e45a-468a-8691-97601b44e956\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" Oct 31 13:31:16.420770 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.420696 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvc5l\" (UniqueName: \"kubernetes.io/projected/9dd14658-e45a-468a-8691-97601b44e956-kube-api-access-qvc5l\") pod \"e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq\" (UID: \"9dd14658-e45a-468a-8691-97601b44e956\") " pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" Oct 31 13:31:16.570571 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.570530 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" Oct 31 13:31:16.908838 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:31:16.908796 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dd14658_e45a_468a_8691_97601b44e956.slice/crio-104f227a6dd81721d2db6f145e9cf42b845e3af3e67e50e9c7416378f26a5b92 WatchSource:0}: Error finding container 104f227a6dd81721d2db6f145e9cf42b845e3af3e67e50e9c7416378f26a5b92: Status 404 returned error can't find the container with id 104f227a6dd81721d2db6f145e9cf42b845e3af3e67e50e9c7416378f26a5b92 Oct 31 13:31:16.910524 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.910497 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq"] Oct 31 13:31:16.912697 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.912657 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:31:16.912778 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.912731 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:31:16.912828 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:16.912775 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:31:17.337555 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:17.337452 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" event={"ID":"9dd14658-e45a-468a-8691-97601b44e956","Type":"ContainerStarted","Data":"6d376b44fe1f910d604f03fe7c25bc2db62dd356ea329885140ce915ec7adf5e"} Oct 31 13:31:17.337555 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:17.337519 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" event={"ID":"9dd14658-e45a-468a-8691-97601b44e956","Type":"ContainerStarted","Data":"104f227a6dd81721d2db6f145e9cf42b845e3af3e67e50e9c7416378f26a5b92"} Oct 31 13:31:17.339285 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:17.339260 5693 generic.go:356] "Generic (PLEG): container finished" podID="4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f" containerID="20c347ba957c868d19521d8a6bb37cd0edc85b5fc4b5d5d52a891fb91a59138a" exitCode=0 Oct 31 13:31:17.339411 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:17.339306 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0ce4a0-l5m59" event={"ID":"4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f","Type":"ContainerDied","Data":"20c347ba957c868d19521d8a6bb37cd0edc85b5fc4b5d5d52a891fb91a59138a"} Oct 31 13:31:17.369132 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:17.369067 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" podStartSLOduration=25.369049084 podStartE2EDuration="25.369049084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:31:17.365030108 +0000 UTC m=+3528.072616416" watchObservedRunningTime="2026-10-31 13:31:17.369049084 +0000 UTC m=+3528.076635379" Oct 31 13:31:17.416724 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:17.416685 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95c1be59-5195-45f3-a004-820466d70099" path="/var/lib/kubelet/pods/95c1be59-5195-45f3-a004-820466d70099/volumes" Oct 31 13:31:17.571631 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:17.571592 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" Oct 31 13:31:18.572624 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:18.572588 5693 patch_prober.go:29] interesting pod/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Oct 31 13:31:18.573027 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:18.572671 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" podUID="9dd14658-e45a-468a-8691-97601b44e956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Oct 31 13:31:18.713832 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:31:18.713795 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:31:18.725134 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:18.725108 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0ce4a0-l5m59" Oct 31 13:31:18.760473 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:18.760423 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f-maas-api-openshift-ca\") pod \"4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f\" (UID: \"4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f\") " Oct 31 13:31:18.796938 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:18.760892 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f-maas-api-openshift-ca" pod "4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f" (UID: "4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:31:18.797501 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:18.797467 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f-kube-api-access-g2j5j\" (UniqueName: \"kubernetes.io/projected/4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f-kube-api-access-g2j5j\") pod \"4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f\" (UID: \"4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f\") " Oct 31 13:31:18.797696 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:18.797677 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f-maas-api-xks-ca\") pod \"4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f\" (UID: \"4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f\") " Oct 31 13:31:18.804667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:18.798742 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:18.819433 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:18.819304 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f-maas-api-xks-ca" pod "4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f" (UID: "4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:31:18.822567 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:18.822539 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f-kube-api-access-g2j5j" pod "4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f" (UID: "4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f"). InnerVolumeSpecName "kube-api-access-g2j5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:31:18.907667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:18.906561 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2j5j\" (UniqueName: \"kubernetes.io/projected/4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f-kube-api-access-g2j5j\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:18.907667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:18.906600 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:19.014632 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.014185 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942"] Oct 31 13:31:19.014904 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.014870 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" podUID="530ebf58-0d39-42be-879f-3c16bdf11d58" containerName="istio-proxy" containerID="cri-o://93bfd3842572f8a090f70df8e792f2756cd560efbc47a45b955c8090efdba7fb" gracePeriod=30 Oct 31 13:31:19.026581 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.026460 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" Oct 31 13:31:19.117886 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.116216 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92d4bf25-3aad-42dc-989f-88c4d68dd188-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92d4bf25-3aad-42dc-989f-88c4d68dd188-istiod-ca-cert\") pod \"92d4bf25-3aad-42dc-989f-88c4d68dd188\" (UID: \"92d4bf25-3aad-42dc-989f-88c4d68dd188\") " Oct 31 13:31:19.117886 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.116327 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92d4bf25-3aad-42dc-989f-88c4d68dd188-kube-api-access-m7p9v\" (UniqueName: \"kubernetes.io/projected/92d4bf25-3aad-42dc-989f-88c4d68dd188-kube-api-access-m7p9v\") pod \"92d4bf25-3aad-42dc-989f-88c4d68dd188\" (UID: \"92d4bf25-3aad-42dc-989f-88c4d68dd188\") " Oct 31 13:31:19.117886 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.116437 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92d4bf25-3aad-42dc-989f-88c4d68dd188-istio-token\" (UniqueName: \"kubernetes.io/projected/92d4bf25-3aad-42dc-989f-88c4d68dd188-istio-token\") pod \"92d4bf25-3aad-42dc-989f-88c4d68dd188\" (UID: \"92d4bf25-3aad-42dc-989f-88c4d68dd188\") " Oct 31 13:31:19.117886 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.116474 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92d4bf25-3aad-42dc-989f-88c4d68dd188-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d4bf25-3aad-42dc-989f-88c4d68dd188-workload-socket\") pod \"92d4bf25-3aad-42dc-989f-88c4d68dd188\" (UID: \"92d4bf25-3aad-42dc-989f-88c4d68dd188\") " Oct 31 13:31:19.117886 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.116685 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92d4bf25-3aad-42dc-989f-88c4d68dd188-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92d4bf25-3aad-42dc-989f-88c4d68dd188-istio-envoy\") pod \"92d4bf25-3aad-42dc-989f-88c4d68dd188\" (UID: \"92d4bf25-3aad-42dc-989f-88c4d68dd188\") " Oct 31 13:31:19.117886 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.116795 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92d4bf25-3aad-42dc-989f-88c4d68dd188-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92d4bf25-3aad-42dc-989f-88c4d68dd188-workload-certs\") pod \"92d4bf25-3aad-42dc-989f-88c4d68dd188\" (UID: \"92d4bf25-3aad-42dc-989f-88c4d68dd188\") " Oct 31 13:31:19.117886 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.116856 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92d4bf25-3aad-42dc-989f-88c4d68dd188-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92d4bf25-3aad-42dc-989f-88c4d68dd188-istio-data\") pod \"92d4bf25-3aad-42dc-989f-88c4d68dd188\" (UID: \"92d4bf25-3aad-42dc-989f-88c4d68dd188\") " Oct 31 13:31:19.117886 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.116921 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92d4bf25-3aad-42dc-989f-88c4d68dd188-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d4bf25-3aad-42dc-989f-88c4d68dd188-credential-socket\") pod \"92d4bf25-3aad-42dc-989f-88c4d68dd188\" (UID: \"92d4bf25-3aad-42dc-989f-88c4d68dd188\") " Oct 31 13:31:19.117886 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.116950 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92d4bf25-3aad-42dc-989f-88c4d68dd188-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92d4bf25-3aad-42dc-989f-88c4d68dd188-istio-podinfo\") pod \"92d4bf25-3aad-42dc-989f-88c4d68dd188\" (UID: \"92d4bf25-3aad-42dc-989f-88c4d68dd188\") " Oct 31 13:31:19.118702 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.118386 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92d4bf25-3aad-42dc-989f-88c4d68dd188-workload-certs" pod "92d4bf25-3aad-42dc-989f-88c4d68dd188" (UID: "92d4bf25-3aad-42dc-989f-88c4d68dd188"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:19.119346 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.119204 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92d4bf25-3aad-42dc-989f-88c4d68dd188-istio-data" pod "92d4bf25-3aad-42dc-989f-88c4d68dd188" (UID: "92d4bf25-3aad-42dc-989f-88c4d68dd188"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:19.119979 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.119940 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92d4bf25-3aad-42dc-989f-88c4d68dd188-workload-socket" pod "92d4bf25-3aad-42dc-989f-88c4d68dd188" (UID: "92d4bf25-3aad-42dc-989f-88c4d68dd188"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:19.120682 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.120600 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92d4bf25-3aad-42dc-989f-88c4d68dd188-credential-socket" pod "92d4bf25-3aad-42dc-989f-88c4d68dd188" (UID: "92d4bf25-3aad-42dc-989f-88c4d68dd188"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:19.121016 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.120978 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92d4bf25-3aad-42dc-989f-88c4d68dd188-istiod-ca-cert" pod "92d4bf25-3aad-42dc-989f-88c4d68dd188" (UID: "92d4bf25-3aad-42dc-989f-88c4d68dd188"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:31:19.121120 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.121027 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-09db67-588458cc74-tg4cx"] Oct 31 13:31:19.122830 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.122786 5693 cpu_manager.go:414] "containerMap: removing container" podUID="4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f" containerName="revoke-keys" Oct 31 13:31:19.122830 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.122812 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f" containerName="revoke-keys" Oct 31 13:31:19.122981 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.122891 5693 cpu_manager.go:414] "containerMap: removing container" podUID="92d4bf25-3aad-42dc-989f-88c4d68dd188" containerName="istio-proxy" Oct 31 13:31:19.122981 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.122901 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92d4bf25-3aad-42dc-989f-88c4d68dd188" containerName="istio-proxy" Oct 31 13:31:19.123286 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.123264 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f" containerName="revoke-keys" Oct 31 13:31:19.123377 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.123303 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="92d4bf25-3aad-42dc-989f-88c4d68dd188" containerName="istio-proxy" Oct 31 13:31:19.127466 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.127419 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92d4bf25-3aad-42dc-989f-88c4d68dd188-kube-api-access-m7p9v" pod "92d4bf25-3aad-42dc-989f-88c4d68dd188" (UID: "92d4bf25-3aad-42dc-989f-88c4d68dd188"). InnerVolumeSpecName "kube-api-access-m7p9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:31:19.127713 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.127681 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92d4bf25-3aad-42dc-989f-88c4d68dd188-istio-podinfo" pod "92d4bf25-3aad-42dc-989f-88c4d68dd188" (UID: "92d4bf25-3aad-42dc-989f-88c4d68dd188"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:31:19.130066 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.128810 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92d4bf25-3aad-42dc-989f-88c4d68dd188-istio-envoy" pod "92d4bf25-3aad-42dc-989f-88c4d68dd188" (UID: "92d4bf25-3aad-42dc-989f-88c4d68dd188"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:19.130717 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.130545 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-09db67-588458cc74-tg4cx"] Oct 31 13:31:19.130717 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.130668 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09db67-588458cc74-tg4cx" Oct 31 13:31:19.132763 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.132730 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92d4bf25-3aad-42dc-989f-88c4d68dd188-istio-token" pod "92d4bf25-3aad-42dc-989f-88c4d68dd188" (UID: "92d4bf25-3aad-42dc-989f-88c4d68dd188"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:31:19.134738 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.134720 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-09db67-serving-cert\"" Oct 31 13:31:19.237671 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.225951 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77ca7503-e170-4c12-af28-8e78cce800a5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-09db67-588458cc74-tg4cx\" (UID: \"77ca7503-e170-4c12-af28-8e78cce800a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09db67-588458cc74-tg4cx" Oct 31 13:31:19.237671 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.226039 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rndvj\" (UniqueName: \"kubernetes.io/projected/77ca7503-e170-4c12-af28-8e78cce800a5-kube-api-access-rndvj\") pod \"maas-api-e2e-inf-a-09db67-588458cc74-tg4cx\" (UID: \"77ca7503-e170-4c12-af28-8e78cce800a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09db67-588458cc74-tg4cx" Oct 31 13:31:19.237671 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.226318 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77ca7503-e170-4c12-af28-8e78cce800a5-maas-api-tls\") pod \"maas-api-e2e-inf-a-09db67-588458cc74-tg4cx\" (UID: \"77ca7503-e170-4c12-af28-8e78cce800a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09db67-588458cc74-tg4cx" Oct 31 13:31:19.237671 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.226401 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77ca7503-e170-4c12-af28-8e78cce800a5-metrics-certs\") pod \"maas-api-e2e-inf-a-09db67-588458cc74-tg4cx\" (UID: \"77ca7503-e170-4c12-af28-8e78cce800a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09db67-588458cc74-tg4cx" Oct 31 13:31:19.237671 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.226541 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92d4bf25-3aad-42dc-989f-88c4d68dd188-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:19.237671 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.226563 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d4bf25-3aad-42dc-989f-88c4d68dd188-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:19.237671 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.226581 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92d4bf25-3aad-42dc-989f-88c4d68dd188-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:19.237671 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.226595 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92d4bf25-3aad-42dc-989f-88c4d68dd188-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:19.237671 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.226608 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7p9v\" (UniqueName: \"kubernetes.io/projected/92d4bf25-3aad-42dc-989f-88c4d68dd188-kube-api-access-m7p9v\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:19.237671 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.226622 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92d4bf25-3aad-42dc-989f-88c4d68dd188-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:19.237671 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.226658 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d4bf25-3aad-42dc-989f-88c4d68dd188-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:19.237671 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.226673 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92d4bf25-3aad-42dc-989f-88c4d68dd188-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:19.237671 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.226687 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92d4bf25-3aad-42dc-989f-88c4d68dd188-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:19.342105 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.342065 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77ca7503-e170-4c12-af28-8e78cce800a5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-09db67-588458cc74-tg4cx\" (UID: \"77ca7503-e170-4c12-af28-8e78cce800a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09db67-588458cc74-tg4cx" Oct 31 13:31:19.350021 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.349981 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rndvj\" (UniqueName: \"kubernetes.io/projected/77ca7503-e170-4c12-af28-8e78cce800a5-kube-api-access-rndvj\") pod \"maas-api-e2e-inf-a-09db67-588458cc74-tg4cx\" (UID: \"77ca7503-e170-4c12-af28-8e78cce800a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09db67-588458cc74-tg4cx" Oct 31 13:31:19.350688 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.350612 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77ca7503-e170-4c12-af28-8e78cce800a5-maas-api-tls\") pod \"maas-api-e2e-inf-a-09db67-588458cc74-tg4cx\" (UID: \"77ca7503-e170-4c12-af28-8e78cce800a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09db67-588458cc74-tg4cx" Oct 31 13:31:19.351057 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.351039 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77ca7503-e170-4c12-af28-8e78cce800a5-metrics-certs\") pod \"maas-api-e2e-inf-a-09db67-588458cc74-tg4cx\" (UID: \"77ca7503-e170-4c12-af28-8e78cce800a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09db67-588458cc74-tg4cx" Oct 31 13:31:19.352717 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.343833 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77ca7503-e170-4c12-af28-8e78cce800a5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-09db67-588458cc74-tg4cx\" (UID: \"77ca7503-e170-4c12-af28-8e78cce800a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09db67-588458cc74-tg4cx" Oct 31 13:31:19.367268 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.359305 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77ca7503-e170-4c12-af28-8e78cce800a5-metrics-certs\") pod \"maas-api-e2e-inf-a-09db67-588458cc74-tg4cx\" (UID: \"77ca7503-e170-4c12-af28-8e78cce800a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09db67-588458cc74-tg4cx" Oct 31 13:31:19.375022 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.371484 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77ca7503-e170-4c12-af28-8e78cce800a5-maas-api-tls\") pod \"maas-api-e2e-inf-a-09db67-588458cc74-tg4cx\" (UID: \"77ca7503-e170-4c12-af28-8e78cce800a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09db67-588458cc74-tg4cx" Oct 31 13:31:19.379826 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.379794 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rndvj\" (UniqueName: \"kubernetes.io/projected/77ca7503-e170-4c12-af28-8e78cce800a5-kube-api-access-rndvj\") pod \"maas-api-e2e-inf-a-09db67-588458cc74-tg4cx\" (UID: \"77ca7503-e170-4c12-af28-8e78cce800a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09db67-588458cc74-tg4cx" Oct 31 13:31:19.414129 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.414097 5693 generic.go:356] "Generic (PLEG): container finished" podID="92d4bf25-3aad-42dc-989f-88c4d68dd188" containerID="e19fa931f2ee8d9f63c64b1975f3a3734222762613789ecd2a3f1c992042f8d6" exitCode=0 Oct 31 13:31:19.414519 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.414497 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0ce4a0-l5m59" Oct 31 13:31:19.414905 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.414886 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" Oct 31 13:31:19.419051 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.419029 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0ce4a0-l5m59" event={"ID":"4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f","Type":"ContainerDied","Data":"7cf985291d51f7f1ef78ad01a92997e568b2c8da6f78a9954c314c34f4852946"} Oct 31 13:31:19.419184 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.419072 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cf985291d51f7f1ef78ad01a92997e568b2c8da6f78a9954c314c34f4852946" Oct 31 13:31:19.419184 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.419086 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" event={"ID":"92d4bf25-3aad-42dc-989f-88c4d68dd188","Type":"ContainerDied","Data":"e19fa931f2ee8d9f63c64b1975f3a3734222762613789ecd2a3f1c992042f8d6"} Oct 31 13:31:19.419184 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.419171 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv" event={"ID":"92d4bf25-3aad-42dc-989f-88c4d68dd188","Type":"ContainerDied","Data":"297dde0307fee3ffa3638a18393bf5b65864c6e11c92635dbd466d530d5d3c3f"} Oct 31 13:31:19.419366 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.419204 5693 scope.go:122] "RemoveContainer" containerID="e19fa931f2ee8d9f63c64b1975f3a3734222762613789ecd2a3f1c992042f8d6" Oct 31 13:31:19.436136 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.436114 5693 scope.go:122] "RemoveContainer" containerID="e19fa931f2ee8d9f63c64b1975f3a3734222762613789ecd2a3f1c992042f8d6" Oct 31 13:31:19.436432 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:31:19.436414 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e19fa931f2ee8d9f63c64b1975f3a3734222762613789ecd2a3f1c992042f8d6\": container with ID starting with e19fa931f2ee8d9f63c64b1975f3a3734222762613789ecd2a3f1c992042f8d6 not found: ID does not exist" containerID="e19fa931f2ee8d9f63c64b1975f3a3734222762613789ecd2a3f1c992042f8d6" Oct 31 13:31:19.436508 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.436440 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e19fa931f2ee8d9f63c64b1975f3a3734222762613789ecd2a3f1c992042f8d6"} err="failed to get container status \"e19fa931f2ee8d9f63c64b1975f3a3734222762613789ecd2a3f1c992042f8d6\": rpc error: code = NotFound desc = could not find container \"e19fa931f2ee8d9f63c64b1975f3a3734222762613789ecd2a3f1c992042f8d6\": container with ID starting with e19fa931f2ee8d9f63c64b1975f3a3734222762613789ecd2a3f1c992042f8d6 not found: ID does not exist" Oct 31 13:31:19.458551 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.458521 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv"] Oct 31 13:31:19.459291 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.459278 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09db67-588458cc74-tg4cx" Oct 31 13:31:19.463776 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.463749 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-7b2gv"] Oct 31 13:31:19.571841 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.571352 5693 patch_prober.go:29] interesting pod/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Oct 31 13:31:19.571841 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.571409 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" podUID="9dd14658-e45a-468a-8691-97601b44e956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Oct 31 13:31:19.809173 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.809141 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-09db67-588458cc74-tg4cx"] Oct 31 13:31:19.812782 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:31:19.812757 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77ca7503_e170_4c12_af28_8e78cce800a5.slice/crio-6646ea7c103d44f3564aca097817f2fe035bf251b31f92cab253607b3f65ed11 WatchSource:0}: Error finding container 6646ea7c103d44f3564aca097817f2fe035bf251b31f92cab253607b3f65ed11: Status 404 returned error can't find the container with id 6646ea7c103d44f3564aca097817f2fe035bf251b31f92cab253607b3f65ed11 Oct 31 13:31:19.814750 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:19.814728 5693 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 13:31:20.431034 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:20.431001 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09db67-588458cc74-tg4cx" event={"ID":"77ca7503-e170-4c12-af28-8e78cce800a5","Type":"ContainerStarted","Data":"74e8a5b1d3fcbceb69439d79e13a367f3628f6d1db5441a64900ddf2113b464e"} Oct 31 13:31:20.431258 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:20.431063 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09db67-588458cc74-tg4cx" event={"ID":"77ca7503-e170-4c12-af28-8e78cce800a5","Type":"ContainerStarted","Data":"6646ea7c103d44f3564aca097817f2fe035bf251b31f92cab253607b3f65ed11"} Oct 31 13:31:20.431258 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:20.431096 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09db67-588458cc74-tg4cx" Oct 31 13:31:20.461941 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:20.461893 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09db67-588458cc74-tg4cx" podStartSLOduration=9.974824089 podStartE2EDuration="10.461876501s" totalImagesPullingTime="487.052412ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:31:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:31:20.457312887 +0000 UTC m=+3531.164899175" watchObservedRunningTime="2026-10-31 13:31:20.461876501 +0000 UTC m=+3531.169462796" Oct 31 13:31:20.601716 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:20.582391 5693 patch_prober.go:29] interesting pod/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Oct 31 13:31:20.601716 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:20.582448 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" podUID="9dd14658-e45a-468a-8691-97601b44e956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Oct 31 13:31:20.934966 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:20.934930 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" Oct 31 13:31:21.034830 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.034042 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e899259-b760-4d69-9576-aca3d72409c8-istio-token\" (UniqueName: \"kubernetes.io/projected/9e899259-b760-4d69-9576-aca3d72409c8-istio-token\") pod \"9e899259-b760-4d69-9576-aca3d72409c8\" (UID: \"9e899259-b760-4d69-9576-aca3d72409c8\") " Oct 31 13:31:21.034830 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.034125 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e899259-b760-4d69-9576-aca3d72409c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e899259-b760-4d69-9576-aca3d72409c8-istio-envoy\") pod \"9e899259-b760-4d69-9576-aca3d72409c8\" (UID: \"9e899259-b760-4d69-9576-aca3d72409c8\") " Oct 31 13:31:21.034830 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.034257 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e899259-b760-4d69-9576-aca3d72409c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e899259-b760-4d69-9576-aca3d72409c8-credential-socket\") pod \"9e899259-b760-4d69-9576-aca3d72409c8\" (UID: \"9e899259-b760-4d69-9576-aca3d72409c8\") " Oct 31 13:31:21.034830 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.034300 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e899259-b760-4d69-9576-aca3d72409c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e899259-b760-4d69-9576-aca3d72409c8-istio-podinfo\") pod \"9e899259-b760-4d69-9576-aca3d72409c8\" (UID: \"9e899259-b760-4d69-9576-aca3d72409c8\") " Oct 31 13:31:21.034830 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.034362 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e899259-b760-4d69-9576-aca3d72409c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e899259-b760-4d69-9576-aca3d72409c8-workload-certs\") pod \"9e899259-b760-4d69-9576-aca3d72409c8\" (UID: \"9e899259-b760-4d69-9576-aca3d72409c8\") " Oct 31 13:31:21.034830 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.034406 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e899259-b760-4d69-9576-aca3d72409c8-kube-api-access-48l74\" (UniqueName: \"kubernetes.io/projected/9e899259-b760-4d69-9576-aca3d72409c8-kube-api-access-48l74\") pod \"9e899259-b760-4d69-9576-aca3d72409c8\" (UID: \"9e899259-b760-4d69-9576-aca3d72409c8\") " Oct 31 13:31:21.034830 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.034483 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e899259-b760-4d69-9576-aca3d72409c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e899259-b760-4d69-9576-aca3d72409c8-istiod-ca-cert\") pod \"9e899259-b760-4d69-9576-aca3d72409c8\" (UID: \"9e899259-b760-4d69-9576-aca3d72409c8\") " Oct 31 13:31:21.035491 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.035254 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e899259-b760-4d69-9576-aca3d72409c8-istiod-ca-cert" pod "9e899259-b760-4d69-9576-aca3d72409c8" (UID: "9e899259-b760-4d69-9576-aca3d72409c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:31:21.035659 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.034634 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e899259-b760-4d69-9576-aca3d72409c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e899259-b760-4d69-9576-aca3d72409c8-istio-data\") pod \"9e899259-b760-4d69-9576-aca3d72409c8\" (UID: \"9e899259-b760-4d69-9576-aca3d72409c8\") " Oct 31 13:31:21.035814 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.035621 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e899259-b760-4d69-9576-aca3d72409c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e899259-b760-4d69-9576-aca3d72409c8-workload-socket\") pod \"9e899259-b760-4d69-9576-aca3d72409c8\" (UID: \"9e899259-b760-4d69-9576-aca3d72409c8\") " Oct 31 13:31:21.035934 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.035898 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e899259-b760-4d69-9576-aca3d72409c8-credential-socket" pod "9e899259-b760-4d69-9576-aca3d72409c8" (UID: "9e899259-b760-4d69-9576-aca3d72409c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:21.036310 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.036137 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e899259-b760-4d69-9576-aca3d72409c8-istio-data" pod "9e899259-b760-4d69-9576-aca3d72409c8" (UID: "9e899259-b760-4d69-9576-aca3d72409c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:21.036880 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.036480 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e899259-b760-4d69-9576-aca3d72409c8-workload-socket" pod "9e899259-b760-4d69-9576-aca3d72409c8" (UID: "9e899259-b760-4d69-9576-aca3d72409c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:21.036985 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.036906 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e899259-b760-4d69-9576-aca3d72409c8-workload-certs" pod "9e899259-b760-4d69-9576-aca3d72409c8" (UID: "9e899259-b760-4d69-9576-aca3d72409c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:21.037346 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.037252 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e899259-b760-4d69-9576-aca3d72409c8-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:21.037346 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.037281 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e899259-b760-4d69-9576-aca3d72409c8-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:21.037346 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.037294 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e899259-b760-4d69-9576-aca3d72409c8-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:21.037346 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.037309 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e899259-b760-4d69-9576-aca3d72409c8-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:21.037346 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.037321 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e899259-b760-4d69-9576-aca3d72409c8-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:21.039730 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.038552 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e899259-b760-4d69-9576-aca3d72409c8-istio-token" pod "9e899259-b760-4d69-9576-aca3d72409c8" (UID: "9e899259-b760-4d69-9576-aca3d72409c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:31:21.043051 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.041165 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e899259-b760-4d69-9576-aca3d72409c8-istio-podinfo" pod "9e899259-b760-4d69-9576-aca3d72409c8" (UID: "9e899259-b760-4d69-9576-aca3d72409c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:31:21.043051 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.041467 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e899259-b760-4d69-9576-aca3d72409c8-kube-api-access-48l74" pod "9e899259-b760-4d69-9576-aca3d72409c8" (UID: "9e899259-b760-4d69-9576-aca3d72409c8"). InnerVolumeSpecName "kube-api-access-48l74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:31:21.043051 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.043018 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e899259-b760-4d69-9576-aca3d72409c8-istio-envoy" pod "9e899259-b760-4d69-9576-aca3d72409c8" (UID: "9e899259-b760-4d69-9576-aca3d72409c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:21.138798 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.138768 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e899259-b760-4d69-9576-aca3d72409c8-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:21.139009 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.138998 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e899259-b760-4d69-9576-aca3d72409c8-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:21.139083 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.139075 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e899259-b760-4d69-9576-aca3d72409c8-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:21.139163 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.139154 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48l74\" (UniqueName: \"kubernetes.io/projected/9e899259-b760-4d69-9576-aca3d72409c8-kube-api-access-48l74\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:21.416994 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.416959 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92d4bf25-3aad-42dc-989f-88c4d68dd188" path="/var/lib/kubelet/pods/92d4bf25-3aad-42dc-989f-88c4d68dd188/volumes" Oct 31 13:31:21.443041 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.443004 5693 generic.go:356] "Generic (PLEG): container finished" podID="9e899259-b760-4d69-9576-aca3d72409c8" containerID="4a598835f59fe61969c5a52abe888aca1a182fb5397dc051c271da43d46bc52f" exitCode=0 Oct 31 13:31:21.444869 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.444492 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" Oct 31 13:31:21.462664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.462146 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" event={"ID":"9e899259-b760-4d69-9576-aca3d72409c8","Type":"ContainerDied","Data":"4a598835f59fe61969c5a52abe888aca1a182fb5397dc051c271da43d46bc52f"} Oct 31 13:31:21.462664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.462228 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q" event={"ID":"9e899259-b760-4d69-9576-aca3d72409c8","Type":"ContainerDied","Data":"5246b83dc927ddd0aa68a2b5139cf12a4460640f5524d964dc312a707d74dcae"} Oct 31 13:31:21.462664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.462259 5693 scope.go:122] "RemoveContainer" containerID="4a598835f59fe61969c5a52abe888aca1a182fb5397dc051c271da43d46bc52f" Oct 31 13:31:21.554848 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.554800 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q"] Oct 31 13:31:21.562338 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.559713 5693 scope.go:122] "RemoveContainer" containerID="4a598835f59fe61969c5a52abe888aca1a182fb5397dc051c271da43d46bc52f" Oct 31 13:31:21.608885 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:31:21.569761 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a598835f59fe61969c5a52abe888aca1a182fb5397dc051c271da43d46bc52f\": container with ID starting with 4a598835f59fe61969c5a52abe888aca1a182fb5397dc051c271da43d46bc52f not found: ID does not exist" containerID="4a598835f59fe61969c5a52abe888aca1a182fb5397dc051c271da43d46bc52f" Oct 31 13:31:21.608885 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.569799 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a598835f59fe61969c5a52abe888aca1a182fb5397dc051c271da43d46bc52f"} err="failed to get container status \"4a598835f59fe61969c5a52abe888aca1a182fb5397dc051c271da43d46bc52f\": rpc error: code = NotFound desc = could not find container \"4a598835f59fe61969c5a52abe888aca1a182fb5397dc051c271da43d46bc52f\": container with ID starting with 4a598835f59fe61969c5a52abe888aca1a182fb5397dc051c271da43d46bc52f not found: ID does not exist" Oct 31 13:31:21.608885 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.571713 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-2fj6q"] Oct 31 13:31:21.608885 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.573275 5693 patch_prober.go:29] interesting pod/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Oct 31 13:31:21.608885 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.573321 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" podUID="9dd14658-e45a-468a-8691-97601b44e956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Oct 31 13:31:21.744722 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.736995 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg"] Oct 31 13:31:21.744722 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:21.737245 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" podUID="fbfd993e-8ff7-4646-a482-e71ab45f2c1a" containerName="istio-proxy" containerID="cri-o://086aac0f7add357ef9c84c7770f9f5b8429e00b748d272219bd3d42ec132cce2" gracePeriod=30 Oct 31 13:31:22.111942 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:22.111729 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d16d82aa-c7cdx"] Oct 31 13:31:22.114712 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:22.114687 5693 cpu_manager.go:414] "containerMap: removing container" podUID="9e899259-b760-4d69-9576-aca3d72409c8" containerName="istio-proxy" Oct 31 13:31:22.114911 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:22.114896 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e899259-b760-4d69-9576-aca3d72409c8" containerName="istio-proxy" Oct 31 13:31:22.115369 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:22.115349 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e899259-b760-4d69-9576-aca3d72409c8" containerName="istio-proxy" Oct 31 13:31:22.124986 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:22.124480 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d16d82aa-c7cdx"] Oct 31 13:31:22.124986 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:22.124596 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16d82aa-c7cdx" Oct 31 13:31:22.129910 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:22.129849 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:31:22.130781 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:22.130762 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:31:22.220412 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:22.220370 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db209c8e-2550-4233-9f7b-a7fdac69ba44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d16d82aa-c7cdx\" (UID: \"db209c8e-2550-4233-9f7b-a7fdac69ba44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16d82aa-c7cdx" Oct 31 13:31:22.220613 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:22.220482 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxpbd\" (UniqueName: \"kubernetes.io/projected/db209c8e-2550-4233-9f7b-a7fdac69ba44-kube-api-access-gxpbd\") pod \"maas-api-revoke-sub-d16d82aa-c7cdx\" (UID: \"db209c8e-2550-4233-9f7b-a7fdac69ba44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16d82aa-c7cdx" Oct 31 13:31:22.220613 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:22.220556 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db209c8e-2550-4233-9f7b-a7fdac69ba44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d16d82aa-c7cdx\" (UID: \"db209c8e-2550-4233-9f7b-a7fdac69ba44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16d82aa-c7cdx" Oct 31 13:31:22.323842 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:22.323804 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db209c8e-2550-4233-9f7b-a7fdac69ba44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d16d82aa-c7cdx\" (UID: \"db209c8e-2550-4233-9f7b-a7fdac69ba44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16d82aa-c7cdx" Oct 31 13:31:22.324034 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:22.323881 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxpbd\" (UniqueName: \"kubernetes.io/projected/db209c8e-2550-4233-9f7b-a7fdac69ba44-kube-api-access-gxpbd\") pod \"maas-api-revoke-sub-d16d82aa-c7cdx\" (UID: \"db209c8e-2550-4233-9f7b-a7fdac69ba44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16d82aa-c7cdx" Oct 31 13:31:22.324034 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:22.323924 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db209c8e-2550-4233-9f7b-a7fdac69ba44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d16d82aa-c7cdx\" (UID: \"db209c8e-2550-4233-9f7b-a7fdac69ba44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16d82aa-c7cdx" Oct 31 13:31:22.324597 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:22.324572 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db209c8e-2550-4233-9f7b-a7fdac69ba44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d16d82aa-c7cdx\" (UID: \"db209c8e-2550-4233-9f7b-a7fdac69ba44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16d82aa-c7cdx" Oct 31 13:31:22.326552 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:22.326526 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db209c8e-2550-4233-9f7b-a7fdac69ba44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d16d82aa-c7cdx\" (UID: \"db209c8e-2550-4233-9f7b-a7fdac69ba44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16d82aa-c7cdx" Oct 31 13:31:22.336414 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:22.336389 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxpbd\" (UniqueName: \"kubernetes.io/projected/db209c8e-2550-4233-9f7b-a7fdac69ba44-kube-api-access-gxpbd\") pod \"maas-api-revoke-sub-d16d82aa-c7cdx\" (UID: \"db209c8e-2550-4233-9f7b-a7fdac69ba44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16d82aa-c7cdx" Oct 31 13:31:22.447194 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:22.447167 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16d82aa-c7cdx" Oct 31 13:31:22.571491 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:22.571442 5693 patch_prober.go:29] interesting pod/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Oct 31 13:31:22.571693 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:22.571514 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" podUID="9dd14658-e45a-468a-8691-97601b44e956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Oct 31 13:31:22.858670 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:22.856064 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d16d82aa-c7cdx"] Oct 31 13:31:23.419879 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:23.419836 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e899259-b760-4d69-9576-aca3d72409c8" path="/var/lib/kubelet/pods/9e899259-b760-4d69-9576-aca3d72409c8/volumes" Oct 31 13:31:23.467921 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:23.467886 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16d82aa-c7cdx" event={"ID":"db209c8e-2550-4233-9f7b-a7fdac69ba44","Type":"ContainerStarted","Data":"d3a99eed7312d4a5bd339a5527d2c8297b4bc27dd81dc60f82ba847830cd91c1"} Oct 31 13:31:23.468121 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:23.467943 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16d82aa-c7cdx" event={"ID":"db209c8e-2550-4233-9f7b-a7fdac69ba44","Type":"ContainerStarted","Data":"7172bc7ed379464e505d4d5de2af9b7b3c2263552ea151641bc1547b2da1d64e"} Oct 31 13:31:23.489248 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:23.489183 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16d82aa-c7cdx" podStartSLOduration=1.489164759 podStartE2EDuration="1.489164759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:31:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:31:23.487571088 +0000 UTC m=+3534.195157419" watchObservedRunningTime="2026-10-31 13:31:23.489164759 +0000 UTC m=+3534.196751054" Oct 31 13:31:23.572117 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:23.572084 5693 patch_prober.go:29] interesting pod/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Oct 31 13:31:23.572284 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:23.572131 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" podUID="9dd14658-e45a-468a-8691-97601b44e956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Oct 31 13:31:24.354867 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.354837 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" Oct 31 13:31:24.474790 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.474761 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/530ebf58-0d39-42be-879f-3c16bdf11d58-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/530ebf58-0d39-42be-879f-3c16bdf11d58-workload-certs\") pod \"530ebf58-0d39-42be-879f-3c16bdf11d58\" (UID: \"530ebf58-0d39-42be-879f-3c16bdf11d58\") " Oct 31 13:31:24.475338 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.474843 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/530ebf58-0d39-42be-879f-3c16bdf11d58-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/530ebf58-0d39-42be-879f-3c16bdf11d58-credential-socket\") pod \"530ebf58-0d39-42be-879f-3c16bdf11d58\" (UID: \"530ebf58-0d39-42be-879f-3c16bdf11d58\") " Oct 31 13:31:24.475338 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.474867 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/530ebf58-0d39-42be-879f-3c16bdf11d58-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/530ebf58-0d39-42be-879f-3c16bdf11d58-workload-socket\") pod \"530ebf58-0d39-42be-879f-3c16bdf11d58\" (UID: \"530ebf58-0d39-42be-879f-3c16bdf11d58\") " Oct 31 13:31:24.475338 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.474935 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/530ebf58-0d39-42be-879f-3c16bdf11d58-kube-api-access-df5z2\" (UniqueName: \"kubernetes.io/projected/530ebf58-0d39-42be-879f-3c16bdf11d58-kube-api-access-df5z2\") pod \"530ebf58-0d39-42be-879f-3c16bdf11d58\" (UID: \"530ebf58-0d39-42be-879f-3c16bdf11d58\") " Oct 31 13:31:24.475338 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.474992 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/530ebf58-0d39-42be-879f-3c16bdf11d58-istio-token\" (UniqueName: \"kubernetes.io/projected/530ebf58-0d39-42be-879f-3c16bdf11d58-istio-token\") pod \"530ebf58-0d39-42be-879f-3c16bdf11d58\" (UID: \"530ebf58-0d39-42be-879f-3c16bdf11d58\") " Oct 31 13:31:24.475338 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.475066 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/530ebf58-0d39-42be-879f-3c16bdf11d58-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/530ebf58-0d39-42be-879f-3c16bdf11d58-istio-podinfo\") pod \"530ebf58-0d39-42be-879f-3c16bdf11d58\" (UID: \"530ebf58-0d39-42be-879f-3c16bdf11d58\") " Oct 31 13:31:24.475338 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.475119 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/530ebf58-0d39-42be-879f-3c16bdf11d58-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/530ebf58-0d39-42be-879f-3c16bdf11d58-istiod-ca-cert\") pod \"530ebf58-0d39-42be-879f-3c16bdf11d58\" (UID: \"530ebf58-0d39-42be-879f-3c16bdf11d58\") " Oct 31 13:31:24.475759 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.475110 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530ebf58-0d39-42be-879f-3c16bdf11d58-workload-certs" pod "530ebf58-0d39-42be-879f-3c16bdf11d58" (UID: "530ebf58-0d39-42be-879f-3c16bdf11d58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:24.475759 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.475232 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530ebf58-0d39-42be-879f-3c16bdf11d58-credential-socket" pod "530ebf58-0d39-42be-879f-3c16bdf11d58" (UID: "530ebf58-0d39-42be-879f-3c16bdf11d58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:24.475759 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.475397 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/530ebf58-0d39-42be-879f-3c16bdf11d58-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/530ebf58-0d39-42be-879f-3c16bdf11d58-istio-data\") pod \"530ebf58-0d39-42be-879f-3c16bdf11d58\" (UID: \"530ebf58-0d39-42be-879f-3c16bdf11d58\") " Oct 31 13:31:24.475759 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.475487 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/530ebf58-0d39-42be-879f-3c16bdf11d58-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/530ebf58-0d39-42be-879f-3c16bdf11d58-istio-envoy\") pod \"530ebf58-0d39-42be-879f-3c16bdf11d58\" (UID: \"530ebf58-0d39-42be-879f-3c16bdf11d58\") " Oct 31 13:31:24.475759 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.475596 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530ebf58-0d39-42be-879f-3c16bdf11d58-workload-socket" pod "530ebf58-0d39-42be-879f-3c16bdf11d58" (UID: "530ebf58-0d39-42be-879f-3c16bdf11d58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:24.475759 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.475605 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/530ebf58-0d39-42be-879f-3c16bdf11d58-istiod-ca-cert" pod "530ebf58-0d39-42be-879f-3c16bdf11d58" (UID: "530ebf58-0d39-42be-879f-3c16bdf11d58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:31:24.475759 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.475750 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530ebf58-0d39-42be-879f-3c16bdf11d58-istio-data" pod "530ebf58-0d39-42be-879f-3c16bdf11d58" (UID: "530ebf58-0d39-42be-879f-3c16bdf11d58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:24.477732 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.477707 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/530ebf58-0d39-42be-879f-3c16bdf11d58-istio-token" pod "530ebf58-0d39-42be-879f-3c16bdf11d58" (UID: "530ebf58-0d39-42be-879f-3c16bdf11d58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:31:24.478009 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.477981 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/530ebf58-0d39-42be-879f-3c16bdf11d58-istio-podinfo" pod "530ebf58-0d39-42be-879f-3c16bdf11d58" (UID: "530ebf58-0d39-42be-879f-3c16bdf11d58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:31:24.478260 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.478244 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/530ebf58-0d39-42be-879f-3c16bdf11d58-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:24.478527 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.478326 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/530ebf58-0d39-42be-879f-3c16bdf11d58-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:24.478527 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.478343 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/530ebf58-0d39-42be-879f-3c16bdf11d58-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:24.478527 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.478356 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/530ebf58-0d39-42be-879f-3c16bdf11d58-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:24.478527 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.478372 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/530ebf58-0d39-42be-879f-3c16bdf11d58-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:24.479838 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.479815 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530ebf58-0d39-42be-879f-3c16bdf11d58-istio-envoy" pod "530ebf58-0d39-42be-879f-3c16bdf11d58" (UID: "530ebf58-0d39-42be-879f-3c16bdf11d58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:24.480155 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.480137 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/530ebf58-0d39-42be-879f-3c16bdf11d58-kube-api-access-df5z2" pod "530ebf58-0d39-42be-879f-3c16bdf11d58" (UID: "530ebf58-0d39-42be-879f-3c16bdf11d58"). InnerVolumeSpecName "kube-api-access-df5z2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:31:24.483901 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.483874 5693 generic.go:356] "Generic (PLEG): container finished" podID="db209c8e-2550-4233-9f7b-a7fdac69ba44" containerID="d3a99eed7312d4a5bd339a5527d2c8297b4bc27dd81dc60f82ba847830cd91c1" exitCode=0 Oct 31 13:31:24.484013 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.483909 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16d82aa-c7cdx" event={"ID":"db209c8e-2550-4233-9f7b-a7fdac69ba44","Type":"ContainerDied","Data":"d3a99eed7312d4a5bd339a5527d2c8297b4bc27dd81dc60f82ba847830cd91c1"} Oct 31 13:31:24.485619 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.485596 5693 generic.go:356] "Generic (PLEG): container finished" podID="530ebf58-0d39-42be-879f-3c16bdf11d58" containerID="93bfd3842572f8a090f70df8e792f2756cd560efbc47a45b955c8090efdba7fb" exitCode=0 Oct 31 13:31:24.485746 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.485621 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" event={"ID":"530ebf58-0d39-42be-879f-3c16bdf11d58","Type":"ContainerDied","Data":"93bfd3842572f8a090f70df8e792f2756cd560efbc47a45b955c8090efdba7fb"} Oct 31 13:31:24.485746 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.485709 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" event={"ID":"530ebf58-0d39-42be-879f-3c16bdf11d58","Type":"ContainerDied","Data":"2141764fc369b3dc9c53e1be1364089692ab51f87a229b007f3e4858b4b160a6"} Oct 31 13:31:24.485746 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.485732 5693 scope.go:122] "RemoveContainer" containerID="93bfd3842572f8a090f70df8e792f2756cd560efbc47a45b955c8090efdba7fb" Oct 31 13:31:24.485883 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.485668 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942" Oct 31 13:31:24.522253 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.522177 5693 scope.go:122] "RemoveContainer" containerID="93bfd3842572f8a090f70df8e792f2756cd560efbc47a45b955c8090efdba7fb" Oct 31 13:31:24.522575 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:31:24.522551 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93bfd3842572f8a090f70df8e792f2756cd560efbc47a45b955c8090efdba7fb\": container with ID starting with 93bfd3842572f8a090f70df8e792f2756cd560efbc47a45b955c8090efdba7fb not found: ID does not exist" containerID="93bfd3842572f8a090f70df8e792f2756cd560efbc47a45b955c8090efdba7fb" Oct 31 13:31:24.522668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.522582 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93bfd3842572f8a090f70df8e792f2756cd560efbc47a45b955c8090efdba7fb"} err="failed to get container status \"93bfd3842572f8a090f70df8e792f2756cd560efbc47a45b955c8090efdba7fb\": rpc error: code = NotFound desc = could not find container \"93bfd3842572f8a090f70df8e792f2756cd560efbc47a45b955c8090efdba7fb\": container with ID starting with 93bfd3842572f8a090f70df8e792f2756cd560efbc47a45b955c8090efdba7fb not found: ID does not exist" Oct 31 13:31:24.540368 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.540336 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942"] Oct 31 13:31:24.545020 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.544991 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-bq942"] Oct 31 13:31:24.577411 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.577336 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" Oct 31 13:31:24.577828 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.577811 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" Oct 31 13:31:24.578577 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.578555 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" Oct 31 13:31:24.579510 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.579491 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df5z2\" (UniqueName: \"kubernetes.io/projected/530ebf58-0d39-42be-879f-3c16bdf11d58-kube-api-access-df5z2\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:24.579597 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.579515 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/530ebf58-0d39-42be-879f-3c16bdf11d58-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:24.579597 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.579532 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/530ebf58-0d39-42be-879f-3c16bdf11d58-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:24.579597 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:24.579555 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/530ebf58-0d39-42be-879f-3c16bdf11d58-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:25.418266 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:25.418229 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="530ebf58-0d39-42be-879f-3c16bdf11d58" path="/var/lib/kubelet/pods/530ebf58-0d39-42be-879f-3c16bdf11d58/volumes" Oct 31 13:31:25.685330 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:25.685304 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16d82aa-c7cdx" Oct 31 13:31:25.803829 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:25.803792 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db209c8e-2550-4233-9f7b-a7fdac69ba44-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db209c8e-2550-4233-9f7b-a7fdac69ba44-maas-api-openshift-ca\") pod \"db209c8e-2550-4233-9f7b-a7fdac69ba44\" (UID: \"db209c8e-2550-4233-9f7b-a7fdac69ba44\") " Oct 31 13:31:25.804035 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:25.803938 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db209c8e-2550-4233-9f7b-a7fdac69ba44-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db209c8e-2550-4233-9f7b-a7fdac69ba44-maas-api-xks-ca\") pod \"db209c8e-2550-4233-9f7b-a7fdac69ba44\" (UID: \"db209c8e-2550-4233-9f7b-a7fdac69ba44\") " Oct 31 13:31:25.804112 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:25.804056 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db209c8e-2550-4233-9f7b-a7fdac69ba44-kube-api-access-gxpbd\" (UniqueName: \"kubernetes.io/projected/db209c8e-2550-4233-9f7b-a7fdac69ba44-kube-api-access-gxpbd\") pod \"db209c8e-2550-4233-9f7b-a7fdac69ba44\" (UID: \"db209c8e-2550-4233-9f7b-a7fdac69ba44\") " Oct 31 13:31:25.804208 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:25.804185 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db209c8e-2550-4233-9f7b-a7fdac69ba44-maas-api-openshift-ca" pod "db209c8e-2550-4233-9f7b-a7fdac69ba44" (UID: "db209c8e-2550-4233-9f7b-a7fdac69ba44"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:31:25.805020 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:25.804997 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db209c8e-2550-4233-9f7b-a7fdac69ba44-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:25.806135 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:25.806106 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db209c8e-2550-4233-9f7b-a7fdac69ba44-maas-api-xks-ca" pod "db209c8e-2550-4233-9f7b-a7fdac69ba44" (UID: "db209c8e-2550-4233-9f7b-a7fdac69ba44"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:31:25.806317 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:25.806296 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db209c8e-2550-4233-9f7b-a7fdac69ba44-kube-api-access-gxpbd" pod "db209c8e-2550-4233-9f7b-a7fdac69ba44" (UID: "db209c8e-2550-4233-9f7b-a7fdac69ba44"). InnerVolumeSpecName "kube-api-access-gxpbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:31:25.906836 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:25.906794 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db209c8e-2550-4233-9f7b-a7fdac69ba44-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:25.906836 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:25.906829 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxpbd\" (UniqueName: \"kubernetes.io/projected/db209c8e-2550-4233-9f7b-a7fdac69ba44-kube-api-access-gxpbd\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:26.327221 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:26.327191 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-3cddb3-6b9b59f449-7cfsr" Oct 31 13:31:26.327675 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:26.327631 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-09db67-568998f649-d2ntj" Oct 31 13:31:26.452015 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:26.451889 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09db67-588458cc74-tg4cx" Oct 31 13:31:26.509131 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:26.509103 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16d82aa-c7cdx" Oct 31 13:31:26.509131 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:26.509109 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16d82aa-c7cdx" event={"ID":"db209c8e-2550-4233-9f7b-a7fdac69ba44","Type":"ContainerDied","Data":"7172bc7ed379464e505d4d5de2af9b7b3c2263552ea151641bc1547b2da1d64e"} Oct 31 13:31:26.509407 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:26.509167 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7172bc7ed379464e505d4d5de2af9b7b3c2263552ea151641bc1547b2da1d64e" Oct 31 13:31:27.523403 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:27.523369 5693 generic.go:356] "Generic (PLEG): container finished" podID="fbfd993e-8ff7-4646-a482-e71ab45f2c1a" containerID="086aac0f7add357ef9c84c7770f9f5b8429e00b748d272219bd3d42ec132cce2" exitCode=0 Oct 31 13:31:27.523803 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:27.523438 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" event={"ID":"fbfd993e-8ff7-4646-a482-e71ab45f2c1a","Type":"ContainerDied","Data":"086aac0f7add357ef9c84c7770f9f5b8429e00b748d272219bd3d42ec132cce2"} Oct 31 13:31:27.523803 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:27.523498 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" event={"ID":"fbfd993e-8ff7-4646-a482-e71ab45f2c1a","Type":"ContainerDied","Data":"c7d5d023a22be6941d3e5be2d36890bf13eb109d9a717b023541f42828586a44"} Oct 31 13:31:27.523803 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:27.523516 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7d5d023a22be6941d3e5be2d36890bf13eb109d9a717b023541f42828586a44" Oct 31 13:31:27.549568 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:27.549544 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" Oct 31 13:31:27.641543 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:27.641430 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-istio-data\") pod \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\" (UID: \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\") " Oct 31 13:31:27.641761 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:27.641542 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-workload-certs\") pod \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\" (UID: \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\") " Oct 31 13:31:27.641761 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:27.641714 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-credential-socket\") pod \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\" (UID: \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\") " Oct 31 13:31:27.641902 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:27.641820 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-istio-data" pod "fbfd993e-8ff7-4646-a482-e71ab45f2c1a" (UID: "fbfd993e-8ff7-4646-a482-e71ab45f2c1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:27.641902 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:27.641850 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-workload-certs" pod "fbfd993e-8ff7-4646-a482-e71ab45f2c1a" (UID: "fbfd993e-8ff7-4646-a482-e71ab45f2c1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:27.641902 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:27.641853 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-istio-token\" (UniqueName: \"kubernetes.io/projected/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-istio-token\") pod \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\" (UID: \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\") " Oct 31 13:31:27.642013 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:27.641999 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-istiod-ca-cert\") pod \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\" (UID: \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\") " Oct 31 13:31:27.642058 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:27.642035 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-credential-socket" pod "fbfd993e-8ff7-4646-a482-e71ab45f2c1a" (UID: "fbfd993e-8ff7-4646-a482-e71ab45f2c1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:27.642095 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:27.642059 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-kube-api-access-8dtb2\" (UniqueName: \"kubernetes.io/projected/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-kube-api-access-8dtb2\") pod \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\" (UID: \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\") " Oct 31 13:31:27.642142 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:27.642093 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-istio-podinfo\") pod \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\" (UID: \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\") " Oct 31 13:31:27.642183 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:27.642167 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-workload-socket\") pod \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\" (UID: \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\") " Oct 31 13:31:27.642325 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:27.642305 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-istiod-ca-cert" pod "fbfd993e-8ff7-4646-a482-e71ab45f2c1a" (UID: "fbfd993e-8ff7-4646-a482-e71ab45f2c1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:31:27.642692 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:27.642433 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-istio-envoy\") pod \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\" (UID: \"fbfd993e-8ff7-4646-a482-e71ab45f2c1a\") " Oct 31 13:31:27.642692 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:27.642514 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-workload-socket" pod "fbfd993e-8ff7-4646-a482-e71ab45f2c1a" (UID: "fbfd993e-8ff7-4646-a482-e71ab45f2c1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:27.643319 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:27.643293 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:27.643413 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:27.643325 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:27.643413 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:27.643343 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:27.643413 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:27.643355 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:27.643413 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:27.643369 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:27.644520 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:27.644492 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-kube-api-access-8dtb2" pod "fbfd993e-8ff7-4646-a482-e71ab45f2c1a" (UID: "fbfd993e-8ff7-4646-a482-e71ab45f2c1a"). InnerVolumeSpecName "kube-api-access-8dtb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:31:27.644834 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:27.644816 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-istio-token" pod "fbfd993e-8ff7-4646-a482-e71ab45f2c1a" (UID: "fbfd993e-8ff7-4646-a482-e71ab45f2c1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:31:27.645003 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:27.644981 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-istio-podinfo" pod "fbfd993e-8ff7-4646-a482-e71ab45f2c1a" (UID: "fbfd993e-8ff7-4646-a482-e71ab45f2c1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:31:27.645415 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:27.645391 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-istio-envoy" pod "fbfd993e-8ff7-4646-a482-e71ab45f2c1a" (UID: "fbfd993e-8ff7-4646-a482-e71ab45f2c1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:27.744932 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:27.744897 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:27.744932 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:27.744932 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dtb2\" (UniqueName: \"kubernetes.io/projected/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-kube-api-access-8dtb2\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:27.745156 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:27.744948 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:27.745156 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:27.744964 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbfd993e-8ff7-4646-a482-e71ab45f2c1a-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:28.534661 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:28.534621 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg" Oct 31 13:31:28.563856 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:28.563826 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg"] Oct 31 13:31:28.570667 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:28.570624 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vtqkg"] Oct 31 13:31:29.413404 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:29.413368 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbfd993e-8ff7-4646-a482-e71ab45f2c1a" path="/var/lib/kubelet/pods/fbfd993e-8ff7-4646-a482-e71ab45f2c1a/volumes" Oct 31 13:31:31.245335 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:31.245305 5693 scope.go:122] "RemoveContainer" containerID="086aac0f7add357ef9c84c7770f9f5b8429e00b748d272219bd3d42ec132cce2" Oct 31 13:31:31.306031 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:31.305999 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4f8865b-x64nl"] Oct 31 13:31:31.307506 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:31.307481 5693 cpu_manager.go:414] "containerMap: removing container" podUID="530ebf58-0d39-42be-879f-3c16bdf11d58" containerName="istio-proxy" Oct 31 13:31:31.307596 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:31.307507 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="530ebf58-0d39-42be-879f-3c16bdf11d58" containerName="istio-proxy" Oct 31 13:31:31.307596 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:31.307580 5693 cpu_manager.go:414] "containerMap: removing container" podUID="db209c8e-2550-4233-9f7b-a7fdac69ba44" containerName="revoke-keys" Oct 31 13:31:31.307596 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:31.307589 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db209c8e-2550-4233-9f7b-a7fdac69ba44" containerName="revoke-keys" Oct 31 13:31:31.307755 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:31.307665 5693 cpu_manager.go:414] "containerMap: removing container" podUID="fbfd993e-8ff7-4646-a482-e71ab45f2c1a" containerName="istio-proxy" Oct 31 13:31:31.307755 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:31.307673 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbfd993e-8ff7-4646-a482-e71ab45f2c1a" containerName="istio-proxy" Oct 31 13:31:31.307905 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:31.307890 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="db209c8e-2550-4233-9f7b-a7fdac69ba44" containerName="revoke-keys" Oct 31 13:31:31.307959 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:31.307915 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbfd993e-8ff7-4646-a482-e71ab45f2c1a" containerName="istio-proxy" Oct 31 13:31:31.307959 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:31.307929 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="530ebf58-0d39-42be-879f-3c16bdf11d58" containerName="istio-proxy" Oct 31 13:31:31.329702 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:31.324952 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4f8865b-x64nl"] Oct 31 13:31:31.329702 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:31.325070 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f8865b-x64nl" Oct 31 13:31:31.333751 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:31.333442 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:31:31.336419 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:31.334625 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:31:31.422625 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:31.422594 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6d9c5e7-3377-48a7-90af-bafebaf8c52a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4f8865b-x64nl\" (UID: \"b6d9c5e7-3377-48a7-90af-bafebaf8c52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f8865b-x64nl" Oct 31 13:31:31.422916 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:31.422894 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpl5v\" (UniqueName: \"kubernetes.io/projected/b6d9c5e7-3377-48a7-90af-bafebaf8c52a-kube-api-access-wpl5v\") pod \"maas-api-revoke-sub-e4f8865b-x64nl\" (UID: \"b6d9c5e7-3377-48a7-90af-bafebaf8c52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f8865b-x64nl" Oct 31 13:31:31.423152 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:31.423106 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6d9c5e7-3377-48a7-90af-bafebaf8c52a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4f8865b-x64nl\" (UID: \"b6d9c5e7-3377-48a7-90af-bafebaf8c52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f8865b-x64nl" Oct 31 13:31:31.524266 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:31.524184 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6d9c5e7-3377-48a7-90af-bafebaf8c52a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4f8865b-x64nl\" (UID: \"b6d9c5e7-3377-48a7-90af-bafebaf8c52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f8865b-x64nl" Oct 31 13:31:31.524428 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:31.524316 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6d9c5e7-3377-48a7-90af-bafebaf8c52a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4f8865b-x64nl\" (UID: \"b6d9c5e7-3377-48a7-90af-bafebaf8c52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f8865b-x64nl" Oct 31 13:31:31.524479 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:31.524425 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpl5v\" (UniqueName: \"kubernetes.io/projected/b6d9c5e7-3377-48a7-90af-bafebaf8c52a-kube-api-access-wpl5v\") pod \"maas-api-revoke-sub-e4f8865b-x64nl\" (UID: \"b6d9c5e7-3377-48a7-90af-bafebaf8c52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f8865b-x64nl" Oct 31 13:31:31.525037 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:31.525003 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6d9c5e7-3377-48a7-90af-bafebaf8c52a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4f8865b-x64nl\" (UID: \"b6d9c5e7-3377-48a7-90af-bafebaf8c52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f8865b-x64nl" Oct 31 13:31:31.527062 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:31.527040 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6d9c5e7-3377-48a7-90af-bafebaf8c52a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4f8865b-x64nl\" (UID: \"b6d9c5e7-3377-48a7-90af-bafebaf8c52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f8865b-x64nl" Oct 31 13:31:31.536909 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:31.536885 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpl5v\" (UniqueName: \"kubernetes.io/projected/b6d9c5e7-3377-48a7-90af-bafebaf8c52a-kube-api-access-wpl5v\") pod \"maas-api-revoke-sub-e4f8865b-x64nl\" (UID: \"b6d9c5e7-3377-48a7-90af-bafebaf8c52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f8865b-x64nl" Oct 31 13:31:31.660538 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:31.660500 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f8865b-x64nl" Oct 31 13:31:31.957149 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:31:31.957120 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6d9c5e7_3377_48a7_90af_bafebaf8c52a.slice/crio-f1b1e61b5e32d204978f942e6a98286e170beab54b3cab06d40d400067677eda WatchSource:0}: Error finding container f1b1e61b5e32d204978f942e6a98286e170beab54b3cab06d40d400067677eda: Status 404 returned error can't find the container with id f1b1e61b5e32d204978f942e6a98286e170beab54b3cab06d40d400067677eda Oct 31 13:31:31.959049 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:31.959027 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4f8865b-x64nl"] Oct 31 13:31:32.614270 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:32.614232 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f8865b-x64nl" event={"ID":"b6d9c5e7-3377-48a7-90af-bafebaf8c52a","Type":"ContainerStarted","Data":"d2dd6fd139a26292b848500cbbcf6fc34f5521b344e17e70f731d273dae272d3"} Oct 31 13:31:32.614773 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:32.614314 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f8865b-x64nl" event={"ID":"b6d9c5e7-3377-48a7-90af-bafebaf8c52a","Type":"ContainerStarted","Data":"f1b1e61b5e32d204978f942e6a98286e170beab54b3cab06d40d400067677eda"} Oct 31 13:31:32.635098 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:32.635044 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f8865b-x64nl" podStartSLOduration=1.635030244 podStartE2EDuration="1.635030244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:31:32.633031634 +0000 UTC m=+3543.340617930" watchObservedRunningTime="2026-10-31 13:31:32.635030244 +0000 UTC m=+3543.342616539" Oct 31 13:31:33.624199 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:33.624112 5693 generic.go:356] "Generic (PLEG): container finished" podID="b6d9c5e7-3377-48a7-90af-bafebaf8c52a" containerID="d2dd6fd139a26292b848500cbbcf6fc34f5521b344e17e70f731d273dae272d3" exitCode=0 Oct 31 13:31:33.624199 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:33.624165 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f8865b-x64nl" event={"ID":"b6d9c5e7-3377-48a7-90af-bafebaf8c52a","Type":"ContainerDied","Data":"d2dd6fd139a26292b848500cbbcf6fc34f5521b344e17e70f731d273dae272d3"} Oct 31 13:31:34.833551 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:34.833526 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f8865b-x64nl" Oct 31 13:31:34.880321 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:34.880291 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6d9c5e7-3377-48a7-90af-bafebaf8c52a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6d9c5e7-3377-48a7-90af-bafebaf8c52a-maas-api-openshift-ca\") pod \"b6d9c5e7-3377-48a7-90af-bafebaf8c52a\" (UID: \"b6d9c5e7-3377-48a7-90af-bafebaf8c52a\") " Oct 31 13:31:34.880488 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:34.880417 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6d9c5e7-3377-48a7-90af-bafebaf8c52a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6d9c5e7-3377-48a7-90af-bafebaf8c52a-maas-api-xks-ca\") pod \"b6d9c5e7-3377-48a7-90af-bafebaf8c52a\" (UID: \"b6d9c5e7-3377-48a7-90af-bafebaf8c52a\") " Oct 31 13:31:34.880573 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:34.880565 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6d9c5e7-3377-48a7-90af-bafebaf8c52a-kube-api-access-wpl5v\" (UniqueName: \"kubernetes.io/projected/b6d9c5e7-3377-48a7-90af-bafebaf8c52a-kube-api-access-wpl5v\") pod \"b6d9c5e7-3377-48a7-90af-bafebaf8c52a\" (UID: \"b6d9c5e7-3377-48a7-90af-bafebaf8c52a\") " Oct 31 13:31:34.881798 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:34.881768 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6d9c5e7-3377-48a7-90af-bafebaf8c52a-maas-api-openshift-ca" pod "b6d9c5e7-3377-48a7-90af-bafebaf8c52a" (UID: "b6d9c5e7-3377-48a7-90af-bafebaf8c52a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:31:34.883619 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:34.883595 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6d9c5e7-3377-48a7-90af-bafebaf8c52a-kube-api-access-wpl5v" pod "b6d9c5e7-3377-48a7-90af-bafebaf8c52a" (UID: "b6d9c5e7-3377-48a7-90af-bafebaf8c52a"). InnerVolumeSpecName "kube-api-access-wpl5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:31:34.884061 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:34.884007 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6d9c5e7-3377-48a7-90af-bafebaf8c52a-maas-api-xks-ca" pod "b6d9c5e7-3377-48a7-90af-bafebaf8c52a" (UID: "b6d9c5e7-3377-48a7-90af-bafebaf8c52a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:31:34.982936 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:34.982905 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6d9c5e7-3377-48a7-90af-bafebaf8c52a-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:34.982936 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:34.982932 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6d9c5e7-3377-48a7-90af-bafebaf8c52a-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:34.983211 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:34.982948 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpl5v\" (UniqueName: \"kubernetes.io/projected/b6d9c5e7-3377-48a7-90af-bafebaf8c52a-kube-api-access-wpl5v\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:35.644068 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:35.644034 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f8865b-x64nl" event={"ID":"b6d9c5e7-3377-48a7-90af-bafebaf8c52a","Type":"ContainerDied","Data":"f1b1e61b5e32d204978f942e6a98286e170beab54b3cab06d40d400067677eda"} Oct 31 13:31:35.644254 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:35.644093 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1b1e61b5e32d204978f942e6a98286e170beab54b3cab06d40d400067677eda" Oct 31 13:31:35.644254 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:35.644044 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f8865b-x64nl" Oct 31 13:31:43.807605 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:43.807560 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5dbd39-6b6f6d9559-9tbjq"] Oct 31 13:31:43.823239 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:43.823202 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5dbd39-5c6f7bcf67-d5m99"] Oct 31 13:31:45.391300 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:45.391268 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c6c445/test-model-c6c445-kserve-69b4dc6d69-nxx4n"] Oct 31 13:31:48.689523 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:31:48.689484 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:31:48.808728 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:48.808614 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-5dbd39-6b6f6d9559-9tbjq" podUID="b8730031-f623-4f9e-ba5f-014340976d85" containerName="payload-processing" containerID="cri-o://a16f39cf357902316f9b559f733bf6a67d98e9d8cee3ef73b846cc9b7c2fd486" gracePeriod=25 Oct 31 13:31:48.824150 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:48.824096 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5dbd39-5c6f7bcf67-d5m99" podUID="5cd6c998-975c-4628-a886-e9eeb20b3a64" containerName="payload-pre-processing" containerID="cri-o://49f0dc0eefa54aab9f5ecedd4715d6b9de50e2f38f03c4f1ce27d5f5149129c2" gracePeriod=25 Oct 31 13:31:51.459444 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:51.459404 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-5dbd39-6b6f6d9559-9tbjq" podUID="b8730031-f623-4f9e-ba5f-014340976d85" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:31:51.460100 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:51.459500 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5dbd39-5c6f7bcf67-d5m99" podUID="5cd6c998-975c-4628-a886-e9eeb20b3a64" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:31:53.659223 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:53.659183 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-50b4aa-58bd97d4c5-mgqbp"] Oct 31 13:31:53.660239 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:53.660219 5693 cpu_manager.go:414] "containerMap: removing container" podUID="b6d9c5e7-3377-48a7-90af-bafebaf8c52a" containerName="revoke-keys" Oct 31 13:31:53.660239 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:53.660239 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6d9c5e7-3377-48a7-90af-bafebaf8c52a" containerName="revoke-keys" Oct 31 13:31:53.660403 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:53.660394 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6d9c5e7-3377-48a7-90af-bafebaf8c52a" containerName="revoke-keys" Oct 31 13:31:53.666029 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:53.666006 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-50b4aa-58bd97d4c5-mgqbp" Oct 31 13:31:53.671871 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:53.671821 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-50b4aa\"" Oct 31 13:31:53.676004 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:53.675969 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-50b4aa-54d6cd77df-zsv8v"] Oct 31 13:31:53.676322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:53.676299 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-50b4aa-dockercfg-79rxt\"" Oct 31 13:31:53.687222 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:53.687197 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-50b4aa-58bd97d4c5-mgqbp"] Oct 31 13:31:53.687508 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:53.687492 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-50b4aa-54d6cd77df-zsv8v" Oct 31 13:31:53.701980 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:53.700867 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-50b4aa-54d6cd77df-zsv8v"] Oct 31 13:31:53.907279 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:53.907240 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d47d728a-192c-4622-99c9-e3f018a8cb8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-50b4aa-58bd97d4c5-mgqbp\" (UID: \"d47d728a-192c-4622-99c9-e3f018a8cb8f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-50b4aa-58bd97d4c5-mgqbp" Oct 31 13:31:53.909844 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:53.909763 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc8l8\" (UniqueName: \"kubernetes.io/projected/d47d728a-192c-4622-99c9-e3f018a8cb8f-kube-api-access-tc8l8\") pod \"payload-pre-processing-e2e-inf-b-50b4aa-58bd97d4c5-mgqbp\" (UID: \"d47d728a-192c-4622-99c9-e3f018a8cb8f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-50b4aa-58bd97d4c5-mgqbp" Oct 31 13:31:54.032331 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:54.032266 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrtl6\" (UniqueName: \"kubernetes.io/projected/6f93d32d-fa13-4c7b-a12f-10a709bed671-kube-api-access-rrtl6\") pod \"payload-processing-e2e-inf-b-50b4aa-54d6cd77df-zsv8v\" (UID: \"6f93d32d-fa13-4c7b-a12f-10a709bed671\") " pod="openshift-ingress/payload-processing-e2e-inf-b-50b4aa-54d6cd77df-zsv8v" Oct 31 13:31:54.037912 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:54.032669 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d47d728a-192c-4622-99c9-e3f018a8cb8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-50b4aa-58bd97d4c5-mgqbp\" (UID: \"d47d728a-192c-4622-99c9-e3f018a8cb8f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-50b4aa-58bd97d4c5-mgqbp" Oct 31 13:31:54.098919 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:54.046246 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d47d728a-192c-4622-99c9-e3f018a8cb8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-50b4aa-58bd97d4c5-mgqbp\" (UID: \"d47d728a-192c-4622-99c9-e3f018a8cb8f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-50b4aa-58bd97d4c5-mgqbp" Oct 31 13:31:54.102156 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:54.101978 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f93d32d-fa13-4c7b-a12f-10a709bed671-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-50b4aa-54d6cd77df-zsv8v\" (UID: \"6f93d32d-fa13-4c7b-a12f-10a709bed671\") " pod="openshift-ingress/payload-processing-e2e-inf-b-50b4aa-54d6cd77df-zsv8v" Oct 31 13:31:54.103088 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:54.103040 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc8l8\" (UniqueName: \"kubernetes.io/projected/d47d728a-192c-4622-99c9-e3f018a8cb8f-kube-api-access-tc8l8\") pod \"payload-pre-processing-e2e-inf-b-50b4aa-58bd97d4c5-mgqbp\" (UID: \"d47d728a-192c-4622-99c9-e3f018a8cb8f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-50b4aa-58bd97d4c5-mgqbp" Oct 31 13:31:54.149443 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:54.149409 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc8l8\" (UniqueName: \"kubernetes.io/projected/d47d728a-192c-4622-99c9-e3f018a8cb8f-kube-api-access-tc8l8\") pod \"payload-pre-processing-e2e-inf-b-50b4aa-58bd97d4c5-mgqbp\" (UID: \"d47d728a-192c-4622-99c9-e3f018a8cb8f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-50b4aa-58bd97d4c5-mgqbp" Oct 31 13:31:54.207698 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:54.207587 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrtl6\" (UniqueName: \"kubernetes.io/projected/6f93d32d-fa13-4c7b-a12f-10a709bed671-kube-api-access-rrtl6\") pod \"payload-processing-e2e-inf-b-50b4aa-54d6cd77df-zsv8v\" (UID: \"6f93d32d-fa13-4c7b-a12f-10a709bed671\") " pod="openshift-ingress/payload-processing-e2e-inf-b-50b4aa-54d6cd77df-zsv8v" Oct 31 13:31:54.213025 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:54.212998 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f93d32d-fa13-4c7b-a12f-10a709bed671-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-50b4aa-54d6cd77df-zsv8v\" (UID: \"6f93d32d-fa13-4c7b-a12f-10a709bed671\") " pod="openshift-ingress/payload-processing-e2e-inf-b-50b4aa-54d6cd77df-zsv8v" Oct 31 13:31:54.214494 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:54.214456 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f93d32d-fa13-4c7b-a12f-10a709bed671-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-50b4aa-54d6cd77df-zsv8v\" (UID: \"6f93d32d-fa13-4c7b-a12f-10a709bed671\") " pod="openshift-ingress/payload-processing-e2e-inf-b-50b4aa-54d6cd77df-zsv8v" Oct 31 13:31:54.239665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:54.238939 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrtl6\" (UniqueName: \"kubernetes.io/projected/6f93d32d-fa13-4c7b-a12f-10a709bed671-kube-api-access-rrtl6\") pod \"payload-processing-e2e-inf-b-50b4aa-54d6cd77df-zsv8v\" (UID: \"6f93d32d-fa13-4c7b-a12f-10a709bed671\") " pod="openshift-ingress/payload-processing-e2e-inf-b-50b4aa-54d6cd77df-zsv8v" Oct 31 13:31:54.291799 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:54.291752 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-50b4aa-58bd97d4c5-mgqbp" Oct 31 13:31:54.364494 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:54.364455 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4023ad-7994b685f7-9m2jh"] Oct 31 13:31:54.522722 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:54.522349 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-50b4aa-54d6cd77df-zsv8v" Oct 31 13:31:54.633360 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:54.631166 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-50b4aa-58bd97d4c5-mgqbp"] Oct 31 13:31:55.001578 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:55.001529 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-50b4aa-54d6cd77df-zsv8v"] Oct 31 13:31:55.002934 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:31:55.002904 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f93d32d_fa13_4c7b_a12f_10a709bed671.slice/crio-e17eaf6b2e3ffc37064d99dd04e4197080fc0e50b48c41910e3592eec646b1d8 WatchSource:0}: Error finding container e17eaf6b2e3ffc37064d99dd04e4197080fc0e50b48c41910e3592eec646b1d8: Status 404 returned error can't find the container with id e17eaf6b2e3ffc37064d99dd04e4197080fc0e50b48c41910e3592eec646b1d8 Oct 31 13:31:55.109749 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:55.109706 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-50b4aa-54d6cd77df-zsv8v" event={"ID":"6f93d32d-fa13-4c7b-a12f-10a709bed671","Type":"ContainerStarted","Data":"e17eaf6b2e3ffc37064d99dd04e4197080fc0e50b48c41910e3592eec646b1d8"} Oct 31 13:31:55.113880 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:55.113847 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-50b4aa-58bd97d4c5-mgqbp" event={"ID":"d47d728a-192c-4622-99c9-e3f018a8cb8f","Type":"ContainerStarted","Data":"32c4addea235dfe178b226bec4fbc220007b4e414a9f21834a7c9668c86c5e5a"} Oct 31 13:31:55.812666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:55.812606 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp"] Oct 31 13:31:55.814503 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:55.812923 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" podUID="c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6" containerName="istio-proxy" containerID="cri-o://f46697bff63215d4e7cca52a22073a800040001ac71ceb1296105962d6143ff5" gracePeriod=30 Oct 31 13:31:55.822371 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:55.820733 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg"] Oct 31 13:31:55.822371 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:55.821022 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" podUID="a9410570-a345-441e-9978-2f9ce48d7a09" containerName="istio-proxy" containerID="cri-o://557ce5b49869163d06f9cba2506416807d41c914db847c420e7ca9bb17aa44f7" gracePeriod=30 Oct 31 13:31:55.827873 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:55.827849 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv"] Oct 31 13:31:55.828205 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:55.828185 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" podUID="6791b24c-6569-46a0-af4c-3b7b7b356a9d" containerName="istio-proxy" containerID="cri-o://3390338bc12883f630fca0b2e5329f29428205b72b1d46e984c6ff4a5edbf0bb" gracePeriod=30 Oct 31 13:31:56.230760 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:56.230717 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-50b4aa-54d6cd77df-zsv8v" event={"ID":"6f93d32d-fa13-4c7b-a12f-10a709bed671","Type":"ContainerStarted","Data":"37b25fd236a25d96cf66cfe2a4021918d097dbfd516b8325c3c511956aaaeca2"} Oct 31 13:31:56.231246 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:56.230797 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-50b4aa-54d6cd77df-zsv8v" Oct 31 13:31:56.233014 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:56.232979 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-50b4aa-58bd97d4c5-mgqbp" event={"ID":"d47d728a-192c-4622-99c9-e3f018a8cb8f","Type":"ContainerStarted","Data":"2728d51cd12169f51c0533d81884e2d717f671b0200d72a3949fc471f8bcc891"} Oct 31 13:31:56.233163 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:56.233118 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-50b4aa-58bd97d4c5-mgqbp" Oct 31 13:31:56.258700 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:56.258617 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-50b4aa-54d6cd77df-zsv8v" podStartSLOduration=3.258603819 podStartE2EDuration="3.258603819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:31:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:31:56.253469072 +0000 UTC m=+3566.961055367" watchObservedRunningTime="2026-10-31 13:31:56.258603819 +0000 UTC m=+3566.966190141" Oct 31 13:31:56.274680 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:56.274567 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-50b4aa-58bd97d4c5-mgqbp" podStartSLOduration=3.274556155 podStartE2EDuration="3.274556155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:31:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:31:56.273941124 +0000 UTC m=+3566.981527419" watchObservedRunningTime="2026-10-31 13:31:56.274556155 +0000 UTC m=+3566.982142449" Oct 31 13:31:59.365315 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:31:59.365255 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-4023ad-7994b685f7-9m2jh" podUID="bc39dc34-d1e4-466d-bc1d-5e77bac8dc39" containerName="payload-processing" containerID="cri-o://14d06494c6a1c4e9297ea00ecd34d28a7928cd274fbe50098db8b70b9853bd05" gracePeriod=25 Oct 31 13:32:00.428830 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:00.428767 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-c6c445/test-model-c6c445-kserve-69b4dc6d69-nxx4n" podUID="768af44a-2a4a-4254-a497-e2969f376cf7" containerName="main" containerID="cri-o://d0d1a27779f09a735fb98bffbaf9b0bd090b5c660136b3f9421f98598e5db960" gracePeriod=45 Oct 31 13:32:01.096926 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.096575 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c6c445/test-model-c6c445-kserve-69b4dc6d69-nxx4n" Oct 31 13:32:01.206066 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.206034 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/768af44a-2a4a-4254-a497-e2969f376cf7-kube-api-access-46dtx\" (UniqueName: \"kubernetes.io/projected/768af44a-2a4a-4254-a497-e2969f376cf7-kube-api-access-46dtx\") pod \"768af44a-2a4a-4254-a497-e2969f376cf7\" (UID: \"768af44a-2a4a-4254-a497-e2969f376cf7\") " Oct 31 13:32:01.206161 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.206091 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/768af44a-2a4a-4254-a497-e2969f376cf7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/768af44a-2a4a-4254-a497-e2969f376cf7-dshm\") pod \"768af44a-2a4a-4254-a497-e2969f376cf7\" (UID: \"768af44a-2a4a-4254-a497-e2969f376cf7\") " Oct 31 13:32:01.206161 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.206120 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/768af44a-2a4a-4254-a497-e2969f376cf7-home\" (UniqueName: \"kubernetes.io/empty-dir/768af44a-2a4a-4254-a497-e2969f376cf7-home\") pod \"768af44a-2a4a-4254-a497-e2969f376cf7\" (UID: \"768af44a-2a4a-4254-a497-e2969f376cf7\") " Oct 31 13:32:01.206337 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.206225 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/768af44a-2a4a-4254-a497-e2969f376cf7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/768af44a-2a4a-4254-a497-e2969f376cf7-tmp-dir\") pod \"768af44a-2a4a-4254-a497-e2969f376cf7\" (UID: \"768af44a-2a4a-4254-a497-e2969f376cf7\") " Oct 31 13:32:01.206451 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.206370 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/768af44a-2a4a-4254-a497-e2969f376cf7-tls-certs\" (UniqueName: \"kubernetes.io/secret/768af44a-2a4a-4254-a497-e2969f376cf7-tls-certs\") pod \"768af44a-2a4a-4254-a497-e2969f376cf7\" (UID: \"768af44a-2a4a-4254-a497-e2969f376cf7\") " Oct 31 13:32:01.206451 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.206437 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/768af44a-2a4a-4254-a497-e2969f376cf7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/768af44a-2a4a-4254-a497-e2969f376cf7-model-cache\") pod \"768af44a-2a4a-4254-a497-e2969f376cf7\" (UID: \"768af44a-2a4a-4254-a497-e2969f376cf7\") " Oct 31 13:32:01.207788 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.207744 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/768af44a-2a4a-4254-a497-e2969f376cf7-model-cache" pod "768af44a-2a4a-4254-a497-e2969f376cf7" (UID: "768af44a-2a4a-4254-a497-e2969f376cf7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:01.207896 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.207780 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/768af44a-2a4a-4254-a497-e2969f376cf7-home" pod "768af44a-2a4a-4254-a497-e2969f376cf7" (UID: "768af44a-2a4a-4254-a497-e2969f376cf7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:01.207896 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.207848 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/768af44a-2a4a-4254-a497-e2969f376cf7-tmp-dir" pod "768af44a-2a4a-4254-a497-e2969f376cf7" (UID: "768af44a-2a4a-4254-a497-e2969f376cf7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:01.210962 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.210931 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/768af44a-2a4a-4254-a497-e2969f376cf7-dshm" pod "768af44a-2a4a-4254-a497-e2969f376cf7" (UID: "768af44a-2a4a-4254-a497-e2969f376cf7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:01.211343 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.211316 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/768af44a-2a4a-4254-a497-e2969f376cf7-tls-certs" pod "768af44a-2a4a-4254-a497-e2969f376cf7" (UID: "768af44a-2a4a-4254-a497-e2969f376cf7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:32:01.212338 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.212308 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/768af44a-2a4a-4254-a497-e2969f376cf7-kube-api-access-46dtx" pod "768af44a-2a4a-4254-a497-e2969f376cf7" (UID: "768af44a-2a4a-4254-a497-e2969f376cf7"). InnerVolumeSpecName "kube-api-access-46dtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:32:01.235514 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.235461 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" Oct 31 13:32:01.308394 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.308361 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-workload-socket\") pod \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\" (UID: \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\") " Oct 31 13:32:01.308957 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.308929 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-workload-certs\") pod \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\" (UID: \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\") " Oct 31 13:32:01.309096 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.309058 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-istio-envoy\") pod \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\" (UID: \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\") " Oct 31 13:32:01.309162 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.309098 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-istio-data\") pod \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\" (UID: \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\") " Oct 31 13:32:01.309218 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.309170 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-istiod-ca-cert\") pod \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\" (UID: \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\") " Oct 31 13:32:01.309283 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.309237 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-credential-socket\") pod \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\" (UID: \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\") " Oct 31 13:32:01.309283 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.309275 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-kube-api-access-wktnv\" (UniqueName: \"kubernetes.io/projected/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-kube-api-access-wktnv\") pod \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\" (UID: \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\") " Oct 31 13:32:01.309421 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.309384 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-istio-token\" (UniqueName: \"kubernetes.io/projected/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-istio-token\") pod \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\" (UID: \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\") " Oct 31 13:32:01.309600 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.309582 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-istio-podinfo\") pod \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\" (UID: \"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6\") " Oct 31 13:32:01.310579 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.310558 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46dtx\" (UniqueName: \"kubernetes.io/projected/768af44a-2a4a-4254-a497-e2969f376cf7-kube-api-access-46dtx\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.310731 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.310720 5693 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/768af44a-2a4a-4254-a497-e2969f376cf7-dshm\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.310820 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.310811 5693 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/768af44a-2a4a-4254-a497-e2969f376cf7-home\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.310902 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.310892 5693 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/768af44a-2a4a-4254-a497-e2969f376cf7-tmp-dir\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.310994 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.310983 5693 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/768af44a-2a4a-4254-a497-e2969f376cf7-tls-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.311094 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.311080 5693 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/768af44a-2a4a-4254-a497-e2969f376cf7-model-cache\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.312147 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.312118 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-workload-certs" pod "c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6" (UID: "c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:01.312508 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.312326 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-workload-socket" pod "c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6" (UID: "c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:01.314551 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.314522 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-istiod-ca-cert" pod "c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6" (UID: "c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:32:01.314880 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.314854 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-credential-socket" pod "c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6" (UID: "c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:01.314972 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.314954 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-istio-envoy" pod "c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6" (UID: "c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:01.316163 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.316136 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-istio-data" pod "c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6" (UID: "c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:01.317760 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.317728 5693 generic.go:356] "Generic (PLEG): container finished" podID="a9410570-a345-441e-9978-2f9ce48d7a09" containerID="557ce5b49869163d06f9cba2506416807d41c914db847c420e7ca9bb17aa44f7" exitCode=0 Oct 31 13:32:01.318074 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.318048 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" event={"ID":"a9410570-a345-441e-9978-2f9ce48d7a09","Type":"ContainerDied","Data":"557ce5b49869163d06f9cba2506416807d41c914db847c420e7ca9bb17aa44f7"} Oct 31 13:32:01.324349 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.324320 5693 generic.go:356] "Generic (PLEG): container finished" podID="c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6" containerID="f46697bff63215d4e7cca52a22073a800040001ac71ceb1296105962d6143ff5" exitCode=0 Oct 31 13:32:01.324473 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.324414 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" event={"ID":"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6","Type":"ContainerDied","Data":"f46697bff63215d4e7cca52a22073a800040001ac71ceb1296105962d6143ff5"} Oct 31 13:32:01.324473 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.324465 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" event={"ID":"c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6","Type":"ContainerDied","Data":"ba4d836b54cc4ab7ef1d3e1d34df8ff2e4cf69fe395b8068d646bc76029d5c53"} Oct 31 13:32:01.324570 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.324498 5693 scope.go:122] "RemoveContainer" containerID="f46697bff63215d4e7cca52a22073a800040001ac71ceb1296105962d6143ff5" Oct 31 13:32:01.324703 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.324684 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp" Oct 31 13:32:01.331049 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.331021 5693 generic.go:356] "Generic (PLEG): container finished" podID="768af44a-2a4a-4254-a497-e2969f376cf7" containerID="d0d1a27779f09a735fb98bffbaf9b0bd090b5c660136b3f9421f98598e5db960" exitCode=0 Oct 31 13:32:01.331178 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.331105 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c6c445/test-model-c6c445-kserve-69b4dc6d69-nxx4n" event={"ID":"768af44a-2a4a-4254-a497-e2969f376cf7","Type":"ContainerDied","Data":"d0d1a27779f09a735fb98bffbaf9b0bd090b5c660136b3f9421f98598e5db960"} Oct 31 13:32:01.331178 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.331139 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c6c445/test-model-c6c445-kserve-69b4dc6d69-nxx4n" event={"ID":"768af44a-2a4a-4254-a497-e2969f376cf7","Type":"ContainerDied","Data":"3f694af94b5133386e16811f4a7d2ce857e512a8c1a266e5ee2adaa19bdff45a"} Oct 31 13:32:01.331362 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.331216 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c6c445/test-model-c6c445-kserve-69b4dc6d69-nxx4n" Oct 31 13:32:01.332072 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.332043 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-kube-api-access-wktnv" pod "c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6" (UID: "c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6"). InnerVolumeSpecName "kube-api-access-wktnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:32:01.336554 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.336531 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" Oct 31 13:32:01.338570 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.338545 5693 generic.go:356] "Generic (PLEG): container finished" podID="6791b24c-6569-46a0-af4c-3b7b7b356a9d" containerID="3390338bc12883f630fca0b2e5329f29428205b72b1d46e984c6ff4a5edbf0bb" exitCode=0 Oct 31 13:32:01.338792 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.338578 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" event={"ID":"6791b24c-6569-46a0-af4c-3b7b7b356a9d","Type":"ContainerDied","Data":"3390338bc12883f630fca0b2e5329f29428205b72b1d46e984c6ff4a5edbf0bb"} Oct 31 13:32:01.341850 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.341823 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-istio-podinfo" pod "c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6" (UID: "c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:32:01.343969 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.343936 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-istio-token" pod "c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6" (UID: "c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:32:01.347465 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.347405 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" Oct 31 13:32:01.354704 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.354678 5693 scope.go:122] "RemoveContainer" containerID="f46697bff63215d4e7cca52a22073a800040001ac71ceb1296105962d6143ff5" Oct 31 13:32:01.355070 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:32:01.355042 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f46697bff63215d4e7cca52a22073a800040001ac71ceb1296105962d6143ff5\": container with ID starting with f46697bff63215d4e7cca52a22073a800040001ac71ceb1296105962d6143ff5 not found: ID does not exist" containerID="f46697bff63215d4e7cca52a22073a800040001ac71ceb1296105962d6143ff5" Oct 31 13:32:01.355164 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.355086 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f46697bff63215d4e7cca52a22073a800040001ac71ceb1296105962d6143ff5"} err="failed to get container status \"f46697bff63215d4e7cca52a22073a800040001ac71ceb1296105962d6143ff5\": rpc error: code = NotFound desc = could not find container \"f46697bff63215d4e7cca52a22073a800040001ac71ceb1296105962d6143ff5\": container with ID starting with f46697bff63215d4e7cca52a22073a800040001ac71ceb1296105962d6143ff5 not found: ID does not exist" Oct 31 13:32:01.355164 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.355109 5693 scope.go:122] "RemoveContainer" containerID="d0d1a27779f09a735fb98bffbaf9b0bd090b5c660136b3f9421f98598e5db960" Oct 31 13:32:01.383877 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.383337 5693 scope.go:122] "RemoveContainer" containerID="d0d1a27779f09a735fb98bffbaf9b0bd090b5c660136b3f9421f98598e5db960" Oct 31 13:32:01.383877 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:32:01.383775 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0d1a27779f09a735fb98bffbaf9b0bd090b5c660136b3f9421f98598e5db960\": container with ID starting with d0d1a27779f09a735fb98bffbaf9b0bd090b5c660136b3f9421f98598e5db960 not found: ID does not exist" containerID="d0d1a27779f09a735fb98bffbaf9b0bd090b5c660136b3f9421f98598e5db960" Oct 31 13:32:01.383877 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.383809 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0d1a27779f09a735fb98bffbaf9b0bd090b5c660136b3f9421f98598e5db960"} err="failed to get container status \"d0d1a27779f09a735fb98bffbaf9b0bd090b5c660136b3f9421f98598e5db960\": rpc error: code = NotFound desc = could not find container \"d0d1a27779f09a735fb98bffbaf9b0bd090b5c660136b3f9421f98598e5db960\": container with ID starting with d0d1a27779f09a735fb98bffbaf9b0bd090b5c660136b3f9421f98598e5db960 not found: ID does not exist" Oct 31 13:32:01.412050 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.412012 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6791b24c-6569-46a0-af4c-3b7b7b356a9d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6791b24c-6569-46a0-af4c-3b7b7b356a9d-credential-socket\") pod \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\" (UID: \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\") " Oct 31 13:32:01.412260 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.412090 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6791b24c-6569-46a0-af4c-3b7b7b356a9d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6791b24c-6569-46a0-af4c-3b7b7b356a9d-workload-certs\") pod \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\" (UID: \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\") " Oct 31 13:32:01.412260 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.412135 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9410570-a345-441e-9978-2f9ce48d7a09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9410570-a345-441e-9978-2f9ce48d7a09-istio-envoy\") pod \"a9410570-a345-441e-9978-2f9ce48d7a09\" (UID: \"a9410570-a345-441e-9978-2f9ce48d7a09\") " Oct 31 13:32:01.412260 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.412171 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9410570-a345-441e-9978-2f9ce48d7a09-istio-token\" (UniqueName: \"kubernetes.io/projected/a9410570-a345-441e-9978-2f9ce48d7a09-istio-token\") pod \"a9410570-a345-441e-9978-2f9ce48d7a09\" (UID: \"a9410570-a345-441e-9978-2f9ce48d7a09\") " Oct 31 13:32:01.412260 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.412213 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9410570-a345-441e-9978-2f9ce48d7a09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9410570-a345-441e-9978-2f9ce48d7a09-workload-socket\") pod \"a9410570-a345-441e-9978-2f9ce48d7a09\" (UID: \"a9410570-a345-441e-9978-2f9ce48d7a09\") " Oct 31 13:32:01.412260 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.412241 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9410570-a345-441e-9978-2f9ce48d7a09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9410570-a345-441e-9978-2f9ce48d7a09-credential-socket\") pod \"a9410570-a345-441e-9978-2f9ce48d7a09\" (UID: \"a9410570-a345-441e-9978-2f9ce48d7a09\") " Oct 31 13:32:01.412538 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.412371 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9410570-a345-441e-9978-2f9ce48d7a09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9410570-a345-441e-9978-2f9ce48d7a09-istio-data\") pod \"a9410570-a345-441e-9978-2f9ce48d7a09\" (UID: \"a9410570-a345-441e-9978-2f9ce48d7a09\") " Oct 31 13:32:01.412538 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.412486 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6791b24c-6569-46a0-af4c-3b7b7b356a9d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6791b24c-6569-46a0-af4c-3b7b7b356a9d-istio-data\") pod \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\" (UID: \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\") " Oct 31 13:32:01.412538 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.412515 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6791b24c-6569-46a0-af4c-3b7b7b356a9d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6791b24c-6569-46a0-af4c-3b7b7b356a9d-istio-envoy\") pod \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\" (UID: \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\") " Oct 31 13:32:01.412767 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.412550 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9410570-a345-441e-9978-2f9ce48d7a09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9410570-a345-441e-9978-2f9ce48d7a09-workload-certs\") pod \"a9410570-a345-441e-9978-2f9ce48d7a09\" (UID: \"a9410570-a345-441e-9978-2f9ce48d7a09\") " Oct 31 13:32:01.412767 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.412599 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6791b24c-6569-46a0-af4c-3b7b7b356a9d-istio-token\" (UniqueName: \"kubernetes.io/projected/6791b24c-6569-46a0-af4c-3b7b7b356a9d-istio-token\") pod \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\" (UID: \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\") " Oct 31 13:32:01.412767 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.412661 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9410570-a345-441e-9978-2f9ce48d7a09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9410570-a345-441e-9978-2f9ce48d7a09-istio-podinfo\") pod \"a9410570-a345-441e-9978-2f9ce48d7a09\" (UID: \"a9410570-a345-441e-9978-2f9ce48d7a09\") " Oct 31 13:32:01.412767 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.412705 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9410570-a345-441e-9978-2f9ce48d7a09-kube-api-access-b425g\" (UniqueName: \"kubernetes.io/projected/a9410570-a345-441e-9978-2f9ce48d7a09-kube-api-access-b425g\") pod \"a9410570-a345-441e-9978-2f9ce48d7a09\" (UID: \"a9410570-a345-441e-9978-2f9ce48d7a09\") " Oct 31 13:32:01.412960 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.412805 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6791b24c-6569-46a0-af4c-3b7b7b356a9d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6791b24c-6569-46a0-af4c-3b7b7b356a9d-workload-socket\") pod \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\" (UID: \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\") " Oct 31 13:32:01.412960 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.412876 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9410570-a345-441e-9978-2f9ce48d7a09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9410570-a345-441e-9978-2f9ce48d7a09-istiod-ca-cert\") pod \"a9410570-a345-441e-9978-2f9ce48d7a09\" (UID: \"a9410570-a345-441e-9978-2f9ce48d7a09\") " Oct 31 13:32:01.413377 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.413343 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6791b24c-6569-46a0-af4c-3b7b7b356a9d-kube-api-access-v7gk7\" (UniqueName: \"kubernetes.io/projected/6791b24c-6569-46a0-af4c-3b7b7b356a9d-kube-api-access-v7gk7\") pod \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\" (UID: \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\") " Oct 31 13:32:01.413473 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.413404 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6791b24c-6569-46a0-af4c-3b7b7b356a9d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6791b24c-6569-46a0-af4c-3b7b7b356a9d-istiod-ca-cert\") pod \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\" (UID: \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\") " Oct 31 13:32:01.413536 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.413500 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6791b24c-6569-46a0-af4c-3b7b7b356a9d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6791b24c-6569-46a0-af4c-3b7b7b356a9d-istio-podinfo\") pod \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\" (UID: \"6791b24c-6569-46a0-af4c-3b7b7b356a9d\") " Oct 31 13:32:01.414776 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.414738 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9410570-a345-441e-9978-2f9ce48d7a09-credential-socket" pod "a9410570-a345-441e-9978-2f9ce48d7a09" (UID: "a9410570-a345-441e-9978-2f9ce48d7a09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:01.414889 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.414848 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9410570-a345-441e-9978-2f9ce48d7a09-istio-data" pod "a9410570-a345-441e-9978-2f9ce48d7a09" (UID: "a9410570-a345-441e-9978-2f9ce48d7a09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:01.415089 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.415063 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9410570-a345-441e-9978-2f9ce48d7a09-workload-socket" pod "a9410570-a345-441e-9978-2f9ce48d7a09" (UID: "a9410570-a345-441e-9978-2f9ce48d7a09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:01.415188 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.415174 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6791b24c-6569-46a0-af4c-3b7b7b356a9d-istio-data" pod "6791b24c-6569-46a0-af4c-3b7b7b356a9d" (UID: "6791b24c-6569-46a0-af4c-3b7b7b356a9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:01.415483 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.415464 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6791b24c-6569-46a0-af4c-3b7b7b356a9d-workload-certs" pod "6791b24c-6569-46a0-af4c-3b7b7b356a9d" (UID: "6791b24c-6569-46a0-af4c-3b7b7b356a9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:01.415616 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.415558 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6791b24c-6569-46a0-af4c-3b7b7b356a9d-workload-socket" pod "6791b24c-6569-46a0-af4c-3b7b7b356a9d" (UID: "6791b24c-6569-46a0-af4c-3b7b7b356a9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:01.415927 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.415907 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6791b24c-6569-46a0-af4c-3b7b7b356a9d-credential-socket" pod "6791b24c-6569-46a0-af4c-3b7b7b356a9d" (UID: "6791b24c-6569-46a0-af4c-3b7b7b356a9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:01.416042 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.415983 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9410570-a345-441e-9978-2f9ce48d7a09-istiod-ca-cert" pod "a9410570-a345-441e-9978-2f9ce48d7a09" (UID: "a9410570-a345-441e-9978-2f9ce48d7a09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:32:01.416347 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.416328 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9410570-a345-441e-9978-2f9ce48d7a09-workload-certs" pod "a9410570-a345-441e-9978-2f9ce48d7a09" (UID: "a9410570-a345-441e-9978-2f9ce48d7a09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:01.423354 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.423242 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9410570-a345-441e-9978-2f9ce48d7a09-istio-podinfo" pod "a9410570-a345-441e-9978-2f9ce48d7a09" (UID: "a9410570-a345-441e-9978-2f9ce48d7a09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:32:01.429503 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.429317 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6791b24c-6569-46a0-af4c-3b7b7b356a9d-istio-podinfo" pod "6791b24c-6569-46a0-af4c-3b7b7b356a9d" (UID: "6791b24c-6569-46a0-af4c-3b7b7b356a9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:32:01.430826 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.430072 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6791b24c-6569-46a0-af4c-3b7b7b356a9d-istio-token" pod "6791b24c-6569-46a0-af4c-3b7b7b356a9d" (UID: "6791b24c-6569-46a0-af4c-3b7b7b356a9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:32:01.430826 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.430124 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6791b24c-6569-46a0-af4c-3b7b7b356a9d-istiod-ca-cert" pod "6791b24c-6569-46a0-af4c-3b7b7b356a9d" (UID: "6791b24c-6569-46a0-af4c-3b7b7b356a9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:32:01.430826 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.430192 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9410570-a345-441e-9978-2f9ce48d7a09-istio-envoy" pod "a9410570-a345-441e-9978-2f9ce48d7a09" (UID: "a9410570-a345-441e-9978-2f9ce48d7a09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:01.430826 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.430442 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6791b24c-6569-46a0-af4c-3b7b7b356a9d-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.430826 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.430465 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.430826 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.430480 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.430826 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.430496 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6791b24c-6569-46a0-af4c-3b7b7b356a9d-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.430826 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.430510 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.430826 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.430522 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6791b24c-6569-46a0-af4c-3b7b7b356a9d-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.430826 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.430537 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.430826 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.430551 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wktnv\" (UniqueName: \"kubernetes.io/projected/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-kube-api-access-wktnv\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.430826 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.430566 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9410570-a345-441e-9978-2f9ce48d7a09-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.430826 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.430581 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9410570-a345-441e-9978-2f9ce48d7a09-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.430826 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.430594 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9410570-a345-441e-9978-2f9ce48d7a09-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.430826 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.430607 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.430826 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.430622 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.430826 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.430667 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9410570-a345-441e-9978-2f9ce48d7a09-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.430826 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.430682 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.430826 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.430696 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6791b24c-6569-46a0-af4c-3b7b7b356a9d-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.430826 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.430710 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9410570-a345-441e-9978-2f9ce48d7a09-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.430826 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.430724 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6791b24c-6569-46a0-af4c-3b7b7b356a9d-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.430826 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.430738 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9410570-a345-441e-9978-2f9ce48d7a09-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.430826 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.430751 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6791b24c-6569-46a0-af4c-3b7b7b356a9d-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.430826 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.430766 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9410570-a345-441e-9978-2f9ce48d7a09-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.430826 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.430779 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.430826 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.430793 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6791b24c-6569-46a0-af4c-3b7b7b356a9d-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.438765 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.431111 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9410570-a345-441e-9978-2f9ce48d7a09-istio-token" pod "a9410570-a345-441e-9978-2f9ce48d7a09" (UID: "a9410570-a345-441e-9978-2f9ce48d7a09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:32:01.438765 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.431890 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9410570-a345-441e-9978-2f9ce48d7a09-kube-api-access-b425g" pod "a9410570-a345-441e-9978-2f9ce48d7a09" (UID: "a9410570-a345-441e-9978-2f9ce48d7a09"). InnerVolumeSpecName "kube-api-access-b425g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:32:01.438765 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.433614 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6791b24c-6569-46a0-af4c-3b7b7b356a9d-istio-envoy" pod "6791b24c-6569-46a0-af4c-3b7b7b356a9d" (UID: "6791b24c-6569-46a0-af4c-3b7b7b356a9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:01.440595 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.439804 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6791b24c-6569-46a0-af4c-3b7b7b356a9d-kube-api-access-v7gk7" pod "6791b24c-6569-46a0-af4c-3b7b7b356a9d" (UID: "6791b24c-6569-46a0-af4c-3b7b7b356a9d"). InnerVolumeSpecName "kube-api-access-v7gk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:32:01.445967 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.440844 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c6c445/test-model-c6c445-kserve-69b4dc6d69-nxx4n"] Oct 31 13:32:01.445967 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.440874 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-c6c445/test-model-c6c445-kserve-69b4dc6d69-nxx4n"] Oct 31 13:32:01.461841 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.461459 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-5dbd39-6b6f6d9559-9tbjq" podUID="b8730031-f623-4f9e-ba5f-014340976d85" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:32:01.462437 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.462296 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5dbd39-5c6f7bcf67-d5m99" podUID="5cd6c998-975c-4628-a886-e9eeb20b3a64" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:32:01.533964 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.533926 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6791b24c-6569-46a0-af4c-3b7b7b356a9d-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.534158 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.533968 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b425g\" (UniqueName: \"kubernetes.io/projected/a9410570-a345-441e-9978-2f9ce48d7a09-kube-api-access-b425g\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.534158 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.533992 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7gk7\" (UniqueName: \"kubernetes.io/projected/6791b24c-6569-46a0-af4c-3b7b7b356a9d-kube-api-access-v7gk7\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.534158 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.534015 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9410570-a345-441e-9978-2f9ce48d7a09-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.648134 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.648100 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp"] Oct 31 13:32:01.655090 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.655056 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-crtwp"] Oct 31 13:32:01.698136 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:01.698088 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-c6c445/test-model-c6c445-kserve-69b4dc6d69-nxx4n" podUID="768af44a-2a4a-4254-a497-e2969f376cf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.239:8000/ready\": context deadline exceeded" Oct 31 13:32:02.352690 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:02.352621 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" Oct 31 13:32:02.352690 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:02.352624 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv" event={"ID":"6791b24c-6569-46a0-af4c-3b7b7b356a9d","Type":"ContainerDied","Data":"32f4b40e7ca1f5e1a08109c60f1af2a9ff21e0999c8861b68284fd415938d3d6"} Oct 31 13:32:02.352954 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:02.352767 5693 scope.go:122] "RemoveContainer" containerID="3390338bc12883f630fca0b2e5329f29428205b72b1d46e984c6ff4a5edbf0bb" Oct 31 13:32:02.355696 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:02.355583 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" Oct 31 13:32:02.355696 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:02.355598 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg" event={"ID":"a9410570-a345-441e-9978-2f9ce48d7a09","Type":"ContainerDied","Data":"4a1119f5dead40badd8cbdc210e4e77dc4d4ca5d2998693762e34054bf920d69"} Oct 31 13:32:02.378769 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:02.378724 5693 scope.go:122] "RemoveContainer" containerID="557ce5b49869163d06f9cba2506416807d41c914db847c420e7ca9bb17aa44f7" Oct 31 13:32:02.385317 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:32:02.385291 5693 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9410570_a345_441e_9978_2f9ce48d7a09.slice/crio-4a1119f5dead40badd8cbdc210e4e77dc4d4ca5d2998693762e34054bf920d69\": RecentStats: unable to find data in memory cache]" Oct 31 13:32:02.398968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:02.398924 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv"] Oct 31 13:32:02.404267 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:02.404235 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-rqdmv"] Oct 31 13:32:02.415505 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:02.415470 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg"] Oct 31 13:32:02.421515 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:02.421483 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-jvpmg"] Oct 31 13:32:02.966925 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:02.966889 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-384dbe-6f9d5d675-qgdfx"] Oct 31 13:32:02.997730 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:02.997691 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-384dbe-57cd8fcd7-rsx6p"] Oct 31 13:32:03.588488 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:03.588449 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6791b24c-6569-46a0-af4c-3b7b7b356a9d" path="/var/lib/kubelet/pods/6791b24c-6569-46a0-af4c-3b7b7b356a9d/volumes" Oct 31 13:32:03.589299 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:03.589213 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="768af44a-2a4a-4254-a497-e2969f376cf7" path="/var/lib/kubelet/pods/768af44a-2a4a-4254-a497-e2969f376cf7/volumes" Oct 31 13:32:03.590219 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:03.589904 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9410570-a345-441e-9978-2f9ce48d7a09" path="/var/lib/kubelet/pods/a9410570-a345-441e-9978-2f9ce48d7a09/volumes" Oct 31 13:32:03.591560 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:03.591004 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6" path="/var/lib/kubelet/pods/c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6/volumes" Oct 31 13:32:06.408733 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:06.408702 5693 generic.go:356] "Generic (PLEG): container finished" podID="d137459e-9920-4c27-a910-0216cc6a7168" containerID="8eac644d96d259e42257cee93694e01758a3f6c3885083b74871f9f8ea701d42" exitCode=0 Oct 31 13:32:06.409257 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:06.408789 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" event={"ID":"d137459e-9920-4c27-a910-0216cc6a7168","Type":"ContainerDied","Data":"8eac644d96d259e42257cee93694e01758a3f6c3885083b74871f9f8ea701d42"} Oct 31 13:32:06.409313 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:06.409301 5693 scope.go:122] "RemoveContainer" containerID="8eac644d96d259e42257cee93694e01758a3f6c3885083b74871f9f8ea701d42" Oct 31 13:32:06.411454 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:06.411425 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:32:06.411525 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:06.411498 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:32:06.411562 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:06.411530 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:32:07.244969 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:07.244943 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-50b4aa-54d6cd77df-zsv8v" Oct 31 13:32:07.245171 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:07.245006 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-50b4aa-58bd97d4c5-mgqbp" Oct 31 13:32:07.429035 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:07.428973 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" event={"ID":"d137459e-9920-4c27-a910-0216cc6a7168","Type":"ContainerStarted","Data":"aa52134bd572d283457eb255e08d0b6a91ba3b48efbc985ba7cd62bcb3e186cb"} Oct 31 13:32:07.483003 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:07.482970 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" Oct 31 13:32:07.484538 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:07.484505 5693 patch_prober.go:29] interesting pod/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" start-of-body= Oct 31 13:32:07.484702 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:07.484566 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" podUID="d137459e-9920-4c27-a910-0216cc6a7168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" Oct 31 13:32:07.967633 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:07.967592 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-384dbe-6f9d5d675-qgdfx" podUID="dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8" containerName="payload-processing" containerID="cri-o://7ec67c3e102d4ea3aee7ee6f26dc18293867defc4089592582d204c331331530" gracePeriod=25 Oct 31 13:32:07.998209 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:07.998167 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-384dbe-57cd8fcd7-rsx6p" podUID="8903924e-49a1-456e-a26d-052f3a5901d8" containerName="payload-pre-processing" containerID="cri-o://62ec82ee6807326506d56993dfc66e8f49c4f27cb476a47c486067965784b1f4" gracePeriod=25 Oct 31 13:32:08.484548 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:08.484494 5693 patch_prober.go:29] interesting pod/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" start-of-body= Oct 31 13:32:08.485345 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:08.485311 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" podUID="d137459e-9920-4c27-a910-0216cc6a7168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" Oct 31 13:32:09.263569 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:09.263514 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-4023ad-7994b685f7-9m2jh" podUID="bc39dc34-d1e4-466d-bc1d-5e77bac8dc39" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:32:09.307674 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:09.300285 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5dbd39-6b6f6d9559-9tbjq" Oct 31 13:32:09.312273 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:09.312239 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5dbd39-5c6f7bcf67-d5m99" Oct 31 13:32:09.448457 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:09.448370 5693 generic.go:356] "Generic (PLEG): container finished" podID="5cd6c998-975c-4628-a886-e9eeb20b3a64" containerID="49f0dc0eefa54aab9f5ecedd4715d6b9de50e2f38f03c4f1ce27d5f5149129c2" exitCode=0 Oct 31 13:32:09.448457 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:09.448419 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5dbd39-5c6f7bcf67-d5m99" Oct 31 13:32:09.448750 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:09.448458 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5dbd39-5c6f7bcf67-d5m99" event={"ID":"5cd6c998-975c-4628-a886-e9eeb20b3a64","Type":"ContainerDied","Data":"49f0dc0eefa54aab9f5ecedd4715d6b9de50e2f38f03c4f1ce27d5f5149129c2"} Oct 31 13:32:09.448750 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:09.448515 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5dbd39-5c6f7bcf67-d5m99" event={"ID":"5cd6c998-975c-4628-a886-e9eeb20b3a64","Type":"ContainerDied","Data":"0077d370a4ef0714c9c060d086ad1bc916b39a461591b5f841b1b5f2c76c660d"} Oct 31 13:32:09.448750 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:09.448544 5693 scope.go:122] "RemoveContainer" containerID="49f0dc0eefa54aab9f5ecedd4715d6b9de50e2f38f03c4f1ce27d5f5149129c2" Oct 31 13:32:09.450170 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:09.450149 5693 generic.go:356] "Generic (PLEG): container finished" podID="b8730031-f623-4f9e-ba5f-014340976d85" containerID="a16f39cf357902316f9b559f733bf6a67d98e9d8cee3ef73b846cc9b7c2fd486" exitCode=0 Oct 31 13:32:09.450272 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:09.450192 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5dbd39-6b6f6d9559-9tbjq" Oct 31 13:32:09.450272 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:09.450221 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5dbd39-6b6f6d9559-9tbjq" event={"ID":"b8730031-f623-4f9e-ba5f-014340976d85","Type":"ContainerDied","Data":"a16f39cf357902316f9b559f733bf6a67d98e9d8cee3ef73b846cc9b7c2fd486"} Oct 31 13:32:09.450272 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:09.450250 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5dbd39-6b6f6d9559-9tbjq" event={"ID":"b8730031-f623-4f9e-ba5f-014340976d85","Type":"ContainerDied","Data":"eaa09af39ea26ea011f162f3c3c4949a67b5491047b421e9f096640a43bfcfc6"} Oct 31 13:32:09.459385 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:09.459362 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cd6c998-975c-4628-a886-e9eeb20b3a64-kube-api-access-9rqs9\" (UniqueName: \"kubernetes.io/projected/5cd6c998-975c-4628-a886-e9eeb20b3a64-kube-api-access-9rqs9\") pod \"5cd6c998-975c-4628-a886-e9eeb20b3a64\" (UID: \"5cd6c998-975c-4628-a886-e9eeb20b3a64\") " Oct 31 13:32:09.459511 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:09.459450 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8730031-f623-4f9e-ba5f-014340976d85-kube-api-access-5g7mw\" (UniqueName: \"kubernetes.io/projected/b8730031-f623-4f9e-ba5f-014340976d85-kube-api-access-5g7mw\") pod \"b8730031-f623-4f9e-ba5f-014340976d85\" (UID: \"b8730031-f623-4f9e-ba5f-014340976d85\") " Oct 31 13:32:09.459511 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:09.459501 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cd6c998-975c-4628-a886-e9eeb20b3a64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cd6c998-975c-4628-a886-e9eeb20b3a64-plugins-config-volume\") pod \"5cd6c998-975c-4628-a886-e9eeb20b3a64\" (UID: \"5cd6c998-975c-4628-a886-e9eeb20b3a64\") " Oct 31 13:32:09.459603 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:09.459568 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8730031-f623-4f9e-ba5f-014340976d85-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8730031-f623-4f9e-ba5f-014340976d85-plugins-config-volume\") pod \"b8730031-f623-4f9e-ba5f-014340976d85\" (UID: \"b8730031-f623-4f9e-ba5f-014340976d85\") " Oct 31 13:32:09.460022 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:09.459982 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8730031-f623-4f9e-ba5f-014340976d85-plugins-config-volume" pod "b8730031-f623-4f9e-ba5f-014340976d85" (UID: "b8730031-f623-4f9e-ba5f-014340976d85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:32:09.460270 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:09.460173 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cd6c998-975c-4628-a886-e9eeb20b3a64-plugins-config-volume" pod "5cd6c998-975c-4628-a886-e9eeb20b3a64" (UID: "5cd6c998-975c-4628-a886-e9eeb20b3a64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:32:09.460675 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:09.460629 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cd6c998-975c-4628-a886-e9eeb20b3a64-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:09.460780 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:09.460680 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8730031-f623-4f9e-ba5f-014340976d85-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:09.461182 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:09.461160 5693 scope.go:122] "RemoveContainer" containerID="49f0dc0eefa54aab9f5ecedd4715d6b9de50e2f38f03c4f1ce27d5f5149129c2" Oct 31 13:32:09.461562 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:32:09.461523 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49f0dc0eefa54aab9f5ecedd4715d6b9de50e2f38f03c4f1ce27d5f5149129c2\": container with ID starting with 49f0dc0eefa54aab9f5ecedd4715d6b9de50e2f38f03c4f1ce27d5f5149129c2 not found: ID does not exist" containerID="49f0dc0eefa54aab9f5ecedd4715d6b9de50e2f38f03c4f1ce27d5f5149129c2" Oct 31 13:32:09.461679 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:09.461576 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49f0dc0eefa54aab9f5ecedd4715d6b9de50e2f38f03c4f1ce27d5f5149129c2"} err="failed to get container status \"49f0dc0eefa54aab9f5ecedd4715d6b9de50e2f38f03c4f1ce27d5f5149129c2\": rpc error: code = NotFound desc = could not find container \"49f0dc0eefa54aab9f5ecedd4715d6b9de50e2f38f03c4f1ce27d5f5149129c2\": container with ID starting with 49f0dc0eefa54aab9f5ecedd4715d6b9de50e2f38f03c4f1ce27d5f5149129c2 not found: ID does not exist" Oct 31 13:32:09.461679 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:09.461602 5693 scope.go:122] "RemoveContainer" containerID="a16f39cf357902316f9b559f733bf6a67d98e9d8cee3ef73b846cc9b7c2fd486" Oct 31 13:32:09.462056 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:09.462020 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cd6c998-975c-4628-a886-e9eeb20b3a64-kube-api-access-9rqs9" pod "5cd6c998-975c-4628-a886-e9eeb20b3a64" (UID: "5cd6c998-975c-4628-a886-e9eeb20b3a64"). InnerVolumeSpecName "kube-api-access-9rqs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:32:09.462144 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:09.462121 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8730031-f623-4f9e-ba5f-014340976d85-kube-api-access-5g7mw" pod "b8730031-f623-4f9e-ba5f-014340976d85" (UID: "b8730031-f623-4f9e-ba5f-014340976d85"). InnerVolumeSpecName "kube-api-access-5g7mw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:32:09.473098 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:09.473076 5693 scope.go:122] "RemoveContainer" containerID="a16f39cf357902316f9b559f733bf6a67d98e9d8cee3ef73b846cc9b7c2fd486" Oct 31 13:32:09.473373 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:32:09.473354 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a16f39cf357902316f9b559f733bf6a67d98e9d8cee3ef73b846cc9b7c2fd486\": container with ID starting with a16f39cf357902316f9b559f733bf6a67d98e9d8cee3ef73b846cc9b7c2fd486 not found: ID does not exist" containerID="a16f39cf357902316f9b559f733bf6a67d98e9d8cee3ef73b846cc9b7c2fd486" Oct 31 13:32:09.473447 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:09.473383 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a16f39cf357902316f9b559f733bf6a67d98e9d8cee3ef73b846cc9b7c2fd486"} err="failed to get container status \"a16f39cf357902316f9b559f733bf6a67d98e9d8cee3ef73b846cc9b7c2fd486\": rpc error: code = NotFound desc = could not find container \"a16f39cf357902316f9b559f733bf6a67d98e9d8cee3ef73b846cc9b7c2fd486\": container with ID starting with a16f39cf357902316f9b559f733bf6a67d98e9d8cee3ef73b846cc9b7c2fd486 not found: ID does not exist" Oct 31 13:32:09.495622 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:09.495587 5693 patch_prober.go:29] interesting pod/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" start-of-body= Oct 31 13:32:09.496232 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:09.496203 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" podUID="d137459e-9920-4c27-a910-0216cc6a7168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" Oct 31 13:32:09.562279 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:09.562253 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g7mw\" (UniqueName: \"kubernetes.io/projected/b8730031-f623-4f9e-ba5f-014340976d85-kube-api-access-5g7mw\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:09.562486 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:09.562301 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rqs9\" (UniqueName: \"kubernetes.io/projected/5cd6c998-975c-4628-a886-e9eeb20b3a64-kube-api-access-9rqs9\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:09.840668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:09.837492 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5dbd39-6b6f6d9559-9tbjq"] Oct 31 13:32:09.855703 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:09.852282 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5dbd39-6b6f6d9559-9tbjq"] Oct 31 13:32:09.859374 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:09.857778 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5dbd39-5c6f7bcf67-d5m99"] Oct 31 13:32:09.863776 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:09.863738 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5dbd39-5c6f7bcf67-d5m99"] Oct 31 13:32:10.501169 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:10.501132 5693 patch_prober.go:29] interesting pod/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" start-of-body= Oct 31 13:32:10.501595 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:10.501199 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" podUID="d137459e-9920-4c27-a910-0216cc6a7168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" Oct 31 13:32:11.414928 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:11.414888 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cd6c998-975c-4628-a886-e9eeb20b3a64" path="/var/lib/kubelet/pods/5cd6c998-975c-4628-a886-e9eeb20b3a64/volumes" Oct 31 13:32:11.415495 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:11.415474 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8730031-f623-4f9e-ba5f-014340976d85" path="/var/lib/kubelet/pods/b8730031-f623-4f9e-ba5f-014340976d85/volumes" Oct 31 13:32:11.499668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:11.498105 5693 patch_prober.go:29] interesting pod/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" start-of-body= Oct 31 13:32:11.499668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:11.498181 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" podUID="d137459e-9920-4c27-a910-0216cc6a7168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" Oct 31 13:32:11.762748 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:11.762674 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" Oct 31 13:32:12.131576 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:12.131104 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h"] Oct 31 13:32:12.131576 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:12.131433 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" podUID="8c08b4ba-3066-4b09-ba62-de15be6ccea2" containerName="istio-proxy" containerID="cri-o://d5054a2c9a14b7b56af8fc1d7e745c8668669c6ce26c003f089c92e9136021cb" gracePeriod=30 Oct 31 13:32:12.504704 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:12.500335 5693 patch_prober.go:29] interesting pod/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" start-of-body= Oct 31 13:32:12.504704 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:12.500413 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" podUID="d137459e-9920-4c27-a910-0216cc6a7168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" Oct 31 13:32:13.483332 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:13.483290 5693 patch_prober.go:29] interesting pod/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" start-of-body= Oct 31 13:32:13.483959 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:13.483924 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" podUID="d137459e-9920-4c27-a910-0216cc6a7168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" Oct 31 13:32:14.487953 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:14.487925 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" Oct 31 13:32:14.489177 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:14.489155 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" Oct 31 13:32:16.551261 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:16.551224 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-384dbe-6f9d5d675-qgdfx" podUID="dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:32:16.551846 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:16.551811 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-384dbe-57cd8fcd7-rsx6p" podUID="8903924e-49a1-456e-a26d-052f3a5901d8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:32:17.475189 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:17.475159 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" Oct 31 13:32:17.526375 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:17.526344 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c08b4ba-3066-4b09-ba62-de15be6ccea2-kube-api-access-zjjp6\" (UniqueName: \"kubernetes.io/projected/8c08b4ba-3066-4b09-ba62-de15be6ccea2-kube-api-access-zjjp6\") pod \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\" (UID: \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\") " Oct 31 13:32:17.526699 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:17.526489 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c08b4ba-3066-4b09-ba62-de15be6ccea2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c08b4ba-3066-4b09-ba62-de15be6ccea2-istio-data\") pod \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\" (UID: \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\") " Oct 31 13:32:17.526699 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:17.526543 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c08b4ba-3066-4b09-ba62-de15be6ccea2-istio-token\" (UniqueName: \"kubernetes.io/projected/8c08b4ba-3066-4b09-ba62-de15be6ccea2-istio-token\") pod \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\" (UID: \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\") " Oct 31 13:32:17.526699 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:17.526621 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c08b4ba-3066-4b09-ba62-de15be6ccea2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c08b4ba-3066-4b09-ba62-de15be6ccea2-istio-envoy\") pod \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\" (UID: \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\") " Oct 31 13:32:17.526925 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:17.526701 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c08b4ba-3066-4b09-ba62-de15be6ccea2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c08b4ba-3066-4b09-ba62-de15be6ccea2-istio-podinfo\") pod \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\" (UID: \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\") " Oct 31 13:32:17.526925 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:17.526738 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c08b4ba-3066-4b09-ba62-de15be6ccea2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c08b4ba-3066-4b09-ba62-de15be6ccea2-credential-socket\") pod \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\" (UID: \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\") " Oct 31 13:32:17.526925 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:17.526814 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c08b4ba-3066-4b09-ba62-de15be6ccea2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c08b4ba-3066-4b09-ba62-de15be6ccea2-workload-certs\") pod \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\" (UID: \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\") " Oct 31 13:32:17.526925 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:17.526844 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c08b4ba-3066-4b09-ba62-de15be6ccea2-istio-data" pod "8c08b4ba-3066-4b09-ba62-de15be6ccea2" (UID: "8c08b4ba-3066-4b09-ba62-de15be6ccea2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:17.527190 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:17.526938 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c08b4ba-3066-4b09-ba62-de15be6ccea2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c08b4ba-3066-4b09-ba62-de15be6ccea2-istiod-ca-cert\") pod \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\" (UID: \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\") " Oct 31 13:32:17.527190 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:17.526967 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c08b4ba-3066-4b09-ba62-de15be6ccea2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c08b4ba-3066-4b09-ba62-de15be6ccea2-workload-socket\") pod \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\" (UID: \"8c08b4ba-3066-4b09-ba62-de15be6ccea2\") " Oct 31 13:32:17.527190 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:17.527042 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c08b4ba-3066-4b09-ba62-de15be6ccea2-credential-socket" pod "8c08b4ba-3066-4b09-ba62-de15be6ccea2" (UID: "8c08b4ba-3066-4b09-ba62-de15be6ccea2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:17.527190 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:17.527214 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c08b4ba-3066-4b09-ba62-de15be6ccea2-workload-certs" pod "8c08b4ba-3066-4b09-ba62-de15be6ccea2" (UID: "8c08b4ba-3066-4b09-ba62-de15be6ccea2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:17.527524 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:17.527465 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c08b4ba-3066-4b09-ba62-de15be6ccea2-istiod-ca-cert" pod "8c08b4ba-3066-4b09-ba62-de15be6ccea2" (UID: "8c08b4ba-3066-4b09-ba62-de15be6ccea2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:32:17.527876 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:17.527806 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c08b4ba-3066-4b09-ba62-de15be6ccea2-workload-socket" pod "8c08b4ba-3066-4b09-ba62-de15be6ccea2" (UID: "8c08b4ba-3066-4b09-ba62-de15be6ccea2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:17.528336 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:17.528313 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c08b4ba-3066-4b09-ba62-de15be6ccea2-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:17.528478 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:17.528341 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c08b4ba-3066-4b09-ba62-de15be6ccea2-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:17.528478 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:17.528356 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c08b4ba-3066-4b09-ba62-de15be6ccea2-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:17.528478 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:17.528369 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c08b4ba-3066-4b09-ba62-de15be6ccea2-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:17.528478 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:17.528383 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c08b4ba-3066-4b09-ba62-de15be6ccea2-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:17.532171 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:17.532143 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c08b4ba-3066-4b09-ba62-de15be6ccea2-kube-api-access-zjjp6" pod "8c08b4ba-3066-4b09-ba62-de15be6ccea2" (UID: "8c08b4ba-3066-4b09-ba62-de15be6ccea2"). InnerVolumeSpecName "kube-api-access-zjjp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:32:17.532293 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:17.532219 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c08b4ba-3066-4b09-ba62-de15be6ccea2-istio-podinfo" pod "8c08b4ba-3066-4b09-ba62-de15be6ccea2" (UID: "8c08b4ba-3066-4b09-ba62-de15be6ccea2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:32:17.532895 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:17.532547 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c08b4ba-3066-4b09-ba62-de15be6ccea2-istio-token" pod "8c08b4ba-3066-4b09-ba62-de15be6ccea2" (UID: "8c08b4ba-3066-4b09-ba62-de15be6ccea2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:32:17.533100 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:17.533082 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c08b4ba-3066-4b09-ba62-de15be6ccea2-istio-envoy" pod "8c08b4ba-3066-4b09-ba62-de15be6ccea2" (UID: "8c08b4ba-3066-4b09-ba62-de15be6ccea2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:17.590529 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:17.590488 5693 generic.go:356] "Generic (PLEG): container finished" podID="8c08b4ba-3066-4b09-ba62-de15be6ccea2" containerID="d5054a2c9a14b7b56af8fc1d7e745c8668669c6ce26c003f089c92e9136021cb" exitCode=0 Oct 31 13:32:17.591057 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:17.590796 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" event={"ID":"8c08b4ba-3066-4b09-ba62-de15be6ccea2","Type":"ContainerDied","Data":"d5054a2c9a14b7b56af8fc1d7e745c8668669c6ce26c003f089c92e9136021cb"} Oct 31 13:32:17.591057 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:17.590853 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" event={"ID":"8c08b4ba-3066-4b09-ba62-de15be6ccea2","Type":"ContainerDied","Data":"7e002553c568441811742f917e89619999698b55d4e91201473b592e0a8623b1"} Oct 31 13:32:17.591057 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:17.590886 5693 scope.go:122] "RemoveContainer" containerID="d5054a2c9a14b7b56af8fc1d7e745c8668669c6ce26c003f089c92e9136021cb" Oct 31 13:32:17.591249 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:17.591063 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h" Oct 31 13:32:17.619033 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:17.619009 5693 scope.go:122] "RemoveContainer" containerID="37714a9885d5013797267376d3e7f865f72904ea538f01a35e946d9fbb52bb90" Oct 31 13:32:17.631440 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:17.631104 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjjp6\" (UniqueName: \"kubernetes.io/projected/8c08b4ba-3066-4b09-ba62-de15be6ccea2-kube-api-access-zjjp6\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:17.631440 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:17.631138 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c08b4ba-3066-4b09-ba62-de15be6ccea2-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:17.631440 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:17.631153 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c08b4ba-3066-4b09-ba62-de15be6ccea2-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:17.631440 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:17.631167 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c08b4ba-3066-4b09-ba62-de15be6ccea2-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:17.631440 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:17.631289 5693 scope.go:122] "RemoveContainer" containerID="d5054a2c9a14b7b56af8fc1d7e745c8668669c6ce26c003f089c92e9136021cb" Oct 31 13:32:17.631774 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:32:17.631614 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5054a2c9a14b7b56af8fc1d7e745c8668669c6ce26c003f089c92e9136021cb\": container with ID starting with d5054a2c9a14b7b56af8fc1d7e745c8668669c6ce26c003f089c92e9136021cb not found: ID does not exist" containerID="d5054a2c9a14b7b56af8fc1d7e745c8668669c6ce26c003f089c92e9136021cb" Oct 31 13:32:17.631774 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:17.631681 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5054a2c9a14b7b56af8fc1d7e745c8668669c6ce26c003f089c92e9136021cb"} err="failed to get container status \"d5054a2c9a14b7b56af8fc1d7e745c8668669c6ce26c003f089c92e9136021cb\": rpc error: code = NotFound desc = could not find container \"d5054a2c9a14b7b56af8fc1d7e745c8668669c6ce26c003f089c92e9136021cb\": container with ID starting with d5054a2c9a14b7b56af8fc1d7e745c8668669c6ce26c003f089c92e9136021cb not found: ID does not exist" Oct 31 13:32:17.631774 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:17.631706 5693 scope.go:122] "RemoveContainer" containerID="37714a9885d5013797267376d3e7f865f72904ea538f01a35e946d9fbb52bb90" Oct 31 13:32:17.632008 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:32:17.631974 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37714a9885d5013797267376d3e7f865f72904ea538f01a35e946d9fbb52bb90\": container with ID starting with 37714a9885d5013797267376d3e7f865f72904ea538f01a35e946d9fbb52bb90 not found: ID does not exist" containerID="37714a9885d5013797267376d3e7f865f72904ea538f01a35e946d9fbb52bb90" Oct 31 13:32:17.632081 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:17.632016 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37714a9885d5013797267376d3e7f865f72904ea538f01a35e946d9fbb52bb90"} err="failed to get container status \"37714a9885d5013797267376d3e7f865f72904ea538f01a35e946d9fbb52bb90\": rpc error: code = NotFound desc = could not find container \"37714a9885d5013797267376d3e7f865f72904ea538f01a35e946d9fbb52bb90\": container with ID starting with 37714a9885d5013797267376d3e7f865f72904ea538f01a35e946d9fbb52bb90 not found: ID does not exist" Oct 31 13:32:17.636518 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:17.636493 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h"] Oct 31 13:32:17.648582 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:17.648548 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-2mb7h"] Oct 31 13:32:18.818668 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:32:18.814901 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:32:19.239582 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:19.239546 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-4023ad-7994b685f7-9m2jh" podUID="bc39dc34-d1e4-466d-bc1d-5e77bac8dc39" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:32:19.420358 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:19.420326 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c08b4ba-3066-4b09-ba62-de15be6ccea2" path="/var/lib/kubelet/pods/8c08b4ba-3066-4b09-ba62-de15be6ccea2/volumes" Oct 31 13:32:19.615088 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:19.615048 5693 generic.go:356] "Generic (PLEG): container finished" podID="bc39dc34-d1e4-466d-bc1d-5e77bac8dc39" containerID="14d06494c6a1c4e9297ea00ecd34d28a7928cd274fbe50098db8b70b9853bd05" exitCode=0 Oct 31 13:32:19.615088 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:19.615136 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4023ad-7994b685f7-9m2jh" event={"ID":"bc39dc34-d1e4-466d-bc1d-5e77bac8dc39","Type":"ContainerDied","Data":"14d06494c6a1c4e9297ea00ecd34d28a7928cd274fbe50098db8b70b9853bd05"} Oct 31 13:32:19.709809 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:19.709777 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4023ad-7994b685f7-9m2jh" Oct 31 13:32:19.796411 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:19.795481 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc39dc34-d1e4-466d-bc1d-5e77bac8dc39-kube-api-access-gxfcg\" (UniqueName: \"kubernetes.io/projected/bc39dc34-d1e4-466d-bc1d-5e77bac8dc39-kube-api-access-gxfcg\") pod \"bc39dc34-d1e4-466d-bc1d-5e77bac8dc39\" (UID: \"bc39dc34-d1e4-466d-bc1d-5e77bac8dc39\") " Oct 31 13:32:19.796411 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:19.795620 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc39dc34-d1e4-466d-bc1d-5e77bac8dc39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc39dc34-d1e4-466d-bc1d-5e77bac8dc39-plugins-config-volume\") pod \"bc39dc34-d1e4-466d-bc1d-5e77bac8dc39\" (UID: \"bc39dc34-d1e4-466d-bc1d-5e77bac8dc39\") " Oct 31 13:32:19.796791 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:19.796770 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc39dc34-d1e4-466d-bc1d-5e77bac8dc39-plugins-config-volume" pod "bc39dc34-d1e4-466d-bc1d-5e77bac8dc39" (UID: "bc39dc34-d1e4-466d-bc1d-5e77bac8dc39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:32:19.799035 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:19.799012 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc39dc34-d1e4-466d-bc1d-5e77bac8dc39-kube-api-access-gxfcg" pod "bc39dc34-d1e4-466d-bc1d-5e77bac8dc39" (UID: "bc39dc34-d1e4-466d-bc1d-5e77bac8dc39"). InnerVolumeSpecName "kube-api-access-gxfcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:32:19.902692 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:19.898192 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxfcg\" (UniqueName: \"kubernetes.io/projected/bc39dc34-d1e4-466d-bc1d-5e77bac8dc39-kube-api-access-gxfcg\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:19.902692 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:19.898245 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc39dc34-d1e4-466d-bc1d-5e77bac8dc39-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:20.625456 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:20.625425 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4023ad-7994b685f7-9m2jh" Oct 31 13:32:20.625622 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:20.625452 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4023ad-7994b685f7-9m2jh" event={"ID":"bc39dc34-d1e4-466d-bc1d-5e77bac8dc39","Type":"ContainerDied","Data":"0c29280248f3f683c5b2e0feb0e4c1d18c472b66fb3cf6aa46a7fd93248e23a4"} Oct 31 13:32:20.625622 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:20.625514 5693 scope.go:122] "RemoveContainer" containerID="14d06494c6a1c4e9297ea00ecd34d28a7928cd274fbe50098db8b70b9853bd05" Oct 31 13:32:20.656032 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:20.656001 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4023ad-7994b685f7-9m2jh"] Oct 31 13:32:20.666314 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:20.666286 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4023ad-7994b685f7-9m2jh"] Oct 31 13:32:21.411517 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:21.411468 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc39dc34-d1e4-466d-bc1d-5e77bac8dc39" path="/var/lib/kubelet/pods/bc39dc34-d1e4-466d-bc1d-5e77bac8dc39/volumes" Oct 31 13:32:25.366983 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.366951 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl"] Oct 31 13:32:25.367946 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.367927 5693 cpu_manager.go:414] "containerMap: removing container" podUID="b8730031-f623-4f9e-ba5f-014340976d85" containerName="payload-processing" Oct 31 13:32:25.367946 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.367947 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8730031-f623-4f9e-ba5f-014340976d85" containerName="payload-processing" Oct 31 13:32:25.368050 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.367963 5693 cpu_manager.go:414] "containerMap: removing container" podUID="8c08b4ba-3066-4b09-ba62-de15be6ccea2" containerName="istio-proxy" Oct 31 13:32:25.368050 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.367968 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c08b4ba-3066-4b09-ba62-de15be6ccea2" containerName="istio-proxy" Oct 31 13:32:25.368050 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.367985 5693 cpu_manager.go:414] "containerMap: removing container" podUID="a9410570-a345-441e-9978-2f9ce48d7a09" containerName="istio-proxy" Oct 31 13:32:25.368050 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.367991 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9410570-a345-441e-9978-2f9ce48d7a09" containerName="istio-proxy" Oct 31 13:32:25.368050 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.367999 5693 cpu_manager.go:414] "containerMap: removing container" podUID="6791b24c-6569-46a0-af4c-3b7b7b356a9d" containerName="istio-proxy" Oct 31 13:32:25.368050 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.368003 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6791b24c-6569-46a0-af4c-3b7b7b356a9d" containerName="istio-proxy" Oct 31 13:32:25.368050 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.368031 5693 cpu_manager.go:414] "containerMap: removing container" podUID="c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6" containerName="istio-proxy" Oct 31 13:32:25.368050 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.368035 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6" containerName="istio-proxy" Oct 31 13:32:25.368050 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.368044 5693 cpu_manager.go:414] "containerMap: removing container" podUID="5cd6c998-975c-4628-a886-e9eeb20b3a64" containerName="payload-pre-processing" Oct 31 13:32:25.368050 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.368049 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cd6c998-975c-4628-a886-e9eeb20b3a64" containerName="payload-pre-processing" Oct 31 13:32:25.368352 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.368063 5693 cpu_manager.go:414] "containerMap: removing container" podUID="768af44a-2a4a-4254-a497-e2969f376cf7" containerName="main" Oct 31 13:32:25.368352 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.368067 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="768af44a-2a4a-4254-a497-e2969f376cf7" containerName="main" Oct 31 13:32:25.368352 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.368081 5693 cpu_manager.go:414] "containerMap: removing container" podUID="bc39dc34-d1e4-466d-bc1d-5e77bac8dc39" containerName="payload-processing" Oct 31 13:32:25.368352 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.368086 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc39dc34-d1e4-466d-bc1d-5e77bac8dc39" containerName="payload-processing" Oct 31 13:32:25.368352 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.368208 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="768af44a-2a4a-4254-a497-e2969f376cf7" containerName="main" Oct 31 13:32:25.368352 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.368217 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc39dc34-d1e4-466d-bc1d-5e77bac8dc39" containerName="payload-processing" Oct 31 13:32:25.368352 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.368223 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8730031-f623-4f9e-ba5f-014340976d85" containerName="payload-processing" Oct 31 13:32:25.368352 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.368232 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c08b4ba-3066-4b09-ba62-de15be6ccea2" containerName="istio-proxy" Oct 31 13:32:25.368352 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.368240 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9410570-a345-441e-9978-2f9ce48d7a09" containerName="istio-proxy" Oct 31 13:32:25.368352 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.368246 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="6791b24c-6569-46a0-af4c-3b7b7b356a9d" containerName="istio-proxy" Oct 31 13:32:25.368352 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.368256 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0e7ecf4-f00a-43f7-8c93-a5af5dfa35c6" containerName="istio-proxy" Oct 31 13:32:25.368352 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.368261 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cd6c998-975c-4628-a886-e9eeb20b3a64" containerName="payload-pre-processing" Oct 31 13:32:25.373578 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.373558 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl" Oct 31 13:32:25.380321 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.380296 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-50b4aa-serving-cert\"" Oct 31 13:32:25.384187 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.384159 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl"] Oct 31 13:32:25.530182 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.530143 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a02b81f9-b4ba-4a1e-a270-62bd75804349-openshift-service-ca\") pod \"maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl\" (UID: \"a02b81f9-b4ba-4a1e-a270-62bd75804349\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl" Oct 31 13:32:25.530394 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.530348 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a02b81f9-b4ba-4a1e-a270-62bd75804349-maas-api-tls\") pod \"maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl\" (UID: \"a02b81f9-b4ba-4a1e-a270-62bd75804349\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl" Oct 31 13:32:25.530561 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.530542 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqg88\" (UniqueName: \"kubernetes.io/projected/a02b81f9-b4ba-4a1e-a270-62bd75804349-kube-api-access-nqg88\") pod \"maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl\" (UID: \"a02b81f9-b4ba-4a1e-a270-62bd75804349\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl" Oct 31 13:32:25.530740 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.530720 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a02b81f9-b4ba-4a1e-a270-62bd75804349-metrics-certs\") pod \"maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl\" (UID: \"a02b81f9-b4ba-4a1e-a270-62bd75804349\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl" Oct 31 13:32:25.632123 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.632038 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqg88\" (UniqueName: \"kubernetes.io/projected/a02b81f9-b4ba-4a1e-a270-62bd75804349-kube-api-access-nqg88\") pod \"maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl\" (UID: \"a02b81f9-b4ba-4a1e-a270-62bd75804349\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl" Oct 31 13:32:25.632123 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.632118 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a02b81f9-b4ba-4a1e-a270-62bd75804349-metrics-certs\") pod \"maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl\" (UID: \"a02b81f9-b4ba-4a1e-a270-62bd75804349\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl" Oct 31 13:32:25.632329 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.632237 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a02b81f9-b4ba-4a1e-a270-62bd75804349-openshift-service-ca\") pod \"maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl\" (UID: \"a02b81f9-b4ba-4a1e-a270-62bd75804349\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl" Oct 31 13:32:25.632329 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.632297 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a02b81f9-b4ba-4a1e-a270-62bd75804349-maas-api-tls\") pod \"maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl\" (UID: \"a02b81f9-b4ba-4a1e-a270-62bd75804349\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl" Oct 31 13:32:25.633040 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.633017 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a02b81f9-b4ba-4a1e-a270-62bd75804349-openshift-service-ca\") pod \"maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl\" (UID: \"a02b81f9-b4ba-4a1e-a270-62bd75804349\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl" Oct 31 13:32:25.634796 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.634775 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a02b81f9-b4ba-4a1e-a270-62bd75804349-metrics-certs\") pod \"maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl\" (UID: \"a02b81f9-b4ba-4a1e-a270-62bd75804349\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl" Oct 31 13:32:25.634984 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.634961 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a02b81f9-b4ba-4a1e-a270-62bd75804349-maas-api-tls\") pod \"maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl\" (UID: \"a02b81f9-b4ba-4a1e-a270-62bd75804349\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl" Oct 31 13:32:25.644978 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.644955 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqg88\" (UniqueName: \"kubernetes.io/projected/a02b81f9-b4ba-4a1e-a270-62bd75804349-kube-api-access-nqg88\") pod \"maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl\" (UID: \"a02b81f9-b4ba-4a1e-a270-62bd75804349\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl" Oct 31 13:32:25.697814 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:25.693440 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl" Oct 31 13:32:26.018694 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:32:26.018635 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda02b81f9_b4ba_4a1e_a270_62bd75804349.slice/crio-35be075eb4ed12d6f894383c2e50f24dfd3f8a24db0c29156a0ae938ccd23fcf WatchSource:0}: Error finding container 35be075eb4ed12d6f894383c2e50f24dfd3f8a24db0c29156a0ae938ccd23fcf: Status 404 returned error can't find the container with id 35be075eb4ed12d6f894383c2e50f24dfd3f8a24db0c29156a0ae938ccd23fcf Oct 31 13:32:26.019543 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:26.019517 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl"] Oct 31 13:32:26.550420 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:26.550383 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-384dbe-6f9d5d675-qgdfx" podUID="dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:32:26.550420 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:26.550401 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-384dbe-57cd8fcd7-rsx6p" podUID="8903924e-49a1-456e-a26d-052f3a5901d8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:32:26.725584 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:26.725547 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl" event={"ID":"a02b81f9-b4ba-4a1e-a270-62bd75804349","Type":"ContainerStarted","Data":"dbe04374ad747eb66d0eb602e8090f70e9ca7b1181e5ed1f99c8ebd95591dfc7"} Oct 31 13:32:26.725861 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:26.725607 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl" event={"ID":"a02b81f9-b4ba-4a1e-a270-62bd75804349","Type":"ContainerStarted","Data":"35be075eb4ed12d6f894383c2e50f24dfd3f8a24db0c29156a0ae938ccd23fcf"} Oct 31 13:32:26.725861 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:26.725675 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl" Oct 31 13:32:26.760475 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:26.760381 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl" podStartSLOduration=1.2827897990000001 podStartE2EDuration="1.76036882s" totalImagesPullingTime="477.579021ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:32:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:32:26.756007724 +0000 UTC m=+3597.463594020" watchObservedRunningTime="2026-10-31 13:32:26.76036882 +0000 UTC m=+3597.467955115" Oct 31 13:32:28.325079 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:28.325051 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-384dbe-6f9d5d675-qgdfx" Oct 31 13:32:28.362205 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:28.362174 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-384dbe-57cd8fcd7-rsx6p" Oct 31 13:32:28.388546 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:28.387325 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8-kube-api-access-8db7l\" (UniqueName: \"kubernetes.io/projected/dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8-kube-api-access-8db7l\") pod \"dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8\" (UID: \"dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8\") " Oct 31 13:32:28.388546 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:28.387489 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8903924e-49a1-456e-a26d-052f3a5901d8-kube-api-access-69zlk\" (UniqueName: \"kubernetes.io/projected/8903924e-49a1-456e-a26d-052f3a5901d8-kube-api-access-69zlk\") pod \"8903924e-49a1-456e-a26d-052f3a5901d8\" (UID: \"8903924e-49a1-456e-a26d-052f3a5901d8\") " Oct 31 13:32:28.388546 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:28.387578 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8-plugins-config-volume\") pod \"dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8\" (UID: \"dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8\") " Oct 31 13:32:28.388546 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:28.387633 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8903924e-49a1-456e-a26d-052f3a5901d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8903924e-49a1-456e-a26d-052f3a5901d8-plugins-config-volume\") pod \"8903924e-49a1-456e-a26d-052f3a5901d8\" (UID: \"8903924e-49a1-456e-a26d-052f3a5901d8\") " Oct 31 13:32:28.389231 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:28.389205 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8903924e-49a1-456e-a26d-052f3a5901d8-plugins-config-volume" pod "8903924e-49a1-456e-a26d-052f3a5901d8" (UID: "8903924e-49a1-456e-a26d-052f3a5901d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:32:28.390366 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:28.390009 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8-plugins-config-volume" pod "dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8" (UID: "dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:32:28.393690 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:28.392440 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8903924e-49a1-456e-a26d-052f3a5901d8-kube-api-access-69zlk" pod "8903924e-49a1-456e-a26d-052f3a5901d8" (UID: "8903924e-49a1-456e-a26d-052f3a5901d8"). InnerVolumeSpecName "kube-api-access-69zlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:32:28.394108 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:28.394084 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8-kube-api-access-8db7l" pod "dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8" (UID: "dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8"). InnerVolumeSpecName "kube-api-access-8db7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:32:28.490220 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:28.490183 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8db7l\" (UniqueName: \"kubernetes.io/projected/dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8-kube-api-access-8db7l\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:28.490220 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:28.490215 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69zlk\" (UniqueName: \"kubernetes.io/projected/8903924e-49a1-456e-a26d-052f3a5901d8-kube-api-access-69zlk\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:28.490220 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:28.490225 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:28.490525 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:28.490234 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8903924e-49a1-456e-a26d-052f3a5901d8-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:28.746507 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:28.746473 5693 generic.go:356] "Generic (PLEG): container finished" podID="8903924e-49a1-456e-a26d-052f3a5901d8" containerID="62ec82ee6807326506d56993dfc66e8f49c4f27cb476a47c486067965784b1f4" exitCode=0 Oct 31 13:32:28.746710 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:28.746518 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-384dbe-57cd8fcd7-rsx6p" Oct 31 13:32:28.746710 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:28.746557 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-384dbe-57cd8fcd7-rsx6p" event={"ID":"8903924e-49a1-456e-a26d-052f3a5901d8","Type":"ContainerDied","Data":"62ec82ee6807326506d56993dfc66e8f49c4f27cb476a47c486067965784b1f4"} Oct 31 13:32:28.746710 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:28.746612 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-384dbe-57cd8fcd7-rsx6p" event={"ID":"8903924e-49a1-456e-a26d-052f3a5901d8","Type":"ContainerDied","Data":"8eceab6b1c750da53a9dd242958c5f2b36149747523fb64743255a1a9001c380"} Oct 31 13:32:28.746710 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:28.746634 5693 scope.go:122] "RemoveContainer" containerID="62ec82ee6807326506d56993dfc66e8f49c4f27cb476a47c486067965784b1f4" Oct 31 13:32:28.748477 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:28.748448 5693 generic.go:356] "Generic (PLEG): container finished" podID="dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8" containerID="7ec67c3e102d4ea3aee7ee6f26dc18293867defc4089592582d204c331331530" exitCode=0 Oct 31 13:32:28.748606 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:28.748492 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-384dbe-6f9d5d675-qgdfx" Oct 31 13:32:28.748606 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:28.748500 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-384dbe-6f9d5d675-qgdfx" event={"ID":"dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8","Type":"ContainerDied","Data":"7ec67c3e102d4ea3aee7ee6f26dc18293867defc4089592582d204c331331530"} Oct 31 13:32:28.748606 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:28.748542 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-384dbe-6f9d5d675-qgdfx" event={"ID":"dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8","Type":"ContainerDied","Data":"2ea141dcf7d7202e581a8a0058dc4397def3ab8b221457a840a3e38f115172f0"} Oct 31 13:32:28.767036 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:28.767011 5693 scope.go:122] "RemoveContainer" containerID="62ec82ee6807326506d56993dfc66e8f49c4f27cb476a47c486067965784b1f4" Oct 31 13:32:28.767441 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:32:28.767411 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62ec82ee6807326506d56993dfc66e8f49c4f27cb476a47c486067965784b1f4\": container with ID starting with 62ec82ee6807326506d56993dfc66e8f49c4f27cb476a47c486067965784b1f4 not found: ID does not exist" containerID="62ec82ee6807326506d56993dfc66e8f49c4f27cb476a47c486067965784b1f4" Oct 31 13:32:28.767570 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:28.767447 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62ec82ee6807326506d56993dfc66e8f49c4f27cb476a47c486067965784b1f4"} err="failed to get container status \"62ec82ee6807326506d56993dfc66e8f49c4f27cb476a47c486067965784b1f4\": rpc error: code = NotFound desc = could not find container \"62ec82ee6807326506d56993dfc66e8f49c4f27cb476a47c486067965784b1f4\": container with ID starting with 62ec82ee6807326506d56993dfc66e8f49c4f27cb476a47c486067965784b1f4 not found: ID does not exist" Oct 31 13:32:28.767570 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:28.767468 5693 scope.go:122] "RemoveContainer" containerID="7ec67c3e102d4ea3aee7ee6f26dc18293867defc4089592582d204c331331530" Oct 31 13:32:28.783684 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:28.783661 5693 scope.go:122] "RemoveContainer" containerID="7ec67c3e102d4ea3aee7ee6f26dc18293867defc4089592582d204c331331530" Oct 31 13:32:28.784134 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:32:28.784111 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ec67c3e102d4ea3aee7ee6f26dc18293867defc4089592582d204c331331530\": container with ID starting with 7ec67c3e102d4ea3aee7ee6f26dc18293867defc4089592582d204c331331530 not found: ID does not exist" containerID="7ec67c3e102d4ea3aee7ee6f26dc18293867defc4089592582d204c331331530" Oct 31 13:32:28.784230 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:28.784145 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ec67c3e102d4ea3aee7ee6f26dc18293867defc4089592582d204c331331530"} err="failed to get container status \"7ec67c3e102d4ea3aee7ee6f26dc18293867defc4089592582d204c331331530\": rpc error: code = NotFound desc = could not find container \"7ec67c3e102d4ea3aee7ee6f26dc18293867defc4089592582d204c331331530\": container with ID starting with 7ec67c3e102d4ea3aee7ee6f26dc18293867defc4089592582d204c331331530 not found: ID does not exist" Oct 31 13:32:28.786456 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:28.786434 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-384dbe-6f9d5d675-qgdfx"] Oct 31 13:32:28.791936 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:28.791915 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-384dbe-6f9d5d675-qgdfx"] Oct 31 13:32:28.800346 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:28.800190 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-384dbe-57cd8fcd7-rsx6p"] Oct 31 13:32:28.804630 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:28.804607 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-384dbe-57cd8fcd7-rsx6p"] Oct 31 13:32:29.413586 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:29.413554 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8903924e-49a1-456e-a26d-052f3a5901d8" path="/var/lib/kubelet/pods/8903924e-49a1-456e-a26d-052f3a5901d8/volumes" Oct 31 13:32:29.414039 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:29.414024 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8" path="/var/lib/kubelet/pods/dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8/volumes" Oct 31 13:32:29.978200 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:29.978165 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-384dbe-674f58d889-lqz29"] Oct 31 13:32:29.978401 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:29.978370 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-384dbe-674f58d889-lqz29" podUID="50a97ab4-3037-46d1-9cc5-68cc95955e9f" containerName="maas-api" containerID="cri-o://11ac8f935bfc0e56753855e6a58d7e252cf97258f9d2efb524524a2fe0fd7bb1" gracePeriod=30 Oct 31 13:32:30.426778 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:30.426734 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-384dbe-674f58d889-lqz29" Oct 31 13:32:30.540765 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:30.540732 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50a97ab4-3037-46d1-9cc5-68cc95955e9f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50a97ab4-3037-46d1-9cc5-68cc95955e9f-maas-api-tls\") pod \"50a97ab4-3037-46d1-9cc5-68cc95955e9f\" (UID: \"50a97ab4-3037-46d1-9cc5-68cc95955e9f\") " Oct 31 13:32:30.540958 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:30.540855 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50a97ab4-3037-46d1-9cc5-68cc95955e9f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/50a97ab4-3037-46d1-9cc5-68cc95955e9f-metrics-certs\") pod \"50a97ab4-3037-46d1-9cc5-68cc95955e9f\" (UID: \"50a97ab4-3037-46d1-9cc5-68cc95955e9f\") " Oct 31 13:32:30.541029 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:30.541008 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50a97ab4-3037-46d1-9cc5-68cc95955e9f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50a97ab4-3037-46d1-9cc5-68cc95955e9f-openshift-service-ca\") pod \"50a97ab4-3037-46d1-9cc5-68cc95955e9f\" (UID: \"50a97ab4-3037-46d1-9cc5-68cc95955e9f\") " Oct 31 13:32:30.541127 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:30.541111 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50a97ab4-3037-46d1-9cc5-68cc95955e9f-kube-api-access-d7fdr\" (UniqueName: \"kubernetes.io/projected/50a97ab4-3037-46d1-9cc5-68cc95955e9f-kube-api-access-d7fdr\") pod \"50a97ab4-3037-46d1-9cc5-68cc95955e9f\" (UID: \"50a97ab4-3037-46d1-9cc5-68cc95955e9f\") " Oct 31 13:32:30.541421 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:30.541392 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50a97ab4-3037-46d1-9cc5-68cc95955e9f-openshift-service-ca" pod "50a97ab4-3037-46d1-9cc5-68cc95955e9f" (UID: "50a97ab4-3037-46d1-9cc5-68cc95955e9f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:32:30.542132 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:30.542108 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50a97ab4-3037-46d1-9cc5-68cc95955e9f-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:30.543312 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:30.543287 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50a97ab4-3037-46d1-9cc5-68cc95955e9f-maas-api-tls" pod "50a97ab4-3037-46d1-9cc5-68cc95955e9f" (UID: "50a97ab4-3037-46d1-9cc5-68cc95955e9f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:32:30.543312 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:30.543304 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50a97ab4-3037-46d1-9cc5-68cc95955e9f-metrics-certs" pod "50a97ab4-3037-46d1-9cc5-68cc95955e9f" (UID: "50a97ab4-3037-46d1-9cc5-68cc95955e9f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:32:30.543488 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:30.543468 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50a97ab4-3037-46d1-9cc5-68cc95955e9f-kube-api-access-d7fdr" pod "50a97ab4-3037-46d1-9cc5-68cc95955e9f" (UID: "50a97ab4-3037-46d1-9cc5-68cc95955e9f"). InnerVolumeSpecName "kube-api-access-d7fdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:32:30.643824 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:30.643787 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7fdr\" (UniqueName: \"kubernetes.io/projected/50a97ab4-3037-46d1-9cc5-68cc95955e9f-kube-api-access-d7fdr\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:30.643824 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:30.643818 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50a97ab4-3037-46d1-9cc5-68cc95955e9f-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:30.643824 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:30.643827 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50a97ab4-3037-46d1-9cc5-68cc95955e9f-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:30.773629 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:30.773587 5693 generic.go:356] "Generic (PLEG): container finished" podID="50a97ab4-3037-46d1-9cc5-68cc95955e9f" containerID="11ac8f935bfc0e56753855e6a58d7e252cf97258f9d2efb524524a2fe0fd7bb1" exitCode=0 Oct 31 13:32:30.773855 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:30.773688 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-384dbe-674f58d889-lqz29" Oct 31 13:32:30.773855 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:30.773690 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-384dbe-674f58d889-lqz29" event={"ID":"50a97ab4-3037-46d1-9cc5-68cc95955e9f","Type":"ContainerDied","Data":"11ac8f935bfc0e56753855e6a58d7e252cf97258f9d2efb524524a2fe0fd7bb1"} Oct 31 13:32:30.773855 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:30.773756 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-384dbe-674f58d889-lqz29" event={"ID":"50a97ab4-3037-46d1-9cc5-68cc95955e9f","Type":"ContainerDied","Data":"e9182135b45e4560d716381ff5b1743b2a4e249d1e2bee18ad96c1acae16a549"} Oct 31 13:32:30.773855 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:30.773779 5693 scope.go:122] "RemoveContainer" containerID="11ac8f935bfc0e56753855e6a58d7e252cf97258f9d2efb524524a2fe0fd7bb1" Oct 31 13:32:30.788006 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:30.787986 5693 scope.go:122] "RemoveContainer" containerID="11ac8f935bfc0e56753855e6a58d7e252cf97258f9d2efb524524a2fe0fd7bb1" Oct 31 13:32:30.788363 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:32:30.788327 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11ac8f935bfc0e56753855e6a58d7e252cf97258f9d2efb524524a2fe0fd7bb1\": container with ID starting with 11ac8f935bfc0e56753855e6a58d7e252cf97258f9d2efb524524a2fe0fd7bb1 not found: ID does not exist" containerID="11ac8f935bfc0e56753855e6a58d7e252cf97258f9d2efb524524a2fe0fd7bb1" Oct 31 13:32:30.788494 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:30.788357 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11ac8f935bfc0e56753855e6a58d7e252cf97258f9d2efb524524a2fe0fd7bb1"} err="failed to get container status \"11ac8f935bfc0e56753855e6a58d7e252cf97258f9d2efb524524a2fe0fd7bb1\": rpc error: code = NotFound desc = could not find container \"11ac8f935bfc0e56753855e6a58d7e252cf97258f9d2efb524524a2fe0fd7bb1\": container with ID starting with 11ac8f935bfc0e56753855e6a58d7e252cf97258f9d2efb524524a2fe0fd7bb1 not found: ID does not exist" Oct 31 13:32:30.801835 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:30.801808 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-384dbe-674f58d889-lqz29"] Oct 31 13:32:30.820157 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:30.820128 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-384dbe-674f58d889-lqz29"] Oct 31 13:32:31.411677 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:31.411598 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50a97ab4-3037-46d1-9cc5-68cc95955e9f" path="/var/lib/kubelet/pods/50a97ab4-3037-46d1-9cc5-68cc95955e9f/volumes" Oct 31 13:32:32.742236 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:32.742207 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl" Oct 31 13:32:44.429696 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:44.429567 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb"] Oct 31 13:32:44.430900 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:44.429814 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb" podUID="a67d0a96-c7c6-416a-9569-bf8905dc6c0c" containerName="maas-api" containerID="cri-o://6d5fd54a2c0f63e68f79da2779f0679c63995c4315c543211f82c317a2ee9e42" gracePeriod=30 Oct 31 13:32:44.952065 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:44.952029 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb" Oct 31 13:32:44.968992 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:44.968959 5693 generic.go:356] "Generic (PLEG): container finished" podID="a67d0a96-c7c6-416a-9569-bf8905dc6c0c" containerID="6d5fd54a2c0f63e68f79da2779f0679c63995c4315c543211f82c317a2ee9e42" exitCode=0 Oct 31 13:32:44.969262 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:44.969024 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb" Oct 31 13:32:44.969262 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:44.969054 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb" event={"ID":"a67d0a96-c7c6-416a-9569-bf8905dc6c0c","Type":"ContainerDied","Data":"6d5fd54a2c0f63e68f79da2779f0679c63995c4315c543211f82c317a2ee9e42"} Oct 31 13:32:44.969262 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:44.969109 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb" event={"ID":"a67d0a96-c7c6-416a-9569-bf8905dc6c0c","Type":"ContainerDied","Data":"2bec6739e06f82a17c388d794419a3052b53cf9a8bc5a6c62e3e492f2eb49a34"} Oct 31 13:32:44.969262 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:44.969144 5693 scope.go:122] "RemoveContainer" containerID="6d5fd54a2c0f63e68f79da2779f0679c63995c4315c543211f82c317a2ee9e42" Oct 31 13:32:44.981520 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:44.981493 5693 scope.go:122] "RemoveContainer" containerID="6d5fd54a2c0f63e68f79da2779f0679c63995c4315c543211f82c317a2ee9e42" Oct 31 13:32:44.984865 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:32:44.984820 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d5fd54a2c0f63e68f79da2779f0679c63995c4315c543211f82c317a2ee9e42\": container with ID starting with 6d5fd54a2c0f63e68f79da2779f0679c63995c4315c543211f82c317a2ee9e42 not found: ID does not exist" containerID="6d5fd54a2c0f63e68f79da2779f0679c63995c4315c543211f82c317a2ee9e42" Oct 31 13:32:44.984999 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:44.984864 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d5fd54a2c0f63e68f79da2779f0679c63995c4315c543211f82c317a2ee9e42"} err="failed to get container status \"6d5fd54a2c0f63e68f79da2779f0679c63995c4315c543211f82c317a2ee9e42\": rpc error: code = NotFound desc = could not find container \"6d5fd54a2c0f63e68f79da2779f0679c63995c4315c543211f82c317a2ee9e42\": container with ID starting with 6d5fd54a2c0f63e68f79da2779f0679c63995c4315c543211f82c317a2ee9e42 not found: ID does not exist" Oct 31 13:32:45.067328 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:45.067298 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a67d0a96-c7c6-416a-9569-bf8905dc6c0c-kube-api-access-57wpr\" (UniqueName: \"kubernetes.io/projected/a67d0a96-c7c6-416a-9569-bf8905dc6c0c-kube-api-access-57wpr\") pod \"a67d0a96-c7c6-416a-9569-bf8905dc6c0c\" (UID: \"a67d0a96-c7c6-416a-9569-bf8905dc6c0c\") " Oct 31 13:32:45.067539 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:45.067394 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a67d0a96-c7c6-416a-9569-bf8905dc6c0c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a67d0a96-c7c6-416a-9569-bf8905dc6c0c-maas-api-tls\") pod \"a67d0a96-c7c6-416a-9569-bf8905dc6c0c\" (UID: \"a67d0a96-c7c6-416a-9569-bf8905dc6c0c\") " Oct 31 13:32:45.067539 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:45.067490 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a67d0a96-c7c6-416a-9569-bf8905dc6c0c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a67d0a96-c7c6-416a-9569-bf8905dc6c0c-metrics-certs\") pod \"a67d0a96-c7c6-416a-9569-bf8905dc6c0c\" (UID: \"a67d0a96-c7c6-416a-9569-bf8905dc6c0c\") " Oct 31 13:32:45.067784 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:45.067767 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a67d0a96-c7c6-416a-9569-bf8905dc6c0c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a67d0a96-c7c6-416a-9569-bf8905dc6c0c-openshift-service-ca\") pod \"a67d0a96-c7c6-416a-9569-bf8905dc6c0c\" (UID: \"a67d0a96-c7c6-416a-9569-bf8905dc6c0c\") " Oct 31 13:32:45.068390 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:45.068363 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a67d0a96-c7c6-416a-9569-bf8905dc6c0c-openshift-service-ca" pod "a67d0a96-c7c6-416a-9569-bf8905dc6c0c" (UID: "a67d0a96-c7c6-416a-9569-bf8905dc6c0c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:32:45.070068 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:45.070011 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a67d0a96-c7c6-416a-9569-bf8905dc6c0c-metrics-certs" pod "a67d0a96-c7c6-416a-9569-bf8905dc6c0c" (UID: "a67d0a96-c7c6-416a-9569-bf8905dc6c0c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:32:45.070218 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:45.070195 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a67d0a96-c7c6-416a-9569-bf8905dc6c0c-kube-api-access-57wpr" pod "a67d0a96-c7c6-416a-9569-bf8905dc6c0c" (UID: "a67d0a96-c7c6-416a-9569-bf8905dc6c0c"). InnerVolumeSpecName "kube-api-access-57wpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:32:45.070218 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:45.070209 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a67d0a96-c7c6-416a-9569-bf8905dc6c0c-maas-api-tls" pod "a67d0a96-c7c6-416a-9569-bf8905dc6c0c" (UID: "a67d0a96-c7c6-416a-9569-bf8905dc6c0c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:32:45.170379 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:45.170302 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a67d0a96-c7c6-416a-9569-bf8905dc6c0c-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:45.170379 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:45.170329 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57wpr\" (UniqueName: \"kubernetes.io/projected/a67d0a96-c7c6-416a-9569-bf8905dc6c0c-kube-api-access-57wpr\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:45.170379 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:45.170340 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a67d0a96-c7c6-416a-9569-bf8905dc6c0c-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:45.170379 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:45.170348 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a67d0a96-c7c6-416a-9569-bf8905dc6c0c-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:45.337827 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:45.337792 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb"] Oct 31 13:32:45.348299 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:45.348267 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c6c445-59f5d9c6cd-zd4cb"] Oct 31 13:32:45.411069 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:45.411026 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a67d0a96-c7c6-416a-9569-bf8905dc6c0c" path="/var/lib/kubelet/pods/a67d0a96-c7c6-416a-9569-bf8905dc6c0c/volumes" Oct 31 13:32:48.689782 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:32:48.689746 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:32:59.443471 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:59.443435 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-09db67/test-model-09db67-kserve-7478f9db55-gz5h5"] Oct 31 13:32:59.444855 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:59.444830 5693 cpu_manager.go:414] "containerMap: removing container" podUID="8903924e-49a1-456e-a26d-052f3a5901d8" containerName="payload-pre-processing" Oct 31 13:32:59.444927 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:59.444859 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8903924e-49a1-456e-a26d-052f3a5901d8" containerName="payload-pre-processing" Oct 31 13:32:59.444927 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:59.444902 5693 cpu_manager.go:414] "containerMap: removing container" podUID="50a97ab4-3037-46d1-9cc5-68cc95955e9f" containerName="maas-api" Oct 31 13:32:59.444927 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:59.444911 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50a97ab4-3037-46d1-9cc5-68cc95955e9f" containerName="maas-api" Oct 31 13:32:59.445027 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:59.444944 5693 cpu_manager.go:414] "containerMap: removing container" podUID="8c08b4ba-3066-4b09-ba62-de15be6ccea2" containerName="istio-proxy" Oct 31 13:32:59.445027 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:59.444951 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c08b4ba-3066-4b09-ba62-de15be6ccea2" containerName="istio-proxy" Oct 31 13:32:59.445027 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:59.444987 5693 cpu_manager.go:414] "containerMap: removing container" podUID="dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8" containerName="payload-processing" Oct 31 13:32:59.445027 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:59.444994 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8" containerName="payload-processing" Oct 31 13:32:59.445027 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:59.445006 5693 cpu_manager.go:414] "containerMap: removing container" podUID="a67d0a96-c7c6-416a-9569-bf8905dc6c0c" containerName="maas-api" Oct 31 13:32:59.445027 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:59.445012 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a67d0a96-c7c6-416a-9569-bf8905dc6c0c" containerName="maas-api" Oct 31 13:32:59.445218 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:59.445186 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="50a97ab4-3037-46d1-9cc5-68cc95955e9f" containerName="maas-api" Oct 31 13:32:59.445218 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:59.445206 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c08b4ba-3066-4b09-ba62-de15be6ccea2" containerName="istio-proxy" Oct 31 13:32:59.445289 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:59.445222 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc81d1ea-3772-4bbe-9ec1-c28bae8e13a8" containerName="payload-processing" Oct 31 13:32:59.445289 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:59.445234 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="a67d0a96-c7c6-416a-9569-bf8905dc6c0c" containerName="maas-api" Oct 31 13:32:59.445289 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:59.445250 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="8903924e-49a1-456e-a26d-052f3a5901d8" containerName="payload-pre-processing" Oct 31 13:32:59.450848 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:59.450821 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-09db67/test-model-09db67-kserve-7478f9db55-gz5h5" Oct 31 13:32:59.465728 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:59.465479 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-09db67/test-model-09db67-kserve-7478f9db55-gz5h5"] Oct 31 13:32:59.468474 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:59.468451 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-09db67\"/\"default-dockercfg-p8rkw\"" Oct 31 13:32:59.468664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:59.468545 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-09db67\"/\"test-model-09db67-kserve-self-signed-certs\"" Oct 31 13:32:59.529163 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:59.529136 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-09db67\"/\"kube-root-ca.crt\"" Oct 31 13:32:59.529408 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:59.529387 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-09db67\"/\"openshift-service-ca.crt\"" Oct 31 13:32:59.607584 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:59.607550 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtlcq\" (UniqueName: \"kubernetes.io/projected/cb838e2b-f206-4da9-97bc-a160de56256f-kube-api-access-mtlcq\") pod \"test-model-09db67-kserve-7478f9db55-gz5h5\" (UID: \"cb838e2b-f206-4da9-97bc-a160de56256f\") " pod="ai-tenant-e2e-inf-a-09db67/test-model-09db67-kserve-7478f9db55-gz5h5" Oct 31 13:32:59.607834 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:59.607626 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb838e2b-f206-4da9-97bc-a160de56256f-model-cache\") pod \"test-model-09db67-kserve-7478f9db55-gz5h5\" (UID: \"cb838e2b-f206-4da9-97bc-a160de56256f\") " pod="ai-tenant-e2e-inf-a-09db67/test-model-09db67-kserve-7478f9db55-gz5h5" Oct 31 13:32:59.607834 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:59.607809 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb838e2b-f206-4da9-97bc-a160de56256f-dshm\") pod \"test-model-09db67-kserve-7478f9db55-gz5h5\" (UID: \"cb838e2b-f206-4da9-97bc-a160de56256f\") " pod="ai-tenant-e2e-inf-a-09db67/test-model-09db67-kserve-7478f9db55-gz5h5" Oct 31 13:32:59.607982 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:59.607911 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb838e2b-f206-4da9-97bc-a160de56256f-tls-certs\") pod \"test-model-09db67-kserve-7478f9db55-gz5h5\" (UID: \"cb838e2b-f206-4da9-97bc-a160de56256f\") " pod="ai-tenant-e2e-inf-a-09db67/test-model-09db67-kserve-7478f9db55-gz5h5" Oct 31 13:32:59.608169 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:59.608133 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb838e2b-f206-4da9-97bc-a160de56256f-home\") pod \"test-model-09db67-kserve-7478f9db55-gz5h5\" (UID: \"cb838e2b-f206-4da9-97bc-a160de56256f\") " pod="ai-tenant-e2e-inf-a-09db67/test-model-09db67-kserve-7478f9db55-gz5h5" Oct 31 13:32:59.608321 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:59.608214 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb838e2b-f206-4da9-97bc-a160de56256f-tmp-dir\") pod \"test-model-09db67-kserve-7478f9db55-gz5h5\" (UID: \"cb838e2b-f206-4da9-97bc-a160de56256f\") " pod="ai-tenant-e2e-inf-a-09db67/test-model-09db67-kserve-7478f9db55-gz5h5" Oct 31 13:32:59.723883 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:59.716766 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb838e2b-f206-4da9-97bc-a160de56256f-tmp-dir\") pod \"test-model-09db67-kserve-7478f9db55-gz5h5\" (UID: \"cb838e2b-f206-4da9-97bc-a160de56256f\") " pod="ai-tenant-e2e-inf-a-09db67/test-model-09db67-kserve-7478f9db55-gz5h5" Oct 31 13:32:59.723883 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:59.716964 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtlcq\" (UniqueName: \"kubernetes.io/projected/cb838e2b-f206-4da9-97bc-a160de56256f-kube-api-access-mtlcq\") pod \"test-model-09db67-kserve-7478f9db55-gz5h5\" (UID: \"cb838e2b-f206-4da9-97bc-a160de56256f\") " pod="ai-tenant-e2e-inf-a-09db67/test-model-09db67-kserve-7478f9db55-gz5h5" Oct 31 13:32:59.723883 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:59.717025 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb838e2b-f206-4da9-97bc-a160de56256f-model-cache\") pod \"test-model-09db67-kserve-7478f9db55-gz5h5\" (UID: \"cb838e2b-f206-4da9-97bc-a160de56256f\") " pod="ai-tenant-e2e-inf-a-09db67/test-model-09db67-kserve-7478f9db55-gz5h5" Oct 31 13:32:59.723883 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:59.717262 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb838e2b-f206-4da9-97bc-a160de56256f-dshm\") pod \"test-model-09db67-kserve-7478f9db55-gz5h5\" (UID: \"cb838e2b-f206-4da9-97bc-a160de56256f\") " pod="ai-tenant-e2e-inf-a-09db67/test-model-09db67-kserve-7478f9db55-gz5h5" Oct 31 13:32:59.723883 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:59.717369 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb838e2b-f206-4da9-97bc-a160de56256f-tls-certs\") pod \"test-model-09db67-kserve-7478f9db55-gz5h5\" (UID: \"cb838e2b-f206-4da9-97bc-a160de56256f\") " pod="ai-tenant-e2e-inf-a-09db67/test-model-09db67-kserve-7478f9db55-gz5h5" Oct 31 13:32:59.723883 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:59.717523 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb838e2b-f206-4da9-97bc-a160de56256f-home\") pod \"test-model-09db67-kserve-7478f9db55-gz5h5\" (UID: \"cb838e2b-f206-4da9-97bc-a160de56256f\") " pod="ai-tenant-e2e-inf-a-09db67/test-model-09db67-kserve-7478f9db55-gz5h5" Oct 31 13:32:59.723883 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:59.718071 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb838e2b-f206-4da9-97bc-a160de56256f-home\") pod \"test-model-09db67-kserve-7478f9db55-gz5h5\" (UID: \"cb838e2b-f206-4da9-97bc-a160de56256f\") " pod="ai-tenant-e2e-inf-a-09db67/test-model-09db67-kserve-7478f9db55-gz5h5" Oct 31 13:32:59.731662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:59.727465 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb838e2b-f206-4da9-97bc-a160de56256f-model-cache\") pod \"test-model-09db67-kserve-7478f9db55-gz5h5\" (UID: \"cb838e2b-f206-4da9-97bc-a160de56256f\") " pod="ai-tenant-e2e-inf-a-09db67/test-model-09db67-kserve-7478f9db55-gz5h5" Oct 31 13:32:59.731662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:59.727597 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb838e2b-f206-4da9-97bc-a160de56256f-tmp-dir\") pod \"test-model-09db67-kserve-7478f9db55-gz5h5\" (UID: \"cb838e2b-f206-4da9-97bc-a160de56256f\") " pod="ai-tenant-e2e-inf-a-09db67/test-model-09db67-kserve-7478f9db55-gz5h5" Oct 31 13:32:59.735373 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:59.735302 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb838e2b-f206-4da9-97bc-a160de56256f-tls-certs\") pod \"test-model-09db67-kserve-7478f9db55-gz5h5\" (UID: \"cb838e2b-f206-4da9-97bc-a160de56256f\") " pod="ai-tenant-e2e-inf-a-09db67/test-model-09db67-kserve-7478f9db55-gz5h5" Oct 31 13:32:59.748951 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:59.741282 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb838e2b-f206-4da9-97bc-a160de56256f-dshm\") pod \"test-model-09db67-kserve-7478f9db55-gz5h5\" (UID: \"cb838e2b-f206-4da9-97bc-a160de56256f\") " pod="ai-tenant-e2e-inf-a-09db67/test-model-09db67-kserve-7478f9db55-gz5h5" Oct 31 13:32:59.787334 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:32:59.787301 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtlcq\" (UniqueName: \"kubernetes.io/projected/cb838e2b-f206-4da9-97bc-a160de56256f-kube-api-access-mtlcq\") pod \"test-model-09db67-kserve-7478f9db55-gz5h5\" (UID: \"cb838e2b-f206-4da9-97bc-a160de56256f\") " pod="ai-tenant-e2e-inf-a-09db67/test-model-09db67-kserve-7478f9db55-gz5h5" Oct 31 13:33:00.076383 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:00.076288 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-09db67/test-model-09db67-kserve-7478f9db55-gz5h5" Oct 31 13:33:00.370231 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:33:00.370200 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb838e2b_f206_4da9_97bc_a160de56256f.slice/crio-f18cf750031d1b4d8ad0f5451849ffdd1eb54a80607916c9f812331c8c8e3de2 WatchSource:0}: Error finding container f18cf750031d1b4d8ad0f5451849ffdd1eb54a80607916c9f812331c8c8e3de2: Status 404 returned error can't find the container with id f18cf750031d1b4d8ad0f5451849ffdd1eb54a80607916c9f812331c8c8e3de2 Oct 31 13:33:00.370387 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:00.370318 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-09db67/test-model-09db67-kserve-7478f9db55-gz5h5"] Oct 31 13:33:01.120389 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:01.120321 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-09db67/test-model-09db67-kserve-7478f9db55-gz5h5" event={"ID":"cb838e2b-f206-4da9-97bc-a160de56256f","Type":"ContainerStarted","Data":"c88488c7133599370b30b2710f1396014636d853148ce28b02bb3fe5e5c335a1"} Oct 31 13:33:01.120389 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:01.120377 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-09db67/test-model-09db67-kserve-7478f9db55-gz5h5" event={"ID":"cb838e2b-f206-4da9-97bc-a160de56256f","Type":"ContainerStarted","Data":"f18cf750031d1b4d8ad0f5451849ffdd1eb54a80607916c9f812331c8c8e3de2"} Oct 31 13:33:01.144088 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:01.144032 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-09db67/test-model-09db67-kserve-7478f9db55-gz5h5" podStartSLOduration=2.144016011 podStartE2EDuration="2.144016011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:32:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:33:01.139381974 +0000 UTC m=+3631.846968268" watchObservedRunningTime="2026-10-31 13:33:01.144016011 +0000 UTC m=+3631.851602306" Oct 31 13:33:02.077175 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:02.077144 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-09db67/test-model-09db67-kserve-7478f9db55-gz5h5" Oct 31 13:33:10.076796 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:10.076761 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-09db67/test-model-09db67-kserve-7478f9db55-gz5h5" Oct 31 13:33:10.091499 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:10.091469 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-09db67/test-model-09db67-kserve-7478f9db55-gz5h5" Oct 31 13:33:10.219652 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:10.219610 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-09db67/test-model-09db67-kserve-7478f9db55-gz5h5" Oct 31 13:33:18.689955 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:33:18.689917 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:33:21.582906 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:21.582875 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2"] Oct 31 13:33:21.583321 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:21.583065 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" podUID="af589ec1-de60-43ff-9bbb-0b298d9c99cc" containerName="istio-proxy" containerID="cri-o://d93345d85a287d60125356684d0bc61b194ee595c01ec4fa30d4af9f720ab9c5" gracePeriod=30 Oct 31 13:33:26.927020 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:26.926991 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" Oct 31 13:33:27.026097 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:27.026064 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af589ec1-de60-43ff-9bbb-0b298d9c99cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af589ec1-de60-43ff-9bbb-0b298d9c99cc-workload-socket\") pod \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\" (UID: \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\") " Oct 31 13:33:27.026378 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:27.026118 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af589ec1-de60-43ff-9bbb-0b298d9c99cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af589ec1-de60-43ff-9bbb-0b298d9c99cc-istio-podinfo\") pod \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\" (UID: \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\") " Oct 31 13:33:27.026378 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:27.026245 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af589ec1-de60-43ff-9bbb-0b298d9c99cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af589ec1-de60-43ff-9bbb-0b298d9c99cc-istio-envoy\") pod \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\" (UID: \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\") " Oct 31 13:33:27.026378 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:27.026289 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af589ec1-de60-43ff-9bbb-0b298d9c99cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af589ec1-de60-43ff-9bbb-0b298d9c99cc-istio-data\") pod \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\" (UID: \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\") " Oct 31 13:33:27.026378 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:27.026337 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af589ec1-de60-43ff-9bbb-0b298d9c99cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af589ec1-de60-43ff-9bbb-0b298d9c99cc-istiod-ca-cert\") pod \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\" (UID: \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\") " Oct 31 13:33:27.026619 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:27.026392 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af589ec1-de60-43ff-9bbb-0b298d9c99cc-workload-socket" pod "af589ec1-de60-43ff-9bbb-0b298d9c99cc" (UID: "af589ec1-de60-43ff-9bbb-0b298d9c99cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:33:27.026619 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:27.026573 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af589ec1-de60-43ff-9bbb-0b298d9c99cc-kube-api-access-lx6v5\" (UniqueName: \"kubernetes.io/projected/af589ec1-de60-43ff-9bbb-0b298d9c99cc-kube-api-access-lx6v5\") pod \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\" (UID: \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\") " Oct 31 13:33:27.026765 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:27.026618 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af589ec1-de60-43ff-9bbb-0b298d9c99cc-istio-token\" (UniqueName: \"kubernetes.io/projected/af589ec1-de60-43ff-9bbb-0b298d9c99cc-istio-token\") pod \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\" (UID: \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\") " Oct 31 13:33:27.026765 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:27.026711 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af589ec1-de60-43ff-9bbb-0b298d9c99cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af589ec1-de60-43ff-9bbb-0b298d9c99cc-credential-socket\") pod \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\" (UID: \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\") " Oct 31 13:33:27.026765 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:27.026756 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af589ec1-de60-43ff-9bbb-0b298d9c99cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af589ec1-de60-43ff-9bbb-0b298d9c99cc-workload-certs\") pod \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\" (UID: \"af589ec1-de60-43ff-9bbb-0b298d9c99cc\") " Oct 31 13:33:27.027240 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:27.027211 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af589ec1-de60-43ff-9bbb-0b298d9c99cc-istiod-ca-cert" pod "af589ec1-de60-43ff-9bbb-0b298d9c99cc" (UID: "af589ec1-de60-43ff-9bbb-0b298d9c99cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:33:27.027393 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:27.027362 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af589ec1-de60-43ff-9bbb-0b298d9c99cc-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:33:27.027393 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:27.027386 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af589ec1-de60-43ff-9bbb-0b298d9c99cc-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:33:27.027568 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:27.027475 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af589ec1-de60-43ff-9bbb-0b298d9c99cc-istio-data" pod "af589ec1-de60-43ff-9bbb-0b298d9c99cc" (UID: "af589ec1-de60-43ff-9bbb-0b298d9c99cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:33:27.027801 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:27.027778 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af589ec1-de60-43ff-9bbb-0b298d9c99cc-credential-socket" pod "af589ec1-de60-43ff-9bbb-0b298d9c99cc" (UID: "af589ec1-de60-43ff-9bbb-0b298d9c99cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:33:27.029005 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:27.028962 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af589ec1-de60-43ff-9bbb-0b298d9c99cc-workload-certs" pod "af589ec1-de60-43ff-9bbb-0b298d9c99cc" (UID: "af589ec1-de60-43ff-9bbb-0b298d9c99cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:33:27.030935 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:27.030909 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af589ec1-de60-43ff-9bbb-0b298d9c99cc-istio-envoy" pod "af589ec1-de60-43ff-9bbb-0b298d9c99cc" (UID: "af589ec1-de60-43ff-9bbb-0b298d9c99cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:33:27.034782 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:27.034748 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af589ec1-de60-43ff-9bbb-0b298d9c99cc-istio-podinfo" pod "af589ec1-de60-43ff-9bbb-0b298d9c99cc" (UID: "af589ec1-de60-43ff-9bbb-0b298d9c99cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:33:27.034898 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:27.034817 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af589ec1-de60-43ff-9bbb-0b298d9c99cc-istio-token" pod "af589ec1-de60-43ff-9bbb-0b298d9c99cc" (UID: "af589ec1-de60-43ff-9bbb-0b298d9c99cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:33:27.034898 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:27.034847 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af589ec1-de60-43ff-9bbb-0b298d9c99cc-kube-api-access-lx6v5" pod "af589ec1-de60-43ff-9bbb-0b298d9c99cc" (UID: "af589ec1-de60-43ff-9bbb-0b298d9c99cc"). InnerVolumeSpecName "kube-api-access-lx6v5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:33:27.128544 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:27.128506 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af589ec1-de60-43ff-9bbb-0b298d9c99cc-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:33:27.128544 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:27.128539 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af589ec1-de60-43ff-9bbb-0b298d9c99cc-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:33:27.128544 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:27.128548 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af589ec1-de60-43ff-9bbb-0b298d9c99cc-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:33:27.128816 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:27.128556 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx6v5\" (UniqueName: \"kubernetes.io/projected/af589ec1-de60-43ff-9bbb-0b298d9c99cc-kube-api-access-lx6v5\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:33:27.128816 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:27.128567 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af589ec1-de60-43ff-9bbb-0b298d9c99cc-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:33:27.128816 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:27.128575 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af589ec1-de60-43ff-9bbb-0b298d9c99cc-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:33:27.128816 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:27.128583 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af589ec1-de60-43ff-9bbb-0b298d9c99cc-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:33:27.362867 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:27.362829 5693 generic.go:356] "Generic (PLEG): container finished" podID="af589ec1-de60-43ff-9bbb-0b298d9c99cc" containerID="d93345d85a287d60125356684d0bc61b194ee595c01ec4fa30d4af9f720ab9c5" exitCode=0 Oct 31 13:33:27.363081 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:27.362885 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" Oct 31 13:33:27.363081 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:27.362915 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" event={"ID":"af589ec1-de60-43ff-9bbb-0b298d9c99cc","Type":"ContainerDied","Data":"d93345d85a287d60125356684d0bc61b194ee595c01ec4fa30d4af9f720ab9c5"} Oct 31 13:33:27.363081 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:27.362970 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" event={"ID":"af589ec1-de60-43ff-9bbb-0b298d9c99cc","Type":"ContainerDied","Data":"3dccf12ce236749964d6d63efcd9a6365161d2fc5d0a56fb215b711e1ef6259c"} Oct 31 13:33:27.363081 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:27.363007 5693 scope.go:122] "RemoveContainer" containerID="d93345d85a287d60125356684d0bc61b194ee595c01ec4fa30d4af9f720ab9c5" Oct 31 13:33:27.375878 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:27.375853 5693 scope.go:122] "RemoveContainer" containerID="d93345d85a287d60125356684d0bc61b194ee595c01ec4fa30d4af9f720ab9c5" Oct 31 13:33:27.376167 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:33:27.376148 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d93345d85a287d60125356684d0bc61b194ee595c01ec4fa30d4af9f720ab9c5\": container with ID starting with d93345d85a287d60125356684d0bc61b194ee595c01ec4fa30d4af9f720ab9c5 not found: ID does not exist" containerID="d93345d85a287d60125356684d0bc61b194ee595c01ec4fa30d4af9f720ab9c5" Oct 31 13:33:27.376235 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:27.376176 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d93345d85a287d60125356684d0bc61b194ee595c01ec4fa30d4af9f720ab9c5"} err="failed to get container status \"d93345d85a287d60125356684d0bc61b194ee595c01ec4fa30d4af9f720ab9c5\": rpc error: code = NotFound desc = could not find container \"d93345d85a287d60125356684d0bc61b194ee595c01ec4fa30d4af9f720ab9c5\": container with ID starting with d93345d85a287d60125356684d0bc61b194ee595c01ec4fa30d4af9f720ab9c5 not found: ID does not exist" Oct 31 13:33:27.389701 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:27.389664 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2"] Oct 31 13:33:27.395399 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:27.395374 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2"] Oct 31 13:33:27.411994 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:27.411969 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af589ec1-de60-43ff-9bbb-0b298d9c99cc" path="/var/lib/kubelet/pods/af589ec1-de60-43ff-9bbb-0b298d9c99cc/volumes" Oct 31 13:33:27.892085 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:27.892042 5693 patch_prober.go:29] interesting pod/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:33:27.892294 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:33:27.892122 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-29e3a0-openshift-default-76b99778bf-ww5b2" podUID="af589ec1-de60-43ff-9bbb-0b298d9c99cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:33:48.688519 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:33:48.688484 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:34:18.690017 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:34:18.689927 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:34:48.688793 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:34:48.688757 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:34:51.026565 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:34:51.026528 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ca4b64b-7kdk6"] Oct 31 13:34:51.031011 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:34:51.030984 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ca4b64b-7kdk6"] Oct 31 13:34:51.412103 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:34:51.412068 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df80a09d-3179-4e70-ae9a-748d67f5c460" path="/var/lib/kubelet/pods/df80a09d-3179-4e70-ae9a-748d67f5c460/volumes" Oct 31 13:35:05.114797 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:05.114760 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a13836/test-model-a13836-kserve-5b7f69c4f5-4gl49"] Oct 31 13:35:12.612199 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:12.612096 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a39960-59f9f54fd9-qlvqh"] Oct 31 13:35:12.619947 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:12.619915 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a39960-84b6db594c-lf44b"] Oct 31 13:35:17.613376 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:17.613330 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a39960-59f9f54fd9-qlvqh" podUID="2852337d-cf60-41cf-a3c1-5d6ff380c97f" containerName="payload-processing" containerID="cri-o://f303f4be3db8255547408ac0caa77b0d7264643eeb22b30dc93ad4de888f3334" gracePeriod=25 Oct 31 13:35:17.620815 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:17.620779 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a39960-84b6db594c-lf44b" podUID="0e759d3e-611f-43e3-b520-2a5d34517d91" containerName="payload-pre-processing" containerID="cri-o://c205c54e0e7cd8a502a036b71bcf839981d0dc6cadeee50f5d7bd35574dd6857" gracePeriod=25 Oct 31 13:35:17.918124 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:17.918095 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a39960-59f9f54fd9-qlvqh" Oct 31 13:35:18.077861 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:18.077828 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2852337d-cf60-41cf-a3c1-5d6ff380c97f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2852337d-cf60-41cf-a3c1-5d6ff380c97f-plugins-config-volume\") pod \"2852337d-cf60-41cf-a3c1-5d6ff380c97f\" (UID: \"2852337d-cf60-41cf-a3c1-5d6ff380c97f\") " Oct 31 13:35:18.078085 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:18.078018 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2852337d-cf60-41cf-a3c1-5d6ff380c97f-kube-api-access-khk6w\" (UniqueName: \"kubernetes.io/projected/2852337d-cf60-41cf-a3c1-5d6ff380c97f-kube-api-access-khk6w\") pod \"2852337d-cf60-41cf-a3c1-5d6ff380c97f\" (UID: \"2852337d-cf60-41cf-a3c1-5d6ff380c97f\") " Oct 31 13:35:18.078331 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:18.078302 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2852337d-cf60-41cf-a3c1-5d6ff380c97f-plugins-config-volume" pod "2852337d-cf60-41cf-a3c1-5d6ff380c97f" (UID: "2852337d-cf60-41cf-a3c1-5d6ff380c97f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:35:18.078960 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:18.078855 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2852337d-cf60-41cf-a3c1-5d6ff380c97f-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:18.080671 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:18.080601 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2852337d-cf60-41cf-a3c1-5d6ff380c97f-kube-api-access-khk6w" pod "2852337d-cf60-41cf-a3c1-5d6ff380c97f" (UID: "2852337d-cf60-41cf-a3c1-5d6ff380c97f"). InnerVolumeSpecName "kube-api-access-khk6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:35:18.180141 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:18.180099 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khk6w\" (UniqueName: \"kubernetes.io/projected/2852337d-cf60-41cf-a3c1-5d6ff380c97f-kube-api-access-khk6w\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:18.447620 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:18.447521 5693 generic.go:356] "Generic (PLEG): container finished" podID="2852337d-cf60-41cf-a3c1-5d6ff380c97f" containerID="f303f4be3db8255547408ac0caa77b0d7264643eeb22b30dc93ad4de888f3334" exitCode=0 Oct 31 13:35:18.447620 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:18.447599 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a39960-59f9f54fd9-qlvqh" Oct 31 13:35:18.447934 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:18.447611 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a39960-59f9f54fd9-qlvqh" event={"ID":"2852337d-cf60-41cf-a3c1-5d6ff380c97f","Type":"ContainerDied","Data":"f303f4be3db8255547408ac0caa77b0d7264643eeb22b30dc93ad4de888f3334"} Oct 31 13:35:18.447934 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:18.447738 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a39960-59f9f54fd9-qlvqh" event={"ID":"2852337d-cf60-41cf-a3c1-5d6ff380c97f","Type":"ContainerDied","Data":"6dc8d807f1d709450b0361602c46288ee3db177e4a254c0c41bb278757404bd2"} Oct 31 13:35:18.447934 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:18.447778 5693 scope.go:122] "RemoveContainer" containerID="f303f4be3db8255547408ac0caa77b0d7264643eeb22b30dc93ad4de888f3334" Oct 31 13:35:18.459846 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:18.459822 5693 scope.go:122] "RemoveContainer" containerID="f303f4be3db8255547408ac0caa77b0d7264643eeb22b30dc93ad4de888f3334" Oct 31 13:35:18.460153 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:35:18.460134 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f303f4be3db8255547408ac0caa77b0d7264643eeb22b30dc93ad4de888f3334\": container with ID starting with f303f4be3db8255547408ac0caa77b0d7264643eeb22b30dc93ad4de888f3334 not found: ID does not exist" containerID="f303f4be3db8255547408ac0caa77b0d7264643eeb22b30dc93ad4de888f3334" Oct 31 13:35:18.460223 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:18.460167 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f303f4be3db8255547408ac0caa77b0d7264643eeb22b30dc93ad4de888f3334"} err="failed to get container status \"f303f4be3db8255547408ac0caa77b0d7264643eeb22b30dc93ad4de888f3334\": rpc error: code = NotFound desc = could not find container \"f303f4be3db8255547408ac0caa77b0d7264643eeb22b30dc93ad4de888f3334\": container with ID starting with f303f4be3db8255547408ac0caa77b0d7264643eeb22b30dc93ad4de888f3334 not found: ID does not exist" Oct 31 13:35:18.476587 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:18.476559 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a39960-59f9f54fd9-qlvqh"] Oct 31 13:35:18.482074 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:18.482052 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a39960-59f9f54fd9-qlvqh"] Oct 31 13:35:18.689302 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:35:18.689263 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:35:19.414034 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:19.414004 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2852337d-cf60-41cf-a3c1-5d6ff380c97f" path="/var/lib/kubelet/pods/2852337d-cf60-41cf-a3c1-5d6ff380c97f/volumes" Oct 31 13:35:19.792606 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:19.792523 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a39960-84b6db594c-lf44b" podUID="0e759d3e-611f-43e3-b520-2a5d34517d91" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:35:20.356229 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:20.356187 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-a13836/test-model-a13836-kserve-5b7f69c4f5-4gl49" podUID="0d58e66c-aad7-488b-8315-3f357c4220c0" containerName="main" containerID="cri-o://1491d613fcee0031ddda199c00a7626fb2060f71ea742bd8727ec324f6d10f62" gracePeriod=45 Oct 31 13:35:20.670212 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:20.670187 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a13836/test-model-a13836-kserve-5b7f69c4f5-4gl49" Oct 31 13:35:20.826269 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:20.826238 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d58e66c-aad7-488b-8315-3f357c4220c0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d58e66c-aad7-488b-8315-3f357c4220c0-tmp-dir\") pod \"0d58e66c-aad7-488b-8315-3f357c4220c0\" (UID: \"0d58e66c-aad7-488b-8315-3f357c4220c0\") " Oct 31 13:35:20.826797 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:20.826341 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d58e66c-aad7-488b-8315-3f357c4220c0-home\" (UniqueName: \"kubernetes.io/empty-dir/0d58e66c-aad7-488b-8315-3f357c4220c0-home\") pod \"0d58e66c-aad7-488b-8315-3f357c4220c0\" (UID: \"0d58e66c-aad7-488b-8315-3f357c4220c0\") " Oct 31 13:35:20.826797 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:20.826404 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d58e66c-aad7-488b-8315-3f357c4220c0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d58e66c-aad7-488b-8315-3f357c4220c0-model-cache\") pod \"0d58e66c-aad7-488b-8315-3f357c4220c0\" (UID: \"0d58e66c-aad7-488b-8315-3f357c4220c0\") " Oct 31 13:35:20.826797 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:20.826515 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d58e66c-aad7-488b-8315-3f357c4220c0-kube-api-access-2vnzx\" (UniqueName: \"kubernetes.io/projected/0d58e66c-aad7-488b-8315-3f357c4220c0-kube-api-access-2vnzx\") pod \"0d58e66c-aad7-488b-8315-3f357c4220c0\" (UID: \"0d58e66c-aad7-488b-8315-3f357c4220c0\") " Oct 31 13:35:20.826797 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:20.826663 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d58e66c-aad7-488b-8315-3f357c4220c0-tls-certs\" (UniqueName: \"kubernetes.io/secret/0d58e66c-aad7-488b-8315-3f357c4220c0-tls-certs\") pod \"0d58e66c-aad7-488b-8315-3f357c4220c0\" (UID: \"0d58e66c-aad7-488b-8315-3f357c4220c0\") " Oct 31 13:35:20.826797 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:20.826630 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d58e66c-aad7-488b-8315-3f357c4220c0-tmp-dir" pod "0d58e66c-aad7-488b-8315-3f357c4220c0" (UID: "0d58e66c-aad7-488b-8315-3f357c4220c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:35:20.826797 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:20.826746 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d58e66c-aad7-488b-8315-3f357c4220c0-model-cache" pod "0d58e66c-aad7-488b-8315-3f357c4220c0" (UID: "0d58e66c-aad7-488b-8315-3f357c4220c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:35:20.826797 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:20.826766 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d58e66c-aad7-488b-8315-3f357c4220c0-home" pod "0d58e66c-aad7-488b-8315-3f357c4220c0" (UID: "0d58e66c-aad7-488b-8315-3f357c4220c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:35:20.826797 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:20.826794 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d58e66c-aad7-488b-8315-3f357c4220c0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d58e66c-aad7-488b-8315-3f357c4220c0-dshm\") pod \"0d58e66c-aad7-488b-8315-3f357c4220c0\" (UID: \"0d58e66c-aad7-488b-8315-3f357c4220c0\") " Oct 31 13:35:20.827727 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:20.827700 5693 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d58e66c-aad7-488b-8315-3f357c4220c0-tmp-dir\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:20.827900 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:20.827731 5693 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d58e66c-aad7-488b-8315-3f357c4220c0-home\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:20.827900 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:20.827746 5693 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d58e66c-aad7-488b-8315-3f357c4220c0-model-cache\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:20.829281 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:20.829252 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d58e66c-aad7-488b-8315-3f357c4220c0-tls-certs" pod "0d58e66c-aad7-488b-8315-3f357c4220c0" (UID: "0d58e66c-aad7-488b-8315-3f357c4220c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:35:20.829523 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:20.829503 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d58e66c-aad7-488b-8315-3f357c4220c0-kube-api-access-2vnzx" pod "0d58e66c-aad7-488b-8315-3f357c4220c0" (UID: "0d58e66c-aad7-488b-8315-3f357c4220c0"). InnerVolumeSpecName "kube-api-access-2vnzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:35:20.829604 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:20.829552 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d58e66c-aad7-488b-8315-3f357c4220c0-dshm" pod "0d58e66c-aad7-488b-8315-3f357c4220c0" (UID: "0d58e66c-aad7-488b-8315-3f357c4220c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:35:20.929332 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:20.929297 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vnzx\" (UniqueName: \"kubernetes.io/projected/0d58e66c-aad7-488b-8315-3f357c4220c0-kube-api-access-2vnzx\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:20.929332 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:20.929328 5693 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d58e66c-aad7-488b-8315-3f357c4220c0-tls-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:20.929332 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:20.929339 5693 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d58e66c-aad7-488b-8315-3f357c4220c0-dshm\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:21.474200 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:21.474165 5693 generic.go:356] "Generic (PLEG): container finished" podID="0d58e66c-aad7-488b-8315-3f357c4220c0" containerID="1491d613fcee0031ddda199c00a7626fb2060f71ea742bd8727ec324f6d10f62" exitCode=0 Oct 31 13:35:21.474387 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:21.474214 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a13836/test-model-a13836-kserve-5b7f69c4f5-4gl49" Oct 31 13:35:21.474387 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:21.474217 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a13836/test-model-a13836-kserve-5b7f69c4f5-4gl49" event={"ID":"0d58e66c-aad7-488b-8315-3f357c4220c0","Type":"ContainerDied","Data":"1491d613fcee0031ddda199c00a7626fb2060f71ea742bd8727ec324f6d10f62"} Oct 31 13:35:21.474387 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:21.474255 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a13836/test-model-a13836-kserve-5b7f69c4f5-4gl49" event={"ID":"0d58e66c-aad7-488b-8315-3f357c4220c0","Type":"ContainerDied","Data":"f46ab19702bf8817c27e9490f995707d861744ed125a0426cf2c2fee2e9319e5"} Oct 31 13:35:21.474387 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:21.474276 5693 scope.go:122] "RemoveContainer" containerID="1491d613fcee0031ddda199c00a7626fb2060f71ea742bd8727ec324f6d10f62" Oct 31 13:35:21.489780 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:21.489748 5693 scope.go:122] "RemoveContainer" containerID="1491d613fcee0031ddda199c00a7626fb2060f71ea742bd8727ec324f6d10f62" Oct 31 13:35:21.490187 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:35:21.490159 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1491d613fcee0031ddda199c00a7626fb2060f71ea742bd8727ec324f6d10f62\": container with ID starting with 1491d613fcee0031ddda199c00a7626fb2060f71ea742bd8727ec324f6d10f62 not found: ID does not exist" containerID="1491d613fcee0031ddda199c00a7626fb2060f71ea742bd8727ec324f6d10f62" Oct 31 13:35:21.490258 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:21.490199 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1491d613fcee0031ddda199c00a7626fb2060f71ea742bd8727ec324f6d10f62"} err="failed to get container status \"1491d613fcee0031ddda199c00a7626fb2060f71ea742bd8727ec324f6d10f62\": rpc error: code = NotFound desc = could not find container \"1491d613fcee0031ddda199c00a7626fb2060f71ea742bd8727ec324f6d10f62\": container with ID starting with 1491d613fcee0031ddda199c00a7626fb2060f71ea742bd8727ec324f6d10f62 not found: ID does not exist" Oct 31 13:35:21.503786 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:21.503749 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a13836/test-model-a13836-kserve-5b7f69c4f5-4gl49"] Oct 31 13:35:21.508041 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:21.508005 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-a13836/test-model-a13836-kserve-5b7f69c4f5-4gl49"] Oct 31 13:35:23.413478 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:23.413440 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d58e66c-aad7-488b-8315-3f357c4220c0" path="/var/lib/kubelet/pods/0d58e66c-aad7-488b-8315-3f357c4220c0/volumes" Oct 31 13:35:28.988405 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:28.988370 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m"] Oct 31 13:35:28.988973 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:28.988591 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" podUID="9735d8ba-9f3e-434c-bb9f-5876f11d9336" containerName="istio-proxy" containerID="cri-o://4388103aa62448775770f44a8297eca0e2a7b6f704fbcf1ceed08ac1dfdd587c" gracePeriod=30 Oct 31 13:35:28.993593 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:28.993565 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq"] Oct 31 13:35:28.993790 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:28.993771 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" podUID="9dd14658-e45a-468a-8691-97601b44e956" containerName="istio-proxy" containerID="cri-o://6d376b44fe1f910d604f03fe7c25bc2db62dd356ea329885140ce915ec7adf5e" gracePeriod=30 Oct 31 13:35:29.792065 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:29.792030 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a39960-84b6db594c-lf44b" podUID="0e759d3e-611f-43e3-b520-2a5d34517d91" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:35:34.158122 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:35:34.158089 5693 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9735d8ba_9f3e_434c_bb9f_5876f11d9336.slice/crio-conmon-4388103aa62448775770f44a8297eca0e2a7b6f704fbcf1ceed08ac1dfdd587c.scope\": RecentStats: unable to find data in memory cache]" Oct 31 13:35:34.309894 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.309866 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" Oct 31 13:35:34.351999 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.351958 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" Oct 31 13:35:34.499826 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.499789 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9735d8ba-9f3e-434c-bb9f-5876f11d9336-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9735d8ba-9f3e-434c-bb9f-5876f11d9336-istio-data\") pod \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\" (UID: \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\") " Oct 31 13:35:34.500034 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.499863 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9735d8ba-9f3e-434c-bb9f-5876f11d9336-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9735d8ba-9f3e-434c-bb9f-5876f11d9336-istio-podinfo\") pod \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\" (UID: \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\") " Oct 31 13:35:34.500034 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.499911 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9735d8ba-9f3e-434c-bb9f-5876f11d9336-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9735d8ba-9f3e-434c-bb9f-5876f11d9336-credential-socket\") pod \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\" (UID: \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\") " Oct 31 13:35:34.500034 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.499945 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dd14658-e45a-468a-8691-97601b44e956-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dd14658-e45a-468a-8691-97601b44e956-credential-socket\") pod \"9dd14658-e45a-468a-8691-97601b44e956\" (UID: \"9dd14658-e45a-468a-8691-97601b44e956\") " Oct 31 13:35:34.500034 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.499992 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9735d8ba-9f3e-434c-bb9f-5876f11d9336-kube-api-access-8c8lx\" (UniqueName: \"kubernetes.io/projected/9735d8ba-9f3e-434c-bb9f-5876f11d9336-kube-api-access-8c8lx\") pod \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\" (UID: \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\") " Oct 31 13:35:34.500273 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.500036 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9735d8ba-9f3e-434c-bb9f-5876f11d9336-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9735d8ba-9f3e-434c-bb9f-5876f11d9336-istio-envoy\") pod \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\" (UID: \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\") " Oct 31 13:35:34.500273 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.500108 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9735d8ba-9f3e-434c-bb9f-5876f11d9336-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9735d8ba-9f3e-434c-bb9f-5876f11d9336-istiod-ca-cert\") pod \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\" (UID: \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\") " Oct 31 13:35:34.500273 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.500153 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9735d8ba-9f3e-434c-bb9f-5876f11d9336-istio-data" pod "9735d8ba-9f3e-434c-bb9f-5876f11d9336" (UID: "9735d8ba-9f3e-434c-bb9f-5876f11d9336"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:35:34.500273 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.500173 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9735d8ba-9f3e-434c-bb9f-5876f11d9336-istio-token\" (UniqueName: \"kubernetes.io/projected/9735d8ba-9f3e-434c-bb9f-5876f11d9336-istio-token\") pod \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\" (UID: \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\") " Oct 31 13:35:34.500273 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.500203 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dd14658-e45a-468a-8691-97601b44e956-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dd14658-e45a-468a-8691-97601b44e956-workload-socket\") pod \"9dd14658-e45a-468a-8691-97601b44e956\" (UID: \"9dd14658-e45a-468a-8691-97601b44e956\") " Oct 31 13:35:34.500273 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.500261 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dd14658-e45a-468a-8691-97601b44e956-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dd14658-e45a-468a-8691-97601b44e956-istio-envoy\") pod \"9dd14658-e45a-468a-8691-97601b44e956\" (UID: \"9dd14658-e45a-468a-8691-97601b44e956\") " Oct 31 13:35:34.500613 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.500268 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dd14658-e45a-468a-8691-97601b44e956-credential-socket" pod "9dd14658-e45a-468a-8691-97601b44e956" (UID: "9dd14658-e45a-468a-8691-97601b44e956"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:35:34.500613 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.500299 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dd14658-e45a-468a-8691-97601b44e956-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dd14658-e45a-468a-8691-97601b44e956-istio-data\") pod \"9dd14658-e45a-468a-8691-97601b44e956\" (UID: \"9dd14658-e45a-468a-8691-97601b44e956\") " Oct 31 13:35:34.500613 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.500347 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dd14658-e45a-468a-8691-97601b44e956-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dd14658-e45a-468a-8691-97601b44e956-workload-certs\") pod \"9dd14658-e45a-468a-8691-97601b44e956\" (UID: \"9dd14658-e45a-468a-8691-97601b44e956\") " Oct 31 13:35:34.500613 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.500385 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9735d8ba-9f3e-434c-bb9f-5876f11d9336-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9735d8ba-9f3e-434c-bb9f-5876f11d9336-workload-socket\") pod \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\" (UID: \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\") " Oct 31 13:35:34.500613 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.500496 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dd14658-e45a-468a-8691-97601b44e956-workload-socket" pod "9dd14658-e45a-468a-8691-97601b44e956" (UID: "9dd14658-e45a-468a-8691-97601b44e956"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:35:34.500613 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.500532 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9735d8ba-9f3e-434c-bb9f-5876f11d9336-istiod-ca-cert" pod "9735d8ba-9f3e-434c-bb9f-5876f11d9336" (UID: "9735d8ba-9f3e-434c-bb9f-5876f11d9336"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:35:34.501169 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.500580 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9735d8ba-9f3e-434c-bb9f-5876f11d9336-credential-socket" pod "9735d8ba-9f3e-434c-bb9f-5876f11d9336" (UID: "9735d8ba-9f3e-434c-bb9f-5876f11d9336"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:35:34.501169 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.500548 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dd14658-e45a-468a-8691-97601b44e956-istio-token\" (UniqueName: \"kubernetes.io/projected/9dd14658-e45a-468a-8691-97601b44e956-istio-token\") pod \"9dd14658-e45a-468a-8691-97601b44e956\" (UID: \"9dd14658-e45a-468a-8691-97601b44e956\") " Oct 31 13:35:34.501169 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.500705 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9735d8ba-9f3e-434c-bb9f-5876f11d9336-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9735d8ba-9f3e-434c-bb9f-5876f11d9336-workload-certs\") pod \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\" (UID: \"9735d8ba-9f3e-434c-bb9f-5876f11d9336\") " Oct 31 13:35:34.501169 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.500730 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dd14658-e45a-468a-8691-97601b44e956-istio-data" pod "9dd14658-e45a-468a-8691-97601b44e956" (UID: "9dd14658-e45a-468a-8691-97601b44e956"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:35:34.501169 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.500737 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9dd14658-e45a-468a-8691-97601b44e956-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dd14658-e45a-468a-8691-97601b44e956-istio-podinfo\") pod \"9dd14658-e45a-468a-8691-97601b44e956\" (UID: \"9dd14658-e45a-468a-8691-97601b44e956\") " Oct 31 13:35:34.501169 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.500816 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dd14658-e45a-468a-8691-97601b44e956-kube-api-access-qvc5l\" (UniqueName: \"kubernetes.io/projected/9dd14658-e45a-468a-8691-97601b44e956-kube-api-access-qvc5l\") pod \"9dd14658-e45a-468a-8691-97601b44e956\" (UID: \"9dd14658-e45a-468a-8691-97601b44e956\") " Oct 31 13:35:34.501169 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.500832 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dd14658-e45a-468a-8691-97601b44e956-workload-certs" pod "9dd14658-e45a-468a-8691-97601b44e956" (UID: "9dd14658-e45a-468a-8691-97601b44e956"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:35:34.501169 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.500930 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dd14658-e45a-468a-8691-97601b44e956-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dd14658-e45a-468a-8691-97601b44e956-istiod-ca-cert\") pod \"9dd14658-e45a-468a-8691-97601b44e956\" (UID: \"9dd14658-e45a-468a-8691-97601b44e956\") " Oct 31 13:35:34.501169 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.501016 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9735d8ba-9f3e-434c-bb9f-5876f11d9336-workload-socket" pod "9735d8ba-9f3e-434c-bb9f-5876f11d9336" (UID: "9735d8ba-9f3e-434c-bb9f-5876f11d9336"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:35:34.502091 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.502064 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9735d8ba-9f3e-434c-bb9f-5876f11d9336-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:34.502203 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.502098 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9735d8ba-9f3e-434c-bb9f-5876f11d9336-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:34.502203 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.502113 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dd14658-e45a-468a-8691-97601b44e956-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:34.502203 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.502129 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9735d8ba-9f3e-434c-bb9f-5876f11d9336-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:34.502203 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.502144 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dd14658-e45a-468a-8691-97601b44e956-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:34.502203 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.502164 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dd14658-e45a-468a-8691-97601b44e956-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:34.502203 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.502178 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dd14658-e45a-468a-8691-97601b44e956-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:34.502203 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.502191 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9735d8ba-9f3e-434c-bb9f-5876f11d9336-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:34.503298 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.503056 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9735d8ba-9f3e-434c-bb9f-5876f11d9336-istio-podinfo" pod "9735d8ba-9f3e-434c-bb9f-5876f11d9336" (UID: "9735d8ba-9f3e-434c-bb9f-5876f11d9336"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:35:34.503298 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.503127 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9735d8ba-9f3e-434c-bb9f-5876f11d9336-istio-envoy" pod "9735d8ba-9f3e-434c-bb9f-5876f11d9336" (UID: "9735d8ba-9f3e-434c-bb9f-5876f11d9336"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:35:34.503298 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.503199 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9735d8ba-9f3e-434c-bb9f-5876f11d9336-kube-api-access-8c8lx" pod "9735d8ba-9f3e-434c-bb9f-5876f11d9336" (UID: "9735d8ba-9f3e-434c-bb9f-5876f11d9336"). InnerVolumeSpecName "kube-api-access-8c8lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:35:34.503298 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.503234 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9735d8ba-9f3e-434c-bb9f-5876f11d9336-workload-certs" pod "9735d8ba-9f3e-434c-bb9f-5876f11d9336" (UID: "9735d8ba-9f3e-434c-bb9f-5876f11d9336"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:35:34.503298 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.503249 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dd14658-e45a-468a-8691-97601b44e956-istio-token" pod "9dd14658-e45a-468a-8691-97601b44e956" (UID: "9dd14658-e45a-468a-8691-97601b44e956"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:35:34.503728 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.503539 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dd14658-e45a-468a-8691-97601b44e956-istio-envoy" pod "9dd14658-e45a-468a-8691-97601b44e956" (UID: "9dd14658-e45a-468a-8691-97601b44e956"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:35:34.503728 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.503568 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dd14658-e45a-468a-8691-97601b44e956-istiod-ca-cert" pod "9dd14658-e45a-468a-8691-97601b44e956" (UID: "9dd14658-e45a-468a-8691-97601b44e956"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:35:34.503965 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.503941 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9dd14658-e45a-468a-8691-97601b44e956-istio-podinfo" pod "9dd14658-e45a-468a-8691-97601b44e956" (UID: "9dd14658-e45a-468a-8691-97601b44e956"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:35:34.504516 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.504495 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9735d8ba-9f3e-434c-bb9f-5876f11d9336-istio-token" pod "9735d8ba-9f3e-434c-bb9f-5876f11d9336" (UID: "9735d8ba-9f3e-434c-bb9f-5876f11d9336"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:35:34.504768 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.504741 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dd14658-e45a-468a-8691-97601b44e956-kube-api-access-qvc5l" pod "9dd14658-e45a-468a-8691-97601b44e956" (UID: "9dd14658-e45a-468a-8691-97601b44e956"). InnerVolumeSpecName "kube-api-access-qvc5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:35:34.604143 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.604049 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9735d8ba-9f3e-434c-bb9f-5876f11d9336-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:34.604143 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.604082 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c8lx\" (UniqueName: \"kubernetes.io/projected/9735d8ba-9f3e-434c-bb9f-5876f11d9336-kube-api-access-8c8lx\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:34.604143 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.604093 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9735d8ba-9f3e-434c-bb9f-5876f11d9336-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:34.604143 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.604102 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9735d8ba-9f3e-434c-bb9f-5876f11d9336-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:34.604143 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.604110 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dd14658-e45a-468a-8691-97601b44e956-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:34.604143 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.604119 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dd14658-e45a-468a-8691-97601b44e956-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:34.604143 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.604126 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9735d8ba-9f3e-434c-bb9f-5876f11d9336-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:34.604143 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.604134 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dd14658-e45a-468a-8691-97601b44e956-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:34.604143 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.604143 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvc5l\" (UniqueName: \"kubernetes.io/projected/9dd14658-e45a-468a-8691-97601b44e956-kube-api-access-qvc5l\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:34.604143 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.604150 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dd14658-e45a-468a-8691-97601b44e956-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:34.651932 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.651825 5693 generic.go:356] "Generic (PLEG): container finished" podID="9735d8ba-9f3e-434c-bb9f-5876f11d9336" containerID="4388103aa62448775770f44a8297eca0e2a7b6f704fbcf1ceed08ac1dfdd587c" exitCode=0 Oct 31 13:35:34.651932 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.651910 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" Oct 31 13:35:34.651932 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.651908 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" event={"ID":"9735d8ba-9f3e-434c-bb9f-5876f11d9336","Type":"ContainerDied","Data":"4388103aa62448775770f44a8297eca0e2a7b6f704fbcf1ceed08ac1dfdd587c"} Oct 31 13:35:34.652266 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.652020 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m" event={"ID":"9735d8ba-9f3e-434c-bb9f-5876f11d9336","Type":"ContainerDied","Data":"fb7b5043990ad54796cdd123b5f20d3e5de087b00eae660486ed59524fe85187"} Oct 31 13:35:34.652266 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.652061 5693 scope.go:122] "RemoveContainer" containerID="4388103aa62448775770f44a8297eca0e2a7b6f704fbcf1ceed08ac1dfdd587c" Oct 31 13:35:34.653934 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.653894 5693 generic.go:356] "Generic (PLEG): container finished" podID="9dd14658-e45a-468a-8691-97601b44e956" containerID="6d376b44fe1f910d604f03fe7c25bc2db62dd356ea329885140ce915ec7adf5e" exitCode=0 Oct 31 13:35:34.654064 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.653983 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" Oct 31 13:35:34.654145 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.653974 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" event={"ID":"9dd14658-e45a-468a-8691-97601b44e956","Type":"ContainerDied","Data":"6d376b44fe1f910d604f03fe7c25bc2db62dd356ea329885140ce915ec7adf5e"} Oct 31 13:35:34.654355 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.654138 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq" event={"ID":"9dd14658-e45a-468a-8691-97601b44e956","Type":"ContainerDied","Data":"104f227a6dd81721d2db6f145e9cf42b845e3af3e67e50e9c7416378f26a5b92"} Oct 31 13:35:34.667151 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.667132 5693 scope.go:122] "RemoveContainer" containerID="523ae8670acc553ccb1385c8cfcad2524d46d9d019df1f3545ff8e15a0d6416e" Oct 31 13:35:34.680746 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.680709 5693 scope.go:122] "RemoveContainer" containerID="4388103aa62448775770f44a8297eca0e2a7b6f704fbcf1ceed08ac1dfdd587c" Oct 31 13:35:34.681830 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:35:34.681290 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4388103aa62448775770f44a8297eca0e2a7b6f704fbcf1ceed08ac1dfdd587c\": container with ID starting with 4388103aa62448775770f44a8297eca0e2a7b6f704fbcf1ceed08ac1dfdd587c not found: ID does not exist" containerID="4388103aa62448775770f44a8297eca0e2a7b6f704fbcf1ceed08ac1dfdd587c" Oct 31 13:35:34.681830 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.681334 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4388103aa62448775770f44a8297eca0e2a7b6f704fbcf1ceed08ac1dfdd587c"} err="failed to get container status \"4388103aa62448775770f44a8297eca0e2a7b6f704fbcf1ceed08ac1dfdd587c\": rpc error: code = NotFound desc = could not find container \"4388103aa62448775770f44a8297eca0e2a7b6f704fbcf1ceed08ac1dfdd587c\": container with ID starting with 4388103aa62448775770f44a8297eca0e2a7b6f704fbcf1ceed08ac1dfdd587c not found: ID does not exist" Oct 31 13:35:34.681830 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.681358 5693 scope.go:122] "RemoveContainer" containerID="523ae8670acc553ccb1385c8cfcad2524d46d9d019df1f3545ff8e15a0d6416e" Oct 31 13:35:34.681830 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.681787 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m"] Oct 31 13:35:34.682186 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:35:34.681854 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"523ae8670acc553ccb1385c8cfcad2524d46d9d019df1f3545ff8e15a0d6416e\": container with ID starting with 523ae8670acc553ccb1385c8cfcad2524d46d9d019df1f3545ff8e15a0d6416e not found: ID does not exist" containerID="523ae8670acc553ccb1385c8cfcad2524d46d9d019df1f3545ff8e15a0d6416e" Oct 31 13:35:34.682186 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.681884 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"523ae8670acc553ccb1385c8cfcad2524d46d9d019df1f3545ff8e15a0d6416e"} err="failed to get container status \"523ae8670acc553ccb1385c8cfcad2524d46d9d019df1f3545ff8e15a0d6416e\": rpc error: code = NotFound desc = could not find container \"523ae8670acc553ccb1385c8cfcad2524d46d9d019df1f3545ff8e15a0d6416e\": container with ID starting with 523ae8670acc553ccb1385c8cfcad2524d46d9d019df1f3545ff8e15a0d6416e not found: ID does not exist" Oct 31 13:35:34.682186 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.681944 5693 scope.go:122] "RemoveContainer" containerID="6d376b44fe1f910d604f03fe7c25bc2db62dd356ea329885140ce915ec7adf5e" Oct 31 13:35:34.688631 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.688606 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-rwp5m"] Oct 31 13:35:34.693999 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.693974 5693 scope.go:122] "RemoveContainer" containerID="6d376b44fe1f910d604f03fe7c25bc2db62dd356ea329885140ce915ec7adf5e" Oct 31 13:35:34.694381 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:35:34.694358 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d376b44fe1f910d604f03fe7c25bc2db62dd356ea329885140ce915ec7adf5e\": container with ID starting with 6d376b44fe1f910d604f03fe7c25bc2db62dd356ea329885140ce915ec7adf5e not found: ID does not exist" containerID="6d376b44fe1f910d604f03fe7c25bc2db62dd356ea329885140ce915ec7adf5e" Oct 31 13:35:34.694482 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.694387 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d376b44fe1f910d604f03fe7c25bc2db62dd356ea329885140ce915ec7adf5e"} err="failed to get container status \"6d376b44fe1f910d604f03fe7c25bc2db62dd356ea329885140ce915ec7adf5e\": rpc error: code = NotFound desc = could not find container \"6d376b44fe1f910d604f03fe7c25bc2db62dd356ea329885140ce915ec7adf5e\": container with ID starting with 6d376b44fe1f910d604f03fe7c25bc2db62dd356ea329885140ce915ec7adf5e not found: ID does not exist" Oct 31 13:35:34.698196 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.698172 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq"] Oct 31 13:35:34.703127 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:34.703105 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a39960-openshift-default-7949cd75-gtcmq"] Oct 31 13:35:35.411955 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:35.411917 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9735d8ba-9f3e-434c-bb9f-5876f11d9336" path="/var/lib/kubelet/pods/9735d8ba-9f3e-434c-bb9f-5876f11d9336/volumes" Oct 31 13:35:35.412516 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:35.412497 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dd14658-e45a-468a-8691-97601b44e956" path="/var/lib/kubelet/pods/9dd14658-e45a-468a-8691-97601b44e956/volumes" Oct 31 13:35:37.028743 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.028714 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6"] Oct 31 13:35:37.029569 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.029554 5693 cpu_manager.go:414] "containerMap: removing container" podUID="9735d8ba-9f3e-434c-bb9f-5876f11d9336" containerName="istio-proxy" Oct 31 13:35:37.029622 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.029571 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9735d8ba-9f3e-434c-bb9f-5876f11d9336" containerName="istio-proxy" Oct 31 13:35:37.029622 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.029602 5693 cpu_manager.go:414] "containerMap: removing container" podUID="af589ec1-de60-43ff-9bbb-0b298d9c99cc" containerName="istio-proxy" Oct 31 13:35:37.029622 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.029607 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af589ec1-de60-43ff-9bbb-0b298d9c99cc" containerName="istio-proxy" Oct 31 13:35:37.029828 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.029628 5693 cpu_manager.go:414] "containerMap: removing container" podUID="0d58e66c-aad7-488b-8315-3f357c4220c0" containerName="main" Oct 31 13:35:37.029828 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.029635 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d58e66c-aad7-488b-8315-3f357c4220c0" containerName="main" Oct 31 13:35:37.029828 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.029663 5693 cpu_manager.go:414] "containerMap: removing container" podUID="9dd14658-e45a-468a-8691-97601b44e956" containerName="istio-proxy" Oct 31 13:35:37.029828 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.029668 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dd14658-e45a-468a-8691-97601b44e956" containerName="istio-proxy" Oct 31 13:35:37.029828 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.029677 5693 cpu_manager.go:414] "containerMap: removing container" podUID="2852337d-cf60-41cf-a3c1-5d6ff380c97f" containerName="payload-processing" Oct 31 13:35:37.029828 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.029682 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2852337d-cf60-41cf-a3c1-5d6ff380c97f" containerName="payload-processing" Oct 31 13:35:37.029828 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.029803 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="9735d8ba-9f3e-434c-bb9f-5876f11d9336" containerName="istio-proxy" Oct 31 13:35:37.029828 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.029816 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="af589ec1-de60-43ff-9bbb-0b298d9c99cc" containerName="istio-proxy" Oct 31 13:35:37.029828 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.029826 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d58e66c-aad7-488b-8315-3f357c4220c0" containerName="main" Oct 31 13:35:37.029828 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.029832 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dd14658-e45a-468a-8691-97601b44e956" containerName="istio-proxy" Oct 31 13:35:37.030243 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.029838 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="2852337d-cf60-41cf-a3c1-5d6ff380c97f" containerName="payload-processing" Oct 31 13:35:37.035410 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.035389 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" Oct 31 13:35:37.051865 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.051836 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6"] Oct 31 13:35:37.079866 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.071823 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcx22\" (UniqueName: \"kubernetes.io/projected/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-kube-api-access-jcx22\") pod \"maas-default-gateway-openshift-default-fc8c56547-6b8s6\" (UID: \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" Oct 31 13:35:37.079866 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.071953 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-6b8s6\" (UID: \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" Oct 31 13:35:37.079866 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.072018 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-6b8s6\" (UID: \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" Oct 31 13:35:37.079866 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.072082 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-6b8s6\" (UID: \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" Oct 31 13:35:37.079866 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.072126 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-6b8s6\" (UID: \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" Oct 31 13:35:37.079866 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.072165 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-6b8s6\" (UID: \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" Oct 31 13:35:37.079866 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.072205 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-6b8s6\" (UID: \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" Oct 31 13:35:37.079866 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.072291 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-6b8s6\" (UID: \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" Oct 31 13:35:37.079866 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.072332 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-6b8s6\" (UID: \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" Oct 31 13:35:37.112705 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.096222 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf"] Oct 31 13:35:37.112705 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.097504 5693 cpu_manager.go:414] "containerMap: removing container" podUID="9735d8ba-9f3e-434c-bb9f-5876f11d9336" containerName="istio-proxy" Oct 31 13:35:37.112705 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.097522 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9735d8ba-9f3e-434c-bb9f-5876f11d9336" containerName="istio-proxy" Oct 31 13:35:37.112705 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.097732 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="9735d8ba-9f3e-434c-bb9f-5876f11d9336" containerName="istio-proxy" Oct 31 13:35:37.112705 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.104317 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" Oct 31 13:35:37.121276 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.121248 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf"] Oct 31 13:35:37.173939 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.173897 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-6b8s6\" (UID: \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" Oct 31 13:35:37.174150 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.173946 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-h4czf\" (UID: \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" Oct 31 13:35:37.174150 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.173997 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-6b8s6\" (UID: \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" Oct 31 13:35:37.174150 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.174039 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-h4czf\" (UID: \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" Oct 31 13:35:37.174150 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.174136 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-6b8s6\" (UID: \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" Oct 31 13:35:37.174392 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.174173 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-h4czf\" (UID: \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" Oct 31 13:35:37.174392 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.174217 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-6b8s6\" (UID: \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" Oct 31 13:35:37.174392 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.174255 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-6b8s6\" (UID: \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" Oct 31 13:35:37.174392 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.174297 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-6b8s6\" (UID: \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" Oct 31 13:35:37.174392 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.174311 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-6b8s6\" (UID: \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" Oct 31 13:35:37.174392 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.174336 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-h4czf\" (UID: \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" Oct 31 13:35:37.174392 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.174365 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-h4czf\" (UID: \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" Oct 31 13:35:37.174392 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.174386 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-6b8s6\" (UID: \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" Oct 31 13:35:37.175054 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.174462 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-6b8s6\" (UID: \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" Oct 31 13:35:37.175054 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.174481 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-h4czf\" (UID: \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" Oct 31 13:35:37.175054 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.174531 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv5dj\" (UniqueName: \"kubernetes.io/projected/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-kube-api-access-cv5dj\") pod \"maas-default-gateway-openshift-default-fc8c56547-h4czf\" (UID: \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" Oct 31 13:35:37.175054 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.174607 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-6b8s6\" (UID: \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" Oct 31 13:35:37.175054 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.174676 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-6b8s6\" (UID: \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" Oct 31 13:35:37.175054 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.174744 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-h4czf\" (UID: \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" Oct 31 13:35:37.175054 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.174850 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcx22\" (UniqueName: \"kubernetes.io/projected/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-kube-api-access-jcx22\") pod \"maas-default-gateway-openshift-default-fc8c56547-6b8s6\" (UID: \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" Oct 31 13:35:37.175054 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.174897 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-h4czf\" (UID: \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" Oct 31 13:35:37.175685 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.175204 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-6b8s6\" (UID: \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" Oct 31 13:35:37.175685 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.175264 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-6b8s6\" (UID: \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" Oct 31 13:35:37.177153 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.177123 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-6b8s6\" (UID: \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" Oct 31 13:35:37.177502 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.177481 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-6b8s6\" (UID: \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" Oct 31 13:35:37.187100 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.187074 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-6b8s6\" (UID: \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" Oct 31 13:35:37.192136 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.192111 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcx22\" (UniqueName: \"kubernetes.io/projected/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-kube-api-access-jcx22\") pod \"maas-default-gateway-openshift-default-fc8c56547-6b8s6\" (UID: \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" Oct 31 13:35:37.276607 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.276571 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-h4czf\" (UID: \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" Oct 31 13:35:37.276843 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.276635 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-h4czf\" (UID: \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" Oct 31 13:35:37.276843 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.276708 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-h4czf\" (UID: \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" Oct 31 13:35:37.276843 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.276757 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-h4czf\" (UID: \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" Oct 31 13:35:37.276843 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.276776 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-h4czf\" (UID: \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" Oct 31 13:35:37.276843 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.276814 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-h4czf\" (UID: \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" Oct 31 13:35:37.276843 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.276839 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv5dj\" (UniqueName: \"kubernetes.io/projected/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-kube-api-access-cv5dj\") pod \"maas-default-gateway-openshift-default-fc8c56547-h4czf\" (UID: \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" Oct 31 13:35:37.277178 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.276931 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-h4czf\" (UID: \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" Oct 31 13:35:37.277178 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.277054 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-h4czf\" (UID: \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" Oct 31 13:35:37.277178 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.277060 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-h4czf\" (UID: \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" Oct 31 13:35:37.277178 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.277164 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-h4czf\" (UID: \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" Oct 31 13:35:37.277383 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.277211 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-h4czf\" (UID: \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" Oct 31 13:35:37.277433 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.277405 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-h4czf\" (UID: \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" Oct 31 13:35:37.277476 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.277449 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-h4czf\" (UID: \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" Oct 31 13:35:37.279922 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.279862 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-h4czf\" (UID: \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" Oct 31 13:35:37.280098 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.280079 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-h4czf\" (UID: \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" Oct 31 13:35:37.289701 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.289671 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-h4czf\" (UID: \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" Oct 31 13:35:37.289818 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.289804 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv5dj\" (UniqueName: \"kubernetes.io/projected/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-kube-api-access-cv5dj\") pod \"maas-default-gateway-openshift-default-fc8c56547-h4czf\" (UID: \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" Oct 31 13:35:37.381855 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.381819 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" Oct 31 13:35:37.437346 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.437313 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" Oct 31 13:35:37.685125 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.685000 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6"] Oct 31 13:35:37.692228 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:35:37.692195 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbedfdf67_dd1e_4902_9ca8_4f404d3a0841.slice/crio-71b188e89648cd743bb9ccac0f81c9668a13bf9746d10bf52a2e5e45c19141c0 WatchSource:0}: Error finding container 71b188e89648cd743bb9ccac0f81c9668a13bf9746d10bf52a2e5e45c19141c0: Status 404 returned error can't find the container with id 71b188e89648cd743bb9ccac0f81c9668a13bf9746d10bf52a2e5e45c19141c0 Oct 31 13:35:37.727756 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.727719 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:35:37.727884 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.727804 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:35:37.727884 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.727837 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:35:37.769926 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:35:37.769754 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddfd9d12_e8bb_42ee_a375_aba470b89eeb.slice/crio-09aee59fb47c0ab5e6b5d0d7bd78dd7a4085ccbcfb779c5b1045f0ed6764737b WatchSource:0}: Error finding container 09aee59fb47c0ab5e6b5d0d7bd78dd7a4085ccbcfb779c5b1045f0ed6764737b: Status 404 returned error can't find the container with id 09aee59fb47c0ab5e6b5d0d7bd78dd7a4085ccbcfb779c5b1045f0ed6764737b Oct 31 13:35:37.771010 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.770984 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf"] Oct 31 13:35:37.772854 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.772811 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:35:37.772955 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.772897 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:35:37.772955 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.772938 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:35:37.936045 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:37.935843 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a39960-84b6db594c-lf44b" Oct 31 13:35:38.030018 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:38.028019 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e759d3e-611f-43e3-b520-2a5d34517d91-kube-api-access-26dmg\" (UniqueName: \"kubernetes.io/projected/0e759d3e-611f-43e3-b520-2a5d34517d91-kube-api-access-26dmg\") pod \"0e759d3e-611f-43e3-b520-2a5d34517d91\" (UID: \"0e759d3e-611f-43e3-b520-2a5d34517d91\") " Oct 31 13:35:38.030018 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:38.028820 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e759d3e-611f-43e3-b520-2a5d34517d91-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e759d3e-611f-43e3-b520-2a5d34517d91-plugins-config-volume\") pod \"0e759d3e-611f-43e3-b520-2a5d34517d91\" (UID: \"0e759d3e-611f-43e3-b520-2a5d34517d91\") " Oct 31 13:35:38.030681 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:38.030609 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e759d3e-611f-43e3-b520-2a5d34517d91-plugins-config-volume" pod "0e759d3e-611f-43e3-b520-2a5d34517d91" (UID: "0e759d3e-611f-43e3-b520-2a5d34517d91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:35:38.033396 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:38.033362 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e759d3e-611f-43e3-b520-2a5d34517d91-kube-api-access-26dmg" pod "0e759d3e-611f-43e3-b520-2a5d34517d91" (UID: "0e759d3e-611f-43e3-b520-2a5d34517d91"). InnerVolumeSpecName "kube-api-access-26dmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:35:38.131258 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:38.131173 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e759d3e-611f-43e3-b520-2a5d34517d91-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:38.131258 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:38.131205 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26dmg\" (UniqueName: \"kubernetes.io/projected/0e759d3e-611f-43e3-b520-2a5d34517d91-kube-api-access-26dmg\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:38.735994 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:38.735902 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" event={"ID":"ddfd9d12-e8bb-42ee-a375-aba470b89eeb","Type":"ContainerStarted","Data":"323676112f62d08aa0da0625ee1bcf7a47d234088e905b86f1ce22e5d68bdd23"} Oct 31 13:35:38.735994 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:38.735966 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" event={"ID":"ddfd9d12-e8bb-42ee-a375-aba470b89eeb","Type":"ContainerStarted","Data":"09aee59fb47c0ab5e6b5d0d7bd78dd7a4085ccbcfb779c5b1045f0ed6764737b"} Oct 31 13:35:38.737579 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:38.737549 5693 generic.go:356] "Generic (PLEG): container finished" podID="0e759d3e-611f-43e3-b520-2a5d34517d91" containerID="c205c54e0e7cd8a502a036b71bcf839981d0dc6cadeee50f5d7bd35574dd6857" exitCode=0 Oct 31 13:35:38.737746 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:38.737589 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a39960-84b6db594c-lf44b" Oct 31 13:35:38.737746 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:38.737621 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a39960-84b6db594c-lf44b" event={"ID":"0e759d3e-611f-43e3-b520-2a5d34517d91","Type":"ContainerDied","Data":"c205c54e0e7cd8a502a036b71bcf839981d0dc6cadeee50f5d7bd35574dd6857"} Oct 31 13:35:38.737746 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:38.737697 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a39960-84b6db594c-lf44b" event={"ID":"0e759d3e-611f-43e3-b520-2a5d34517d91","Type":"ContainerDied","Data":"4e1150c5b9dfe2b229e455fe4bf890ce0f4ffe67357a2085596b2948b3233dce"} Oct 31 13:35:38.737746 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:38.737719 5693 scope.go:122] "RemoveContainer" containerID="c205c54e0e7cd8a502a036b71bcf839981d0dc6cadeee50f5d7bd35574dd6857" Oct 31 13:35:38.743207 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:38.743178 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" event={"ID":"bedfdf67-dd1e-4902-9ca8-4f404d3a0841","Type":"ContainerStarted","Data":"ecfb8eaafa0fca016f14c25f3264ff12834189fcd99bb34db8642a0b516d0679"} Oct 31 13:35:38.743324 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:38.743228 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" event={"ID":"bedfdf67-dd1e-4902-9ca8-4f404d3a0841","Type":"ContainerStarted","Data":"71b188e89648cd743bb9ccac0f81c9668a13bf9746d10bf52a2e5e45c19141c0"} Oct 31 13:35:38.752103 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:38.752002 5693 scope.go:122] "RemoveContainer" containerID="c205c54e0e7cd8a502a036b71bcf839981d0dc6cadeee50f5d7bd35574dd6857" Oct 31 13:35:38.752413 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:35:38.752390 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c205c54e0e7cd8a502a036b71bcf839981d0dc6cadeee50f5d7bd35574dd6857\": container with ID starting with c205c54e0e7cd8a502a036b71bcf839981d0dc6cadeee50f5d7bd35574dd6857 not found: ID does not exist" containerID="c205c54e0e7cd8a502a036b71bcf839981d0dc6cadeee50f5d7bd35574dd6857" Oct 31 13:35:38.752478 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:38.752422 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c205c54e0e7cd8a502a036b71bcf839981d0dc6cadeee50f5d7bd35574dd6857"} err="failed to get container status \"c205c54e0e7cd8a502a036b71bcf839981d0dc6cadeee50f5d7bd35574dd6857\": rpc error: code = NotFound desc = could not find container \"c205c54e0e7cd8a502a036b71bcf839981d0dc6cadeee50f5d7bd35574dd6857\": container with ID starting with c205c54e0e7cd8a502a036b71bcf839981d0dc6cadeee50f5d7bd35574dd6857 not found: ID does not exist" Oct 31 13:35:38.769194 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:38.769105 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" podStartSLOduration=1.7690916620000001 podStartE2EDuration="1.769091662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:35:38.763720889 +0000 UTC m=+3789.471307213" watchObservedRunningTime="2026-10-31 13:35:38.769091662 +0000 UTC m=+3789.476677957" Oct 31 13:35:38.845102 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:38.839287 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a39960-84b6db594c-lf44b"] Oct 31 13:35:38.847662 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:38.847624 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a39960-84b6db594c-lf44b"] Oct 31 13:35:38.850074 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:38.850033 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" podStartSLOduration=1.850018784 podStartE2EDuration="1.850018784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:35:38.838021284 +0000 UTC m=+3789.545607581" watchObservedRunningTime="2026-10-31 13:35:38.850018784 +0000 UTC m=+3789.557605147" Oct 31 13:35:39.388085 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:39.388053 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" Oct 31 13:35:39.407666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:39.406788 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-6b8s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body= Oct 31 13:35:39.407666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:39.406856 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" podUID="bedfdf67-dd1e-4902-9ca8-4f404d3a0841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" Oct 31 13:35:39.439231 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:39.439199 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e759d3e-611f-43e3-b520-2a5d34517d91" path="/var/lib/kubelet/pods/0e759d3e-611f-43e3-b520-2a5d34517d91/volumes" Oct 31 13:35:39.439386 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:39.439343 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-h4czf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body= Oct 31 13:35:39.439447 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:39.439388 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" podUID="ddfd9d12-e8bb-42ee-a375-aba470b89eeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" Oct 31 13:35:39.439654 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:39.439624 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" Oct 31 13:35:40.411826 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:40.411789 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-6b8s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body= Oct 31 13:35:40.412566 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:40.412532 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" podUID="bedfdf67-dd1e-4902-9ca8-4f404d3a0841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" Oct 31 13:35:40.446665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:40.438948 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-h4czf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body= Oct 31 13:35:40.446665 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:40.439002 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" podUID="ddfd9d12-e8bb-42ee-a375-aba470b89eeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" Oct 31 13:35:41.386996 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:41.386946 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-6b8s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body= Oct 31 13:35:41.387338 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:41.387284 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" podUID="bedfdf67-dd1e-4902-9ca8-4f404d3a0841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" Oct 31 13:35:41.440804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:41.440770 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-h4czf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body= Oct 31 13:35:41.441263 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:41.441240 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" podUID="ddfd9d12-e8bb-42ee-a375-aba470b89eeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" Oct 31 13:35:42.383920 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:42.383879 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-6b8s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body= Oct 31 13:35:42.384276 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:42.384239 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" podUID="bedfdf67-dd1e-4902-9ca8-4f404d3a0841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" Oct 31 13:35:42.439802 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:42.439760 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-h4czf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body= Oct 31 13:35:42.440102 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:42.440071 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" podUID="ddfd9d12-e8bb-42ee-a375-aba470b89eeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" Oct 31 13:35:43.382744 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:43.382702 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-6b8s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body= Oct 31 13:35:43.383915 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:43.383465 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" podUID="bedfdf67-dd1e-4902-9ca8-4f404d3a0841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" Oct 31 13:35:43.438203 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:43.438170 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-h4czf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body= Oct 31 13:35:43.438509 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:43.438482 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" podUID="ddfd9d12-e8bb-42ee-a375-aba470b89eeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" Oct 31 13:35:44.382561 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:44.382516 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-6b8s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body= Oct 31 13:35:44.382774 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:44.382598 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" podUID="bedfdf67-dd1e-4902-9ca8-4f404d3a0841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" Oct 31 13:35:44.449669 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:44.442031 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-h4czf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body= Oct 31 13:35:44.449669 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:44.442094 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" podUID="ddfd9d12-e8bb-42ee-a375-aba470b89eeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" Oct 31 13:35:45.382704 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:45.382661 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-6b8s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body= Oct 31 13:35:45.382864 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:45.382733 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" podUID="bedfdf67-dd1e-4902-9ca8-4f404d3a0841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" Oct 31 13:35:45.438422 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:45.438378 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-h4czf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body= Oct 31 13:35:45.438684 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:45.438447 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" podUID="ddfd9d12-e8bb-42ee-a375-aba470b89eeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" Oct 31 13:35:46.934259 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:46.934226 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" Oct 31 13:35:46.934746 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:46.934729 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" Oct 31 13:35:46.935704 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:46.935679 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" Oct 31 13:35:46.982868 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:46.982833 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" Oct 31 13:35:46.983352 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:46.983322 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" Oct 31 13:35:46.984241 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:35:46.984221 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" Oct 31 13:35:48.731380 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:35:48.731339 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:36:15.039111 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:36:15.039070 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61110dd0-xpjvk"] Oct 31 13:36:15.043031 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:36:15.042997 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61110dd0-xpjvk"] Oct 31 13:36:15.414146 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:36:15.414104 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e9833c8-a068-4613-983b-de79cae7c50e" path="/var/lib/kubelet/pods/4e9833c8-a068-4613-983b-de79cae7c50e/volumes" Oct 31 13:36:18.690809 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:36:18.690769 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:36:19.032091 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:36:19.032011 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d0ce4a0-l5m59"] Oct 31 13:36:19.049795 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:36:19.049766 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d0ce4a0-l5m59"] Oct 31 13:36:19.411517 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:36:19.411486 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f" path="/var/lib/kubelet/pods/4d3d89e5-7c07-4a62-85d1-9fb51c6e1a3f/volumes" Oct 31 13:36:26.036810 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:36:26.036774 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d16d82aa-c7cdx"] Oct 31 13:36:26.062889 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:36:26.062857 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d16d82aa-c7cdx"] Oct 31 13:36:27.410873 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:36:27.410841 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db209c8e-2550-4233-9f7b-a7fdac69ba44" path="/var/lib/kubelet/pods/db209c8e-2550-4233-9f7b-a7fdac69ba44/volumes" Oct 31 13:36:31.587376 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:36:31.587344 5693 scope.go:122] "RemoveContainer" containerID="c590d9c3a9e8cc8d801040a1c6323d96b2842c4a69627895099adfaa6ea168e0" Oct 31 13:36:35.024613 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:36:35.024578 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4f8865b-x64nl"] Oct 31 13:36:35.028626 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:36:35.028595 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4f8865b-x64nl"] Oct 31 13:36:35.419858 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:36:35.419826 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6d9c5e7-3377-48a7-90af-bafebaf8c52a" path="/var/lib/kubelet/pods/b6d9c5e7-3377-48a7-90af-bafebaf8c52a/volumes" Oct 31 13:36:48.691722 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:36:48.691682 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:36:53.531038 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:36:53.530989 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" podUID="ff9a9909-6fc3-475b-a9fe-562ec150c76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 31 13:36:53.782835 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:36:53.782719 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" podUID="710a52fa-2bc4-4bb9-96be-4b08dc60de67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 31 13:36:54.004864 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:36:54.004821 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" podUID="e5a40b79-9879-4981-ab9f-446e1750a013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 13:37:02.412953 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:02.412917 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-691f1e-569d8fc9cc-k6fn8"] Oct 31 13:37:02.420492 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:02.420458 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-691f1e-68bd657c5-whf6d"] Oct 31 13:37:07.413712 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:07.413673 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-691f1e-569d8fc9cc-k6fn8" podUID="30cc0f1f-cfc3-475d-8d48-a98f210b91f6" containerName="payload-processing" containerID="cri-o://65209c72d8b5a3378ac7d6e207966f4f8df1e679bdbe7a08e5420517ecfe0e00" gracePeriod=25 Oct 31 13:37:07.420925 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:07.420887 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-691f1e-68bd657c5-whf6d" podUID="497f3ecd-3d10-4ffd-ae25-490c570c4b1f" containerName="payload-pre-processing" containerID="cri-o://bdb16dab5bb358e91cad68c315c38f01eae7f3660d8c5c93a787facbe35994af" gracePeriod=25 Oct 31 13:37:07.810732 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:07.789870 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-691f1e-569d8fc9cc-k6fn8" Oct 31 13:37:07.904529 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:07.903529 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30cc0f1f-cfc3-475d-8d48-a98f210b91f6-kube-api-access-b4rfm\" (UniqueName: \"kubernetes.io/projected/30cc0f1f-cfc3-475d-8d48-a98f210b91f6-kube-api-access-b4rfm\") pod \"30cc0f1f-cfc3-475d-8d48-a98f210b91f6\" (UID: \"30cc0f1f-cfc3-475d-8d48-a98f210b91f6\") " Oct 31 13:37:07.904529 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:07.903743 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30cc0f1f-cfc3-475d-8d48-a98f210b91f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30cc0f1f-cfc3-475d-8d48-a98f210b91f6-plugins-config-volume\") pod \"30cc0f1f-cfc3-475d-8d48-a98f210b91f6\" (UID: \"30cc0f1f-cfc3-475d-8d48-a98f210b91f6\") " Oct 31 13:37:07.929057 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:07.928936 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30cc0f1f-cfc3-475d-8d48-a98f210b91f6-plugins-config-volume" pod "30cc0f1f-cfc3-475d-8d48-a98f210b91f6" (UID: "30cc0f1f-cfc3-475d-8d48-a98f210b91f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:37:07.954670 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:07.954569 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30cc0f1f-cfc3-475d-8d48-a98f210b91f6-kube-api-access-b4rfm" pod "30cc0f1f-cfc3-475d-8d48-a98f210b91f6" (UID: "30cc0f1f-cfc3-475d-8d48-a98f210b91f6"). InnerVolumeSpecName "kube-api-access-b4rfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:08.006504 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:08.006471 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4rfm\" (UniqueName: \"kubernetes.io/projected/30cc0f1f-cfc3-475d-8d48-a98f210b91f6-kube-api-access-b4rfm\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:08.006504 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:08.006501 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30cc0f1f-cfc3-475d-8d48-a98f210b91f6-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:08.019348 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:08.019317 5693 generic.go:356] "Generic (PLEG): container finished" podID="30cc0f1f-cfc3-475d-8d48-a98f210b91f6" containerID="65209c72d8b5a3378ac7d6e207966f4f8df1e679bdbe7a08e5420517ecfe0e00" exitCode=0 Oct 31 13:37:08.019552 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:08.019374 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-691f1e-569d8fc9cc-k6fn8" Oct 31 13:37:08.019552 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:08.019412 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-691f1e-569d8fc9cc-k6fn8" event={"ID":"30cc0f1f-cfc3-475d-8d48-a98f210b91f6","Type":"ContainerDied","Data":"65209c72d8b5a3378ac7d6e207966f4f8df1e679bdbe7a08e5420517ecfe0e00"} Oct 31 13:37:08.019552 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:08.019505 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-691f1e-569d8fc9cc-k6fn8" event={"ID":"30cc0f1f-cfc3-475d-8d48-a98f210b91f6","Type":"ContainerDied","Data":"60256b760ef1fecfd864811de61f9bebad84596945f75f989827cd155b48b371"} Oct 31 13:37:08.019552 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:08.019536 5693 scope.go:122] "RemoveContainer" containerID="65209c72d8b5a3378ac7d6e207966f4f8df1e679bdbe7a08e5420517ecfe0e00" Oct 31 13:37:08.031942 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:08.031920 5693 scope.go:122] "RemoveContainer" containerID="65209c72d8b5a3378ac7d6e207966f4f8df1e679bdbe7a08e5420517ecfe0e00" Oct 31 13:37:08.032210 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:37:08.032193 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65209c72d8b5a3378ac7d6e207966f4f8df1e679bdbe7a08e5420517ecfe0e00\": container with ID starting with 65209c72d8b5a3378ac7d6e207966f4f8df1e679bdbe7a08e5420517ecfe0e00 not found: ID does not exist" containerID="65209c72d8b5a3378ac7d6e207966f4f8df1e679bdbe7a08e5420517ecfe0e00" Oct 31 13:37:08.032262 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:08.032222 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65209c72d8b5a3378ac7d6e207966f4f8df1e679bdbe7a08e5420517ecfe0e00"} err="failed to get container status \"65209c72d8b5a3378ac7d6e207966f4f8df1e679bdbe7a08e5420517ecfe0e00\": rpc error: code = NotFound desc = could not find container \"65209c72d8b5a3378ac7d6e207966f4f8df1e679bdbe7a08e5420517ecfe0e00\": container with ID starting with 65209c72d8b5a3378ac7d6e207966f4f8df1e679bdbe7a08e5420517ecfe0e00 not found: ID does not exist" Oct 31 13:37:08.045554 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:08.045525 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-691f1e-569d8fc9cc-k6fn8"] Oct 31 13:37:08.051284 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:08.051260 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-691f1e-569d8fc9cc-k6fn8"] Oct 31 13:37:09.411202 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:09.411125 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30cc0f1f-cfc3-475d-8d48-a98f210b91f6" path="/var/lib/kubelet/pods/30cc0f1f-cfc3-475d-8d48-a98f210b91f6/volumes" Oct 31 13:37:10.127325 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:10.127290 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-691f1e-68bd657c5-whf6d" podUID="497f3ecd-3d10-4ffd-ae25-490c570c4b1f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:37:10.599785 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:10.599743 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp"] Oct 31 13:37:10.600218 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:10.600026 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" podUID="efccc572-e9f9-4f4a-ab45-c7458b0e2e58" containerName="istio-proxy" containerID="cri-o://53f8b1541faf3c232db0d1c3cb272695960884b719c385e5decd2325e2f290ae" gracePeriod=30 Oct 31 13:37:15.890022 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:15.889963 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" Oct 31 13:37:15.930687 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:15.930612 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df"] Oct 31 13:37:15.930895 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:15.930873 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" podUID="b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab" containerName="istio-proxy" containerID="cri-o://61c01bf4b1dabc339d8fbf113111f0c6be8374695a537a4d7a39824c1549d19f" gracePeriod=30 Oct 31 13:37:15.983867 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:15.983830 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx"] Oct 31 13:37:15.984068 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:15.984046 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" podUID="f368dfe0-177d-4e5f-9d68-f0f440717fe2" containerName="istio-proxy" containerID="cri-o://f4ae3f8c7bcbeb87b58a8a8db24a1c50661483091f435f72d16ab0b17d318bef" gracePeriod=30 Oct 31 13:37:16.063889 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:16.063858 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-istio-envoy\") pod \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\" (UID: \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\") " Oct 31 13:37:16.063889 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:16.063895 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-workload-certs\") pod \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\" (UID: \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\") " Oct 31 13:37:16.064165 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:16.064023 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-kube-api-access-2pw84\" (UniqueName: \"kubernetes.io/projected/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-kube-api-access-2pw84\") pod \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\" (UID: \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\") " Oct 31 13:37:16.064165 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:16.064102 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-istio-podinfo\") pod \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\" (UID: \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\") " Oct 31 13:37:16.064286 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:16.064194 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-istio-token\" (UniqueName: \"kubernetes.io/projected/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-istio-token\") pod \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\" (UID: \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\") " Oct 31 13:37:16.064286 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:16.064243 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-workload-certs" pod "efccc572-e9f9-4f4a-ab45-c7458b0e2e58" (UID: "efccc572-e9f9-4f4a-ab45-c7458b0e2e58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:16.064286 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:16.064280 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-workload-socket\") pod \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\" (UID: \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\") " Oct 31 13:37:16.064462 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:16.064347 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-credential-socket\") pod \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\" (UID: \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\") " Oct 31 13:37:16.064462 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:16.064454 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-istio-data\") pod \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\" (UID: \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\") " Oct 31 13:37:16.064580 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:16.064515 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-workload-socket" pod "efccc572-e9f9-4f4a-ab45-c7458b0e2e58" (UID: "efccc572-e9f9-4f4a-ab45-c7458b0e2e58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:16.064580 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:16.064536 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-istiod-ca-cert\") pod \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\" (UID: \"efccc572-e9f9-4f4a-ab45-c7458b0e2e58\") " Oct 31 13:37:16.064756 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:16.064725 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-credential-socket" pod "efccc572-e9f9-4f4a-ab45-c7458b0e2e58" (UID: "efccc572-e9f9-4f4a-ab45-c7458b0e2e58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:16.064817 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:16.064754 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-istio-data" pod "efccc572-e9f9-4f4a-ab45-c7458b0e2e58" (UID: "efccc572-e9f9-4f4a-ab45-c7458b0e2e58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:16.065111 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:16.065090 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-istiod-ca-cert" pod "efccc572-e9f9-4f4a-ab45-c7458b0e2e58" (UID: "efccc572-e9f9-4f4a-ab45-c7458b0e2e58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:37:16.065524 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:16.065488 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:16.065577 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:16.065521 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:16.065577 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:16.065536 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:16.065577 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:16.065550 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:16.065577 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:16.065563 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:16.066565 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:16.066545 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-istio-token" pod "efccc572-e9f9-4f4a-ab45-c7458b0e2e58" (UID: "efccc572-e9f9-4f4a-ab45-c7458b0e2e58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:16.066679 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:16.066628 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-kube-api-access-2pw84" pod "efccc572-e9f9-4f4a-ab45-c7458b0e2e58" (UID: "efccc572-e9f9-4f4a-ab45-c7458b0e2e58"). InnerVolumeSpecName "kube-api-access-2pw84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:16.066846 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:16.066828 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-istio-envoy" pod "efccc572-e9f9-4f4a-ab45-c7458b0e2e58" (UID: "efccc572-e9f9-4f4a-ab45-c7458b0e2e58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:16.066895 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:16.066849 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-istio-podinfo" pod "efccc572-e9f9-4f4a-ab45-c7458b0e2e58" (UID: "efccc572-e9f9-4f4a-ab45-c7458b0e2e58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:37:16.091907 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:16.091867 5693 generic.go:356] "Generic (PLEG): container finished" podID="efccc572-e9f9-4f4a-ab45-c7458b0e2e58" containerID="53f8b1541faf3c232db0d1c3cb272695960884b719c385e5decd2325e2f290ae" exitCode=0 Oct 31 13:37:16.092079 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:16.091941 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" Oct 31 13:37:16.092157 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:16.092100 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" event={"ID":"efccc572-e9f9-4f4a-ab45-c7458b0e2e58","Type":"ContainerDied","Data":"53f8b1541faf3c232db0d1c3cb272695960884b719c385e5decd2325e2f290ae"} Oct 31 13:37:16.092294 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:16.092265 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp" event={"ID":"efccc572-e9f9-4f4a-ab45-c7458b0e2e58","Type":"ContainerDied","Data":"94228db49135d44ced3fb1feb111429775198883770eae5a13f12083f71f79d1"} Oct 31 13:37:16.092412 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:16.092318 5693 scope.go:122] "RemoveContainer" containerID="53f8b1541faf3c232db0d1c3cb272695960884b719c385e5decd2325e2f290ae" Oct 31 13:37:16.116207 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:16.116183 5693 scope.go:122] "RemoveContainer" containerID="15a949ec75dd1373b9bbb8b3ec5121bcd53fe94ec3e99f9369b78d8e66383412" Oct 31 13:37:16.128431 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:16.128405 5693 scope.go:122] "RemoveContainer" containerID="53f8b1541faf3c232db0d1c3cb272695960884b719c385e5decd2325e2f290ae" Oct 31 13:37:16.128765 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:37:16.128740 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53f8b1541faf3c232db0d1c3cb272695960884b719c385e5decd2325e2f290ae\": container with ID starting with 53f8b1541faf3c232db0d1c3cb272695960884b719c385e5decd2325e2f290ae not found: ID does not exist" containerID="53f8b1541faf3c232db0d1c3cb272695960884b719c385e5decd2325e2f290ae" Oct 31 13:37:16.128824 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:16.128773 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53f8b1541faf3c232db0d1c3cb272695960884b719c385e5decd2325e2f290ae"} err="failed to get container status \"53f8b1541faf3c232db0d1c3cb272695960884b719c385e5decd2325e2f290ae\": rpc error: code = NotFound desc = could not find container \"53f8b1541faf3c232db0d1c3cb272695960884b719c385e5decd2325e2f290ae\": container with ID starting with 53f8b1541faf3c232db0d1c3cb272695960884b719c385e5decd2325e2f290ae not found: ID does not exist" Oct 31 13:37:16.128824 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:16.128802 5693 scope.go:122] "RemoveContainer" containerID="15a949ec75dd1373b9bbb8b3ec5121bcd53fe94ec3e99f9369b78d8e66383412" Oct 31 13:37:16.129163 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:37:16.129122 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15a949ec75dd1373b9bbb8b3ec5121bcd53fe94ec3e99f9369b78d8e66383412\": container with ID starting with 15a949ec75dd1373b9bbb8b3ec5121bcd53fe94ec3e99f9369b78d8e66383412 not found: ID does not exist" containerID="15a949ec75dd1373b9bbb8b3ec5121bcd53fe94ec3e99f9369b78d8e66383412" Oct 31 13:37:16.129242 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:16.129152 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15a949ec75dd1373b9bbb8b3ec5121bcd53fe94ec3e99f9369b78d8e66383412"} err="failed to get container status \"15a949ec75dd1373b9bbb8b3ec5121bcd53fe94ec3e99f9369b78d8e66383412\": rpc error: code = NotFound desc = could not find container \"15a949ec75dd1373b9bbb8b3ec5121bcd53fe94ec3e99f9369b78d8e66383412\": container with ID starting with 15a949ec75dd1373b9bbb8b3ec5121bcd53fe94ec3e99f9369b78d8e66383412 not found: ID does not exist" Oct 31 13:37:16.130469 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:16.130412 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp"] Oct 31 13:37:16.134663 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:16.134625 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-691f1e-openshift-default-8684f6ff95-4g4gp"] Oct 31 13:37:16.167167 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:16.167140 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:16.167291 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:16.167168 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pw84\" (UniqueName: \"kubernetes.io/projected/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-kube-api-access-2pw84\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:16.167291 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:16.167182 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:16.167291 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:16.167190 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efccc572-e9f9-4f4a-ab45-c7458b0e2e58-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:17.411226 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:17.411189 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efccc572-e9f9-4f4a-ab45-c7458b0e2e58" path="/var/lib/kubelet/pods/efccc572-e9f9-4f4a-ab45-c7458b0e2e58/volumes" Oct 31 13:37:17.986739 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:17.986705 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3cddb3-6b9b59f449-7cfsr"] Oct 31 13:37:18.691509 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:37:18.691476 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:37:20.127110 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:20.127068 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-691f1e-68bd657c5-whf6d" podUID="497f3ecd-3d10-4ffd-ae25-490c570c4b1f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:37:20.810917 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:20.810880 5693 patch_prober.go:29] interesting pod/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:37:20.811150 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:20.810942 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" podUID="b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:37:21.149194 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.147503 5693 generic.go:356] "Generic (PLEG): container finished" podID="f368dfe0-177d-4e5f-9d68-f0f440717fe2" containerID="f4ae3f8c7bcbeb87b58a8a8db24a1c50661483091f435f72d16ab0b17d318bef" exitCode=0 Oct 31 13:37:21.149194 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.147863 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" event={"ID":"f368dfe0-177d-4e5f-9d68-f0f440717fe2","Type":"ContainerDied","Data":"f4ae3f8c7bcbeb87b58a8a8db24a1c50661483091f435f72d16ab0b17d318bef"} Oct 31 13:37:21.152414 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.150065 5693 generic.go:356] "Generic (PLEG): container finished" podID="b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab" containerID="61c01bf4b1dabc339d8fbf113111f0c6be8374695a537a4d7a39824c1549d19f" exitCode=0 Oct 31 13:37:21.152414 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.150150 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" event={"ID":"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab","Type":"ContainerDied","Data":"61c01bf4b1dabc339d8fbf113111f0c6be8374695a537a4d7a39824c1549d19f"} Oct 31 13:37:21.266008 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.265978 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" Oct 31 13:37:21.301510 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.301476 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-istiod-ca-cert\") pod \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\" (UID: \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\") " Oct 31 13:37:21.301746 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.301550 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-istio-token\" (UniqueName: \"kubernetes.io/projected/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-istio-token\") pod \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\" (UID: \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\") " Oct 31 13:37:21.301746 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.301615 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-istio-data\") pod \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\" (UID: \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\") " Oct 31 13:37:21.302525 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.301884 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-workload-socket\") pod \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\" (UID: \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\") " Oct 31 13:37:21.302525 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.301970 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-credential-socket\") pod \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\" (UID: \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\") " Oct 31 13:37:21.302525 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.301985 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-istiod-ca-cert" pod "b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab" (UID: "b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:37:21.302525 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.302024 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-istio-envoy\") pod \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\" (UID: \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\") " Oct 31 13:37:21.302525 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.302079 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-kube-api-access-pg7z7\" (UniqueName: \"kubernetes.io/projected/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-kube-api-access-pg7z7\") pod \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\" (UID: \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\") " Oct 31 13:37:21.302525 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.302262 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-workload-certs\") pod \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\" (UID: \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\") " Oct 31 13:37:21.302525 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.302317 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-istio-podinfo\") pod \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\" (UID: \"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab\") " Oct 31 13:37:21.303313 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.302789 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-workload-certs" pod "b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab" (UID: "b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:21.303313 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.303089 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:21.303313 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.303110 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:21.311685 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.306308 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-istio-data" pod "b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab" (UID: "b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:21.311685 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.306376 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-credential-socket" pod "b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab" (UID: "b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:21.311685 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.306590 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-workload-socket" pod "b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab" (UID: "b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:21.311685 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.308465 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-istio-envoy" pod "b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab" (UID: "b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:21.311685 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.308796 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-istio-podinfo" pod "b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab" (UID: "b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:37:21.311685 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.309007 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-kube-api-access-pg7z7" pod "b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab" (UID: "b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab"). InnerVolumeSpecName "kube-api-access-pg7z7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:21.311685 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.309068 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-istio-token" pod "b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab" (UID: "b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:21.342896 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.342865 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" Oct 31 13:37:21.403582 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.403527 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f368dfe0-177d-4e5f-9d68-f0f440717fe2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f368dfe0-177d-4e5f-9d68-f0f440717fe2-workload-certs\") pod \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\" (UID: \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\") " Oct 31 13:37:21.403811 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.403631 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f368dfe0-177d-4e5f-9d68-f0f440717fe2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f368dfe0-177d-4e5f-9d68-f0f440717fe2-istio-data\") pod \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\" (UID: \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\") " Oct 31 13:37:21.403811 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.403708 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f368dfe0-177d-4e5f-9d68-f0f440717fe2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f368dfe0-177d-4e5f-9d68-f0f440717fe2-credential-socket\") pod \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\" (UID: \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\") " Oct 31 13:37:21.403811 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.403747 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f368dfe0-177d-4e5f-9d68-f0f440717fe2-istio-token\" (UniqueName: \"kubernetes.io/projected/f368dfe0-177d-4e5f-9d68-f0f440717fe2-istio-token\") pod \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\" (UID: \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\") " Oct 31 13:37:21.403937 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.403924 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f368dfe0-177d-4e5f-9d68-f0f440717fe2-kube-api-access-bdz9r\" (UniqueName: \"kubernetes.io/projected/f368dfe0-177d-4e5f-9d68-f0f440717fe2-kube-api-access-bdz9r\") pod \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\" (UID: \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\") " Oct 31 13:37:21.403971 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.403961 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f368dfe0-177d-4e5f-9d68-f0f440717fe2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f368dfe0-177d-4e5f-9d68-f0f440717fe2-istiod-ca-cert\") pod \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\" (UID: \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\") " Oct 31 13:37:21.404033 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.404016 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f368dfe0-177d-4e5f-9d68-f0f440717fe2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f368dfe0-177d-4e5f-9d68-f0f440717fe2-istio-podinfo\") pod \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\" (UID: \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\") " Oct 31 13:37:21.404113 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.404079 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f368dfe0-177d-4e5f-9d68-f0f440717fe2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f368dfe0-177d-4e5f-9d68-f0f440717fe2-istio-envoy\") pod \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\" (UID: \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\") " Oct 31 13:37:21.404168 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.404129 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f368dfe0-177d-4e5f-9d68-f0f440717fe2-istio-data" pod "f368dfe0-177d-4e5f-9d68-f0f440717fe2" (UID: "f368dfe0-177d-4e5f-9d68-f0f440717fe2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:21.404232 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.404173 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f368dfe0-177d-4e5f-9d68-f0f440717fe2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f368dfe0-177d-4e5f-9d68-f0f440717fe2-workload-socket\") pod \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\" (UID: \"f368dfe0-177d-4e5f-9d68-f0f440717fe2\") " Oct 31 13:37:21.404354 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.404337 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f368dfe0-177d-4e5f-9d68-f0f440717fe2-workload-certs" pod "f368dfe0-177d-4e5f-9d68-f0f440717fe2" (UID: "f368dfe0-177d-4e5f-9d68-f0f440717fe2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:21.405109 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.404546 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f368dfe0-177d-4e5f-9d68-f0f440717fe2-workload-socket" pod "f368dfe0-177d-4e5f-9d68-f0f440717fe2" (UID: "f368dfe0-177d-4e5f-9d68-f0f440717fe2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:21.405109 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.404831 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f368dfe0-177d-4e5f-9d68-f0f440717fe2-credential-socket" pod "f368dfe0-177d-4e5f-9d68-f0f440717fe2" (UID: "f368dfe0-177d-4e5f-9d68-f0f440717fe2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:21.405109 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.404893 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f368dfe0-177d-4e5f-9d68-f0f440717fe2-istiod-ca-cert" pod "f368dfe0-177d-4e5f-9d68-f0f440717fe2" (UID: "f368dfe0-177d-4e5f-9d68-f0f440717fe2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:37:21.405373 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.405235 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f368dfe0-177d-4e5f-9d68-f0f440717fe2-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:21.405373 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.405260 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:21.405373 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.405276 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:21.405373 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.405295 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg7z7\" (UniqueName: \"kubernetes.io/projected/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-kube-api-access-pg7z7\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:21.405373 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.405310 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f368dfe0-177d-4e5f-9d68-f0f440717fe2-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:21.405373 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.405325 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:21.405373 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.405339 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:21.405373 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.405353 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:21.405373 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.405369 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f368dfe0-177d-4e5f-9d68-f0f440717fe2-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:21.406062 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.405383 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:21.406911 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.406881 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f368dfe0-177d-4e5f-9d68-f0f440717fe2-istio-token" pod "f368dfe0-177d-4e5f-9d68-f0f440717fe2" (UID: "f368dfe0-177d-4e5f-9d68-f0f440717fe2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:21.407510 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.407295 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f368dfe0-177d-4e5f-9d68-f0f440717fe2-istio-envoy" pod "f368dfe0-177d-4e5f-9d68-f0f440717fe2" (UID: "f368dfe0-177d-4e5f-9d68-f0f440717fe2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:21.430185 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.430152 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f368dfe0-177d-4e5f-9d68-f0f440717fe2-kube-api-access-bdz9r" pod "f368dfe0-177d-4e5f-9d68-f0f440717fe2" (UID: "f368dfe0-177d-4e5f-9d68-f0f440717fe2"). InnerVolumeSpecName "kube-api-access-bdz9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:21.430185 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.430165 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f368dfe0-177d-4e5f-9d68-f0f440717fe2-istio-podinfo" pod "f368dfe0-177d-4e5f-9d68-f0f440717fe2" (UID: "f368dfe0-177d-4e5f-9d68-f0f440717fe2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:37:21.507701 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.507614 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f368dfe0-177d-4e5f-9d68-f0f440717fe2-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:21.507701 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.507662 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f368dfe0-177d-4e5f-9d68-f0f440717fe2-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:21.507701 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.507676 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f368dfe0-177d-4e5f-9d68-f0f440717fe2-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:21.507701 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.507688 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f368dfe0-177d-4e5f-9d68-f0f440717fe2-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:21.507701 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.507701 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdz9r\" (UniqueName: \"kubernetes.io/projected/f368dfe0-177d-4e5f-9d68-f0f440717fe2-kube-api-access-bdz9r\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:21.508180 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:21.507716 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f368dfe0-177d-4e5f-9d68-f0f440717fe2-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:22.173076 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:22.173048 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" Oct 31 13:37:22.173817 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:22.173044 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df" event={"ID":"b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab","Type":"ContainerDied","Data":"988fe92c11a2fe8487af03b8ec52666d8680b388b7e0109cbda51e2167d25ff3"} Oct 31 13:37:22.173936 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:22.173875 5693 scope.go:122] "RemoveContainer" containerID="61c01bf4b1dabc339d8fbf113111f0c6be8374695a537a4d7a39824c1549d19f" Oct 31 13:37:22.176330 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:22.176305 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" event={"ID":"f368dfe0-177d-4e5f-9d68-f0f440717fe2","Type":"ContainerDied","Data":"74430db3282c9b59a1ed9925855e82e1e45174ef20d522084c82cb9d0fab71a5"} Oct 31 13:37:22.176452 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:22.176404 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx" Oct 31 13:37:22.187430 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:22.187412 5693 scope.go:122] "RemoveContainer" containerID="f4ae3f8c7bcbeb87b58a8a8db24a1c50661483091f435f72d16ab0b17d318bef" Oct 31 13:37:22.207050 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:22.207029 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df"] Oct 31 13:37:22.211149 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:22.211125 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-562df"] Oct 31 13:37:22.222158 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:22.222086 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx"] Oct 31 13:37:22.225874 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:22.225854 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-m6mmx"] Oct 31 13:37:22.988009 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:22.987961 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-3cddb3-6b9b59f449-7cfsr" podUID="d92607b7-b885-43e2-9694-7bcb94d4f6b5" containerName="payload-processing" containerID="cri-o://c1a915dbcedb5f1aa3720a47b8409db69fe2a6b9e8b1502b760e0dd8fdc4015d" gracePeriod=25 Oct 31 13:37:23.190543 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:23.190512 5693 generic.go:356] "Generic (PLEG): container finished" podID="d92607b7-b885-43e2-9694-7bcb94d4f6b5" containerID="c1a915dbcedb5f1aa3720a47b8409db69fe2a6b9e8b1502b760e0dd8fdc4015d" exitCode=0 Oct 31 13:37:23.191062 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:23.190690 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3cddb3-6b9b59f449-7cfsr" event={"ID":"d92607b7-b885-43e2-9694-7bcb94d4f6b5","Type":"ContainerDied","Data":"c1a915dbcedb5f1aa3720a47b8409db69fe2a6b9e8b1502b760e0dd8fdc4015d"} Oct 31 13:37:23.282326 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:23.282290 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3cddb3-6b9b59f449-7cfsr" Oct 31 13:37:23.336222 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:23.336194 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d92607b7-b885-43e2-9694-7bcb94d4f6b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d92607b7-b885-43e2-9694-7bcb94d4f6b5-plugins-config-volume\") pod \"d92607b7-b885-43e2-9694-7bcb94d4f6b5\" (UID: \"d92607b7-b885-43e2-9694-7bcb94d4f6b5\") " Oct 31 13:37:23.336399 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:23.336314 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d92607b7-b885-43e2-9694-7bcb94d4f6b5-kube-api-access-5gtfh\" (UniqueName: \"kubernetes.io/projected/d92607b7-b885-43e2-9694-7bcb94d4f6b5-kube-api-access-5gtfh\") pod \"d92607b7-b885-43e2-9694-7bcb94d4f6b5\" (UID: \"d92607b7-b885-43e2-9694-7bcb94d4f6b5\") " Oct 31 13:37:23.336633 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:23.336612 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d92607b7-b885-43e2-9694-7bcb94d4f6b5-plugins-config-volume" pod "d92607b7-b885-43e2-9694-7bcb94d4f6b5" (UID: "d92607b7-b885-43e2-9694-7bcb94d4f6b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:37:23.337004 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:23.336984 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d92607b7-b885-43e2-9694-7bcb94d4f6b5-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:23.339091 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:23.339053 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d92607b7-b885-43e2-9694-7bcb94d4f6b5-kube-api-access-5gtfh" pod "d92607b7-b885-43e2-9694-7bcb94d4f6b5" (UID: "d92607b7-b885-43e2-9694-7bcb94d4f6b5"). InnerVolumeSpecName "kube-api-access-5gtfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:23.412167 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:23.412129 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab" path="/var/lib/kubelet/pods/b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab/volumes" Oct 31 13:37:23.412803 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:23.412779 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f368dfe0-177d-4e5f-9d68-f0f440717fe2" path="/var/lib/kubelet/pods/f368dfe0-177d-4e5f-9d68-f0f440717fe2/volumes" Oct 31 13:37:23.438480 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:23.438445 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gtfh\" (UniqueName: \"kubernetes.io/projected/d92607b7-b885-43e2-9694-7bcb94d4f6b5-kube-api-access-5gtfh\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:24.203551 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:24.203510 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3cddb3-6b9b59f449-7cfsr" event={"ID":"d92607b7-b885-43e2-9694-7bcb94d4f6b5","Type":"ContainerDied","Data":"4c553ac9d2b645f24b2c2f8dbfc6fa73d12ea4f926e51e1636dc836af8688744"} Oct 31 13:37:24.204098 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:24.203577 5693 scope.go:122] "RemoveContainer" containerID="c1a915dbcedb5f1aa3720a47b8409db69fe2a6b9e8b1502b760e0dd8fdc4015d" Oct 31 13:37:24.204098 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:24.203518 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3cddb3-6b9b59f449-7cfsr" Oct 31 13:37:24.224243 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:24.224215 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3cddb3-6b9b59f449-7cfsr"] Oct 31 13:37:24.228447 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:24.228423 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3cddb3-6b9b59f449-7cfsr"] Oct 31 13:37:25.412368 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:25.412331 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d92607b7-b885-43e2-9694-7bcb94d4f6b5" path="/var/lib/kubelet/pods/d92607b7-b885-43e2-9694-7bcb94d4f6b5/volumes" Oct 31 13:37:27.712720 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:27.712691 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-691f1e-68bd657c5-whf6d" Oct 31 13:37:27.827511 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:27.827479 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/497f3ecd-3d10-4ffd-ae25-490c570c4b1f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/497f3ecd-3d10-4ffd-ae25-490c570c4b1f-plugins-config-volume\") pod \"497f3ecd-3d10-4ffd-ae25-490c570c4b1f\" (UID: \"497f3ecd-3d10-4ffd-ae25-490c570c4b1f\") " Oct 31 13:37:27.827741 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:27.827722 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/497f3ecd-3d10-4ffd-ae25-490c570c4b1f-kube-api-access-nm5tk\" (UniqueName: \"kubernetes.io/projected/497f3ecd-3d10-4ffd-ae25-490c570c4b1f-kube-api-access-nm5tk\") pod \"497f3ecd-3d10-4ffd-ae25-490c570c4b1f\" (UID: \"497f3ecd-3d10-4ffd-ae25-490c570c4b1f\") " Oct 31 13:37:27.827890 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:27.827867 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/497f3ecd-3d10-4ffd-ae25-490c570c4b1f-plugins-config-volume" pod "497f3ecd-3d10-4ffd-ae25-490c570c4b1f" (UID: "497f3ecd-3d10-4ffd-ae25-490c570c4b1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:37:27.828337 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:27.828317 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/497f3ecd-3d10-4ffd-ae25-490c570c4b1f-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:27.830104 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:27.830073 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/497f3ecd-3d10-4ffd-ae25-490c570c4b1f-kube-api-access-nm5tk" pod "497f3ecd-3d10-4ffd-ae25-490c570c4b1f" (UID: "497f3ecd-3d10-4ffd-ae25-490c570c4b1f"). InnerVolumeSpecName "kube-api-access-nm5tk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:27.930093 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:27.930062 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm5tk\" (UniqueName: \"kubernetes.io/projected/497f3ecd-3d10-4ffd-ae25-490c570c4b1f-kube-api-access-nm5tk\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:28.004247 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:28.004153 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-09db67/test-model-09db67-kserve-7478f9db55-gz5h5"] Oct 31 13:37:28.242020 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:28.241986 5693 generic.go:356] "Generic (PLEG): container finished" podID="497f3ecd-3d10-4ffd-ae25-490c570c4b1f" containerID="bdb16dab5bb358e91cad68c315c38f01eae7f3660d8c5c93a787facbe35994af" exitCode=0 Oct 31 13:37:28.242182 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:28.242045 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-691f1e-68bd657c5-whf6d" Oct 31 13:37:28.242182 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:28.242062 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-691f1e-68bd657c5-whf6d" event={"ID":"497f3ecd-3d10-4ffd-ae25-490c570c4b1f","Type":"ContainerDied","Data":"bdb16dab5bb358e91cad68c315c38f01eae7f3660d8c5c93a787facbe35994af"} Oct 31 13:37:28.242182 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:28.242120 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-691f1e-68bd657c5-whf6d" event={"ID":"497f3ecd-3d10-4ffd-ae25-490c570c4b1f","Type":"ContainerDied","Data":"140a321a52480f28ede4750f2bddb854e027015c63cc4a370c1d361da3fd6a5e"} Oct 31 13:37:28.242182 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:28.242141 5693 scope.go:122] "RemoveContainer" containerID="bdb16dab5bb358e91cad68c315c38f01eae7f3660d8c5c93a787facbe35994af" Oct 31 13:37:28.255479 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:28.255460 5693 scope.go:122] "RemoveContainer" containerID="bdb16dab5bb358e91cad68c315c38f01eae7f3660d8c5c93a787facbe35994af" Oct 31 13:37:28.255862 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:37:28.255839 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdb16dab5bb358e91cad68c315c38f01eae7f3660d8c5c93a787facbe35994af\": container with ID starting with bdb16dab5bb358e91cad68c315c38f01eae7f3660d8c5c93a787facbe35994af not found: ID does not exist" containerID="bdb16dab5bb358e91cad68c315c38f01eae7f3660d8c5c93a787facbe35994af" Oct 31 13:37:28.255956 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:28.255870 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdb16dab5bb358e91cad68c315c38f01eae7f3660d8c5c93a787facbe35994af"} err="failed to get container status \"bdb16dab5bb358e91cad68c315c38f01eae7f3660d8c5c93a787facbe35994af\": rpc error: code = NotFound desc = could not find container \"bdb16dab5bb358e91cad68c315c38f01eae7f3660d8c5c93a787facbe35994af\": container with ID starting with bdb16dab5bb358e91cad68c315c38f01eae7f3660d8c5c93a787facbe35994af not found: ID does not exist" Oct 31 13:37:28.268618 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:28.268589 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-691f1e-68bd657c5-whf6d"] Oct 31 13:37:28.275145 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:28.275117 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-691f1e-68bd657c5-whf6d"] Oct 31 13:37:29.410704 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:29.410677 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="497f3ecd-3d10-4ffd-ae25-490c570c4b1f" path="/var/lib/kubelet/pods/497f3ecd-3d10-4ffd-ae25-490c570c4b1f/volumes" Oct 31 13:37:31.637417 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:31.637385 5693 scope.go:122] "RemoveContainer" containerID="d3a99eed7312d4a5bd339a5527d2c8297b4bc27dd81dc60f82ba847830cd91c1" Oct 31 13:37:31.649186 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:31.649164 5693 scope.go:122] "RemoveContainer" containerID="20c347ba957c868d19521d8a6bb37cd0edc85b5fc4b5d5d52a891fb91a59138a" Oct 31 13:37:31.659688 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:31.659667 5693 scope.go:122] "RemoveContainer" containerID="3ac9f31da84d6a9c3f0b3ea588a14b5c9a79f7ba158b3c2dfc01428c2159080b" Oct 31 13:37:43.039046 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:43.038989 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-09db67/test-model-09db67-kserve-7478f9db55-gz5h5" podUID="cb838e2b-f206-4da9-97bc-a160de56256f" containerName="main" containerID="cri-o://c88488c7133599370b30b2710f1396014636d853148ce28b02bb3fe5e5c335a1" gracePeriod=45 Oct 31 13:37:43.209039 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:43.209000 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-09db67/test-model-09db67-kserve-7478f9db55-gz5h5" podUID="cb838e2b-f206-4da9-97bc-a160de56256f" containerName="main" probeResult="failure" output="Get \"https://10.133.1.23:8000/ready\": dial tcp 10.133.1.23:8000: connect: connection refused" Oct 31 13:37:43.456678 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:43.450732 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-09db67/test-model-09db67-kserve-7478f9db55-gz5h5" Oct 31 13:37:43.575239 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:43.575197 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb838e2b-f206-4da9-97bc-a160de56256f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb838e2b-f206-4da9-97bc-a160de56256f-model-cache\") pod \"cb838e2b-f206-4da9-97bc-a160de56256f\" (UID: \"cb838e2b-f206-4da9-97bc-a160de56256f\") " Oct 31 13:37:43.575426 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:43.575372 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb838e2b-f206-4da9-97bc-a160de56256f-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb838e2b-f206-4da9-97bc-a160de56256f-tls-certs\") pod \"cb838e2b-f206-4da9-97bc-a160de56256f\" (UID: \"cb838e2b-f206-4da9-97bc-a160de56256f\") " Oct 31 13:37:43.575474 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:43.575454 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb838e2b-f206-4da9-97bc-a160de56256f-kube-api-access-mtlcq\" (UniqueName: \"kubernetes.io/projected/cb838e2b-f206-4da9-97bc-a160de56256f-kube-api-access-mtlcq\") pod \"cb838e2b-f206-4da9-97bc-a160de56256f\" (UID: \"cb838e2b-f206-4da9-97bc-a160de56256f\") " Oct 31 13:37:43.575520 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:43.575495 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb838e2b-f206-4da9-97bc-a160de56256f-model-cache" pod "cb838e2b-f206-4da9-97bc-a160de56256f" (UID: "cb838e2b-f206-4da9-97bc-a160de56256f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:43.575593 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:43.575560 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb838e2b-f206-4da9-97bc-a160de56256f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb838e2b-f206-4da9-97bc-a160de56256f-dshm\") pod \"cb838e2b-f206-4da9-97bc-a160de56256f\" (UID: \"cb838e2b-f206-4da9-97bc-a160de56256f\") " Oct 31 13:37:43.575680 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:43.575623 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb838e2b-f206-4da9-97bc-a160de56256f-home\" (UniqueName: \"kubernetes.io/empty-dir/cb838e2b-f206-4da9-97bc-a160de56256f-home\") pod \"cb838e2b-f206-4da9-97bc-a160de56256f\" (UID: \"cb838e2b-f206-4da9-97bc-a160de56256f\") " Oct 31 13:37:43.575817 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:43.575798 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb838e2b-f206-4da9-97bc-a160de56256f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb838e2b-f206-4da9-97bc-a160de56256f-tmp-dir\") pod \"cb838e2b-f206-4da9-97bc-a160de56256f\" (UID: \"cb838e2b-f206-4da9-97bc-a160de56256f\") " Oct 31 13:37:43.575916 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:43.575883 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb838e2b-f206-4da9-97bc-a160de56256f-home" pod "cb838e2b-f206-4da9-97bc-a160de56256f" (UID: "cb838e2b-f206-4da9-97bc-a160de56256f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:43.576245 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:43.576058 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb838e2b-f206-4da9-97bc-a160de56256f-tmp-dir" pod "cb838e2b-f206-4da9-97bc-a160de56256f" (UID: "cb838e2b-f206-4da9-97bc-a160de56256f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:43.576969 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:43.576944 5693 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb838e2b-f206-4da9-97bc-a160de56256f-home\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:43.577070 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:43.576976 5693 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb838e2b-f206-4da9-97bc-a160de56256f-tmp-dir\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:43.577070 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:43.576991 5693 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb838e2b-f206-4da9-97bc-a160de56256f-model-cache\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:43.578056 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:43.578027 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb838e2b-f206-4da9-97bc-a160de56256f-dshm" pod "cb838e2b-f206-4da9-97bc-a160de56256f" (UID: "cb838e2b-f206-4da9-97bc-a160de56256f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:43.578147 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:43.578030 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb838e2b-f206-4da9-97bc-a160de56256f-tls-certs" pod "cb838e2b-f206-4da9-97bc-a160de56256f" (UID: "cb838e2b-f206-4da9-97bc-a160de56256f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:37:43.578147 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:43.578095 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb838e2b-f206-4da9-97bc-a160de56256f-kube-api-access-mtlcq" pod "cb838e2b-f206-4da9-97bc-a160de56256f" (UID: "cb838e2b-f206-4da9-97bc-a160de56256f"). InnerVolumeSpecName "kube-api-access-mtlcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:43.667019 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:43.666955 5693 generic.go:356] "Generic (PLEG): container finished" podID="cb838e2b-f206-4da9-97bc-a160de56256f" containerID="c88488c7133599370b30b2710f1396014636d853148ce28b02bb3fe5e5c335a1" exitCode=0 Oct 31 13:37:43.667227 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:43.667082 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-09db67/test-model-09db67-kserve-7478f9db55-gz5h5" Oct 31 13:37:43.667227 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:43.667077 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-09db67/test-model-09db67-kserve-7478f9db55-gz5h5" event={"ID":"cb838e2b-f206-4da9-97bc-a160de56256f","Type":"ContainerDied","Data":"c88488c7133599370b30b2710f1396014636d853148ce28b02bb3fe5e5c335a1"} Oct 31 13:37:43.667227 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:43.667140 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-09db67/test-model-09db67-kserve-7478f9db55-gz5h5" event={"ID":"cb838e2b-f206-4da9-97bc-a160de56256f","Type":"ContainerDied","Data":"f18cf750031d1b4d8ad0f5451849ffdd1eb54a80607916c9f812331c8c8e3de2"} Oct 31 13:37:43.667227 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:43.667165 5693 scope.go:122] "RemoveContainer" containerID="c88488c7133599370b30b2710f1396014636d853148ce28b02bb3fe5e5c335a1" Oct 31 13:37:43.678615 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:43.678583 5693 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb838e2b-f206-4da9-97bc-a160de56256f-tls-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:43.678848 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:43.678620 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtlcq\" (UniqueName: \"kubernetes.io/projected/cb838e2b-f206-4da9-97bc-a160de56256f-kube-api-access-mtlcq\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:43.678848 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:43.678654 5693 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb838e2b-f206-4da9-97bc-a160de56256f-dshm\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:43.687740 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:43.687720 5693 scope.go:122] "RemoveContainer" containerID="c88488c7133599370b30b2710f1396014636d853148ce28b02bb3fe5e5c335a1" Oct 31 13:37:43.688109 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:37:43.688087 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c88488c7133599370b30b2710f1396014636d853148ce28b02bb3fe5e5c335a1\": container with ID starting with c88488c7133599370b30b2710f1396014636d853148ce28b02bb3fe5e5c335a1 not found: ID does not exist" containerID="c88488c7133599370b30b2710f1396014636d853148ce28b02bb3fe5e5c335a1" Oct 31 13:37:43.688182 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:43.688123 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c88488c7133599370b30b2710f1396014636d853148ce28b02bb3fe5e5c335a1"} err="failed to get container status \"c88488c7133599370b30b2710f1396014636d853148ce28b02bb3fe5e5c335a1\": rpc error: code = NotFound desc = could not find container \"c88488c7133599370b30b2710f1396014636d853148ce28b02bb3fe5e5c335a1\": container with ID starting with c88488c7133599370b30b2710f1396014636d853148ce28b02bb3fe5e5c335a1 not found: ID does not exist" Oct 31 13:37:43.715473 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:43.715438 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-09db67/test-model-09db67-kserve-7478f9db55-gz5h5"] Oct 31 13:37:43.726594 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:43.726567 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-09db67/test-model-09db67-kserve-7478f9db55-gz5h5"] Oct 31 13:37:45.417657 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:45.417619 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb838e2b-f206-4da9-97bc-a160de56256f" path="/var/lib/kubelet/pods/cb838e2b-f206-4da9-97bc-a160de56256f/volumes" Oct 31 13:37:48.264043 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:48.264000 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs"] Oct 31 13:37:48.264715 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:48.264667 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" podUID="d137459e-9920-4c27-a910-0216cc6a7168" containerName="istio-proxy" containerID="cri-o://aa52134bd572d283457eb255e08d0b6a91ba3b48efbc985ba7cd62bcb3e186cb" gracePeriod=30 Oct 31 13:37:48.692368 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:37:48.692333 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:37:49.363346 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:49.363311 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-09db67-568998f649-d2ntj"] Oct 31 13:37:53.586498 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:53.586473 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" Oct 31 13:37:53.672006 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:53.671965 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d137459e-9920-4c27-a910-0216cc6a7168-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d137459e-9920-4c27-a910-0216cc6a7168-workload-socket\") pod \"d137459e-9920-4c27-a910-0216cc6a7168\" (UID: \"d137459e-9920-4c27-a910-0216cc6a7168\") " Oct 31 13:37:53.672220 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:53.672029 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d137459e-9920-4c27-a910-0216cc6a7168-istio-token\" (UniqueName: \"kubernetes.io/projected/d137459e-9920-4c27-a910-0216cc6a7168-istio-token\") pod \"d137459e-9920-4c27-a910-0216cc6a7168\" (UID: \"d137459e-9920-4c27-a910-0216cc6a7168\") " Oct 31 13:37:53.672298 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:53.672228 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d137459e-9920-4c27-a910-0216cc6a7168-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d137459e-9920-4c27-a910-0216cc6a7168-istio-envoy\") pod \"d137459e-9920-4c27-a910-0216cc6a7168\" (UID: \"d137459e-9920-4c27-a910-0216cc6a7168\") " Oct 31 13:37:53.672298 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:53.672272 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d137459e-9920-4c27-a910-0216cc6a7168-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d137459e-9920-4c27-a910-0216cc6a7168-istiod-ca-cert\") pod \"d137459e-9920-4c27-a910-0216cc6a7168\" (UID: \"d137459e-9920-4c27-a910-0216cc6a7168\") " Oct 31 13:37:53.672416 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:53.672307 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d137459e-9920-4c27-a910-0216cc6a7168-workload-socket" pod "d137459e-9920-4c27-a910-0216cc6a7168" (UID: "d137459e-9920-4c27-a910-0216cc6a7168"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:53.672416 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:53.672331 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d137459e-9920-4c27-a910-0216cc6a7168-kube-api-access-f4mr9\" (UniqueName: \"kubernetes.io/projected/d137459e-9920-4c27-a910-0216cc6a7168-kube-api-access-f4mr9\") pod \"d137459e-9920-4c27-a910-0216cc6a7168\" (UID: \"d137459e-9920-4c27-a910-0216cc6a7168\") " Oct 31 13:37:53.672416 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:53.672373 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d137459e-9920-4c27-a910-0216cc6a7168-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d137459e-9920-4c27-a910-0216cc6a7168-istio-data\") pod \"d137459e-9920-4c27-a910-0216cc6a7168\" (UID: \"d137459e-9920-4c27-a910-0216cc6a7168\") " Oct 31 13:37:53.672598 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:53.672446 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d137459e-9920-4c27-a910-0216cc6a7168-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d137459e-9920-4c27-a910-0216cc6a7168-workload-certs\") pod \"d137459e-9920-4c27-a910-0216cc6a7168\" (UID: \"d137459e-9920-4c27-a910-0216cc6a7168\") " Oct 31 13:37:53.672598 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:53.672560 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d137459e-9920-4c27-a910-0216cc6a7168-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d137459e-9920-4c27-a910-0216cc6a7168-istio-podinfo\") pod \"d137459e-9920-4c27-a910-0216cc6a7168\" (UID: \"d137459e-9920-4c27-a910-0216cc6a7168\") " Oct 31 13:37:53.672598 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:53.672580 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d137459e-9920-4c27-a910-0216cc6a7168-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d137459e-9920-4c27-a910-0216cc6a7168-credential-socket\") pod \"d137459e-9920-4c27-a910-0216cc6a7168\" (UID: \"d137459e-9920-4c27-a910-0216cc6a7168\") " Oct 31 13:37:53.673243 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:53.673182 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d137459e-9920-4c27-a910-0216cc6a7168-istiod-ca-cert" pod "d137459e-9920-4c27-a910-0216cc6a7168" (UID: "d137459e-9920-4c27-a910-0216cc6a7168"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:37:53.673453 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:53.673402 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d137459e-9920-4c27-a910-0216cc6a7168-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:53.673453 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:53.673413 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d137459e-9920-4c27-a910-0216cc6a7168-workload-certs" pod "d137459e-9920-4c27-a910-0216cc6a7168" (UID: "d137459e-9920-4c27-a910-0216cc6a7168"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:53.673453 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:53.673425 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d137459e-9920-4c27-a910-0216cc6a7168-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:53.673453 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:53.673427 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d137459e-9920-4c27-a910-0216cc6a7168-istio-data" pod "d137459e-9920-4c27-a910-0216cc6a7168" (UID: "d137459e-9920-4c27-a910-0216cc6a7168"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:53.673723 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:53.673665 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d137459e-9920-4c27-a910-0216cc6a7168-credential-socket" pod "d137459e-9920-4c27-a910-0216cc6a7168" (UID: "d137459e-9920-4c27-a910-0216cc6a7168"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:53.674873 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:53.674843 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d137459e-9920-4c27-a910-0216cc6a7168-istio-envoy" pod "d137459e-9920-4c27-a910-0216cc6a7168" (UID: "d137459e-9920-4c27-a910-0216cc6a7168"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:53.675072 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:53.675047 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d137459e-9920-4c27-a910-0216cc6a7168-istio-token" pod "d137459e-9920-4c27-a910-0216cc6a7168" (UID: "d137459e-9920-4c27-a910-0216cc6a7168"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:53.675128 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:53.675091 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d137459e-9920-4c27-a910-0216cc6a7168-kube-api-access-f4mr9" pod "d137459e-9920-4c27-a910-0216cc6a7168" (UID: "d137459e-9920-4c27-a910-0216cc6a7168"). InnerVolumeSpecName "kube-api-access-f4mr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:53.676150 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:53.676131 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d137459e-9920-4c27-a910-0216cc6a7168-istio-podinfo" pod "d137459e-9920-4c27-a910-0216cc6a7168" (UID: "d137459e-9920-4c27-a910-0216cc6a7168"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:37:53.761857 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:53.761818 5693 generic.go:356] "Generic (PLEG): container finished" podID="d137459e-9920-4c27-a910-0216cc6a7168" containerID="aa52134bd572d283457eb255e08d0b6a91ba3b48efbc985ba7cd62bcb3e186cb" exitCode=0 Oct 31 13:37:53.762166 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:53.761991 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" event={"ID":"d137459e-9920-4c27-a910-0216cc6a7168","Type":"ContainerDied","Data":"aa52134bd572d283457eb255e08d0b6a91ba3b48efbc985ba7cd62bcb3e186cb"} Oct 31 13:37:53.762166 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:53.762080 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" event={"ID":"d137459e-9920-4c27-a910-0216cc6a7168","Type":"ContainerDied","Data":"7ef47b380dd6dfbcc6fb95269446aee029b284cef76b0e8f838637141b5ad432"} Oct 31 13:37:53.762166 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:53.762119 5693 scope.go:122] "RemoveContainer" containerID="aa52134bd572d283457eb255e08d0b6a91ba3b48efbc985ba7cd62bcb3e186cb" Oct 31 13:37:53.762166 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:53.762122 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs" Oct 31 13:37:53.776512 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:53.776482 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d137459e-9920-4c27-a910-0216cc6a7168-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:53.776623 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:53.776515 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d137459e-9920-4c27-a910-0216cc6a7168-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:53.776623 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:53.776535 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d137459e-9920-4c27-a910-0216cc6a7168-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:53.776623 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:53.776549 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d137459e-9920-4c27-a910-0216cc6a7168-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:53.776623 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:53.776562 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4mr9\" (UniqueName: \"kubernetes.io/projected/d137459e-9920-4c27-a910-0216cc6a7168-kube-api-access-f4mr9\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:53.776623 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:53.776576 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d137459e-9920-4c27-a910-0216cc6a7168-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:53.776623 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:53.776592 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d137459e-9920-4c27-a910-0216cc6a7168-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:53.780175 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:53.780155 5693 scope.go:122] "RemoveContainer" containerID="8eac644d96d259e42257cee93694e01758a3f6c3885083b74871f9f8ea701d42" Oct 31 13:37:53.791197 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:53.791170 5693 scope.go:122] "RemoveContainer" containerID="aa52134bd572d283457eb255e08d0b6a91ba3b48efbc985ba7cd62bcb3e186cb" Oct 31 13:37:53.791518 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:37:53.791495 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa52134bd572d283457eb255e08d0b6a91ba3b48efbc985ba7cd62bcb3e186cb\": container with ID starting with aa52134bd572d283457eb255e08d0b6a91ba3b48efbc985ba7cd62bcb3e186cb not found: ID does not exist" containerID="aa52134bd572d283457eb255e08d0b6a91ba3b48efbc985ba7cd62bcb3e186cb" Oct 31 13:37:53.791613 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:53.791526 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa52134bd572d283457eb255e08d0b6a91ba3b48efbc985ba7cd62bcb3e186cb"} err="failed to get container status \"aa52134bd572d283457eb255e08d0b6a91ba3b48efbc985ba7cd62bcb3e186cb\": rpc error: code = NotFound desc = could not find container \"aa52134bd572d283457eb255e08d0b6a91ba3b48efbc985ba7cd62bcb3e186cb\": container with ID starting with aa52134bd572d283457eb255e08d0b6a91ba3b48efbc985ba7cd62bcb3e186cb not found: ID does not exist" Oct 31 13:37:53.791613 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:53.791545 5693 scope.go:122] "RemoveContainer" containerID="8eac644d96d259e42257cee93694e01758a3f6c3885083b74871f9f8ea701d42" Oct 31 13:37:53.791861 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:37:53.791841 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eac644d96d259e42257cee93694e01758a3f6c3885083b74871f9f8ea701d42\": container with ID starting with 8eac644d96d259e42257cee93694e01758a3f6c3885083b74871f9f8ea701d42 not found: ID does not exist" containerID="8eac644d96d259e42257cee93694e01758a3f6c3885083b74871f9f8ea701d42" Oct 31 13:37:53.791927 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:53.791871 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eac644d96d259e42257cee93694e01758a3f6c3885083b74871f9f8ea701d42"} err="failed to get container status \"8eac644d96d259e42257cee93694e01758a3f6c3885083b74871f9f8ea701d42\": rpc error: code = NotFound desc = could not find container \"8eac644d96d259e42257cee93694e01758a3f6c3885083b74871f9f8ea701d42\": container with ID starting with 8eac644d96d259e42257cee93694e01758a3f6c3885083b74871f9f8ea701d42 not found: ID does not exist" Oct 31 13:37:53.812188 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:53.812127 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs"] Oct 31 13:37:53.826420 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:53.826388 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-3cddb3-openshift-default-777f85c855-8l6fs"] Oct 31 13:37:54.364034 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:54.363996 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-09db67-568998f649-d2ntj" podUID="05078422-1fc4-4bf1-ab82-673c654c7b0a" containerName="payload-processing" containerID="cri-o://6a4017a99987b2be0fd7820ee3309640494b0062d34b6a41034473d1392d4040" gracePeriod=25 Oct 31 13:37:55.410531 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:55.410504 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d137459e-9920-4c27-a910-0216cc6a7168" path="/var/lib/kubelet/pods/d137459e-9920-4c27-a910-0216cc6a7168/volumes" Oct 31 13:37:56.325809 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:37:56.325774 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-09db67-568998f649-d2ntj" podUID="05078422-1fc4-4bf1-ab82-673c654c7b0a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:38:06.326650 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:06.326601 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-09db67-568998f649-d2ntj" podUID="05078422-1fc4-4bf1-ab82-673c654c7b0a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:38:08.839130 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:08.839097 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-09db67-588458cc74-tg4cx"] Oct 31 13:38:08.839582 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:08.839322 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09db67-588458cc74-tg4cx" podUID="77ca7503-e170-4c12-af28-8e78cce800a5" containerName="maas-api" containerID="cri-o://74e8a5b1d3fcbceb69439d79e13a367f3628f6d1db5441a64900ddf2113b464e" gracePeriod=30 Oct 31 13:38:09.242443 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:09.242374 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09db67-588458cc74-tg4cx" Oct 31 13:38:09.346174 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:09.346139 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77ca7503-e170-4c12-af28-8e78cce800a5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77ca7503-e170-4c12-af28-8e78cce800a5-openshift-service-ca\") pod \"77ca7503-e170-4c12-af28-8e78cce800a5\" (UID: \"77ca7503-e170-4c12-af28-8e78cce800a5\") " Oct 31 13:38:09.346391 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:09.346195 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77ca7503-e170-4c12-af28-8e78cce800a5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/77ca7503-e170-4c12-af28-8e78cce800a5-metrics-certs\") pod \"77ca7503-e170-4c12-af28-8e78cce800a5\" (UID: \"77ca7503-e170-4c12-af28-8e78cce800a5\") " Oct 31 13:38:09.346472 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:09.346425 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77ca7503-e170-4c12-af28-8e78cce800a5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77ca7503-e170-4c12-af28-8e78cce800a5-maas-api-tls\") pod \"77ca7503-e170-4c12-af28-8e78cce800a5\" (UID: \"77ca7503-e170-4c12-af28-8e78cce800a5\") " Oct 31 13:38:09.346532 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:09.346469 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77ca7503-e170-4c12-af28-8e78cce800a5-kube-api-access-rndvj\" (UniqueName: \"kubernetes.io/projected/77ca7503-e170-4c12-af28-8e78cce800a5-kube-api-access-rndvj\") pod \"77ca7503-e170-4c12-af28-8e78cce800a5\" (UID: \"77ca7503-e170-4c12-af28-8e78cce800a5\") " Oct 31 13:38:09.346599 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:09.346560 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77ca7503-e170-4c12-af28-8e78cce800a5-openshift-service-ca" pod "77ca7503-e170-4c12-af28-8e78cce800a5" (UID: "77ca7503-e170-4c12-af28-8e78cce800a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:38:09.347211 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:09.347182 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77ca7503-e170-4c12-af28-8e78cce800a5-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:09.349347 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:09.349324 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77ca7503-e170-4c12-af28-8e78cce800a5-metrics-certs" pod "77ca7503-e170-4c12-af28-8e78cce800a5" (UID: "77ca7503-e170-4c12-af28-8e78cce800a5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:38:09.350044 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:09.350020 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77ca7503-e170-4c12-af28-8e78cce800a5-kube-api-access-rndvj" pod "77ca7503-e170-4c12-af28-8e78cce800a5" (UID: "77ca7503-e170-4c12-af28-8e78cce800a5"). InnerVolumeSpecName "kube-api-access-rndvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:38:09.350168 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:09.350058 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77ca7503-e170-4c12-af28-8e78cce800a5-maas-api-tls" pod "77ca7503-e170-4c12-af28-8e78cce800a5" (UID: "77ca7503-e170-4c12-af28-8e78cce800a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:38:09.455440 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:09.452033 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77ca7503-e170-4c12-af28-8e78cce800a5-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:09.455440 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:09.452072 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rndvj\" (UniqueName: \"kubernetes.io/projected/77ca7503-e170-4c12-af28-8e78cce800a5-kube-api-access-rndvj\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:09.455440 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:09.452088 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77ca7503-e170-4c12-af28-8e78cce800a5-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:09.898244 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:09.898210 5693 generic.go:356] "Generic (PLEG): container finished" podID="77ca7503-e170-4c12-af28-8e78cce800a5" containerID="74e8a5b1d3fcbceb69439d79e13a367f3628f6d1db5441a64900ddf2113b464e" exitCode=0 Oct 31 13:38:09.899336 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:09.899010 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09db67-588458cc74-tg4cx" Oct 31 13:38:09.899336 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:09.899070 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09db67-588458cc74-tg4cx" event={"ID":"77ca7503-e170-4c12-af28-8e78cce800a5","Type":"ContainerDied","Data":"74e8a5b1d3fcbceb69439d79e13a367f3628f6d1db5441a64900ddf2113b464e"} Oct 31 13:38:09.899336 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:09.899144 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09db67-588458cc74-tg4cx" event={"ID":"77ca7503-e170-4c12-af28-8e78cce800a5","Type":"ContainerDied","Data":"6646ea7c103d44f3564aca097817f2fe035bf251b31f92cab253607b3f65ed11"} Oct 31 13:38:09.899336 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:09.899173 5693 scope.go:122] "RemoveContainer" containerID="74e8a5b1d3fcbceb69439d79e13a367f3628f6d1db5441a64900ddf2113b464e" Oct 31 13:38:09.909824 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:09.909778 5693 scope.go:122] "RemoveContainer" containerID="74e8a5b1d3fcbceb69439d79e13a367f3628f6d1db5441a64900ddf2113b464e" Oct 31 13:38:09.910125 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:38:09.910097 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74e8a5b1d3fcbceb69439d79e13a367f3628f6d1db5441a64900ddf2113b464e\": container with ID starting with 74e8a5b1d3fcbceb69439d79e13a367f3628f6d1db5441a64900ddf2113b464e not found: ID does not exist" containerID="74e8a5b1d3fcbceb69439d79e13a367f3628f6d1db5441a64900ddf2113b464e" Oct 31 13:38:09.910209 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:09.910136 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74e8a5b1d3fcbceb69439d79e13a367f3628f6d1db5441a64900ddf2113b464e"} err="failed to get container status \"74e8a5b1d3fcbceb69439d79e13a367f3628f6d1db5441a64900ddf2113b464e\": rpc error: code = NotFound desc = could not find container \"74e8a5b1d3fcbceb69439d79e13a367f3628f6d1db5441a64900ddf2113b464e\": container with ID starting with 74e8a5b1d3fcbceb69439d79e13a367f3628f6d1db5441a64900ddf2113b464e not found: ID does not exist" Oct 31 13:38:09.936209 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:09.936177 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-09db67-588458cc74-tg4cx"] Oct 31 13:38:09.939831 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:09.939805 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-09db67-588458cc74-tg4cx"] Oct 31 13:38:11.429664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:11.427270 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77ca7503-e170-4c12-af28-8e78cce800a5" path="/var/lib/kubelet/pods/77ca7503-e170-4c12-af28-8e78cce800a5/volumes" Oct 31 13:38:14.654763 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:14.654738 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-09db67-568998f649-d2ntj" Oct 31 13:38:14.726260 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:14.726227 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05078422-1fc4-4bf1-ab82-673c654c7b0a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05078422-1fc4-4bf1-ab82-673c654c7b0a-plugins-config-volume\") pod \"05078422-1fc4-4bf1-ab82-673c654c7b0a\" (UID: \"05078422-1fc4-4bf1-ab82-673c654c7b0a\") " Oct 31 13:38:14.726476 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:14.726292 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05078422-1fc4-4bf1-ab82-673c654c7b0a-kube-api-access-5kjbp\" (UniqueName: \"kubernetes.io/projected/05078422-1fc4-4bf1-ab82-673c654c7b0a-kube-api-access-5kjbp\") pod \"05078422-1fc4-4bf1-ab82-673c654c7b0a\" (UID: \"05078422-1fc4-4bf1-ab82-673c654c7b0a\") " Oct 31 13:38:14.726777 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:14.726751 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05078422-1fc4-4bf1-ab82-673c654c7b0a-plugins-config-volume" pod "05078422-1fc4-4bf1-ab82-673c654c7b0a" (UID: "05078422-1fc4-4bf1-ab82-673c654c7b0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:38:14.727439 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:14.727415 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05078422-1fc4-4bf1-ab82-673c654c7b0a-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:14.730115 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:14.730042 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05078422-1fc4-4bf1-ab82-673c654c7b0a-kube-api-access-5kjbp" pod "05078422-1fc4-4bf1-ab82-673c654c7b0a" (UID: "05078422-1fc4-4bf1-ab82-673c654c7b0a"). InnerVolumeSpecName "kube-api-access-5kjbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:38:14.828577 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:14.828546 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kjbp\" (UniqueName: \"kubernetes.io/projected/05078422-1fc4-4bf1-ab82-673c654c7b0a-kube-api-access-5kjbp\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:14.940596 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:14.940564 5693 generic.go:356] "Generic (PLEG): container finished" podID="05078422-1fc4-4bf1-ab82-673c654c7b0a" containerID="6a4017a99987b2be0fd7820ee3309640494b0062d34b6a41034473d1392d4040" exitCode=0 Oct 31 13:38:14.940805 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:14.940608 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-09db67-568998f649-d2ntj" Oct 31 13:38:14.940805 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:14.940684 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-09db67-568998f649-d2ntj" event={"ID":"05078422-1fc4-4bf1-ab82-673c654c7b0a","Type":"ContainerDied","Data":"6a4017a99987b2be0fd7820ee3309640494b0062d34b6a41034473d1392d4040"} Oct 31 13:38:14.940805 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:14.940735 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-09db67-568998f649-d2ntj" event={"ID":"05078422-1fc4-4bf1-ab82-673c654c7b0a","Type":"ContainerDied","Data":"c770e29b68e4d829a2f99fda507fcff6787fc08d6fa11861a3d6fff5fd5ab621"} Oct 31 13:38:14.940805 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:14.940759 5693 scope.go:122] "RemoveContainer" containerID="6a4017a99987b2be0fd7820ee3309640494b0062d34b6a41034473d1392d4040" Oct 31 13:38:14.952377 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:14.952359 5693 scope.go:122] "RemoveContainer" containerID="6a4017a99987b2be0fd7820ee3309640494b0062d34b6a41034473d1392d4040" Oct 31 13:38:14.952623 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:38:14.952599 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a4017a99987b2be0fd7820ee3309640494b0062d34b6a41034473d1392d4040\": container with ID starting with 6a4017a99987b2be0fd7820ee3309640494b0062d34b6a41034473d1392d4040 not found: ID does not exist" containerID="6a4017a99987b2be0fd7820ee3309640494b0062d34b6a41034473d1392d4040" Oct 31 13:38:14.952722 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:14.952630 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a4017a99987b2be0fd7820ee3309640494b0062d34b6a41034473d1392d4040"} err="failed to get container status \"6a4017a99987b2be0fd7820ee3309640494b0062d34b6a41034473d1392d4040\": rpc error: code = NotFound desc = could not find container \"6a4017a99987b2be0fd7820ee3309640494b0062d34b6a41034473d1392d4040\": container with ID starting with 6a4017a99987b2be0fd7820ee3309640494b0062d34b6a41034473d1392d4040 not found: ID does not exist" Oct 31 13:38:14.965143 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:14.965119 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-09db67-568998f649-d2ntj"] Oct 31 13:38:14.975723 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:14.975696 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-09db67-568998f649-d2ntj"] Oct 31 13:38:15.411504 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:15.411470 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05078422-1fc4-4bf1-ab82-673c654c7b0a" path="/var/lib/kubelet/pods/05078422-1fc4-4bf1-ab82-673c654c7b0a/volumes" Oct 31 13:38:18.691812 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:38:18.691766 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:38:18.876686 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.876653 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-1e72f6/rate-test-model-1e72f6-kserve-8454cfd857-tbqhz"] Oct 31 13:38:18.877808 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.877781 5693 cpu_manager.go:414] "containerMap: removing container" podUID="d92607b7-b885-43e2-9694-7bcb94d4f6b5" containerName="payload-processing" Oct 31 13:38:18.877808 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.877807 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d92607b7-b885-43e2-9694-7bcb94d4f6b5" containerName="payload-processing" Oct 31 13:38:18.878027 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.877829 5693 cpu_manager.go:414] "containerMap: removing container" podUID="cb838e2b-f206-4da9-97bc-a160de56256f" containerName="main" Oct 31 13:38:18.878027 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.877837 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb838e2b-f206-4da9-97bc-a160de56256f" containerName="main" Oct 31 13:38:18.878027 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.877855 5693 cpu_manager.go:414] "containerMap: removing container" podUID="b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab" containerName="istio-proxy" Oct 31 13:38:18.878027 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.877863 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab" containerName="istio-proxy" Oct 31 13:38:18.878027 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.877872 5693 cpu_manager.go:414] "containerMap: removing container" podUID="efccc572-e9f9-4f4a-ab45-c7458b0e2e58" containerName="istio-proxy" Oct 31 13:38:18.878027 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.877880 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efccc572-e9f9-4f4a-ab45-c7458b0e2e58" containerName="istio-proxy" Oct 31 13:38:18.878027 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.877889 5693 cpu_manager.go:414] "containerMap: removing container" podUID="77ca7503-e170-4c12-af28-8e78cce800a5" containerName="maas-api" Oct 31 13:38:18.878027 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.877896 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77ca7503-e170-4c12-af28-8e78cce800a5" containerName="maas-api" Oct 31 13:38:18.878027 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.877928 5693 cpu_manager.go:414] "containerMap: removing container" podUID="d137459e-9920-4c27-a910-0216cc6a7168" containerName="istio-proxy" Oct 31 13:38:18.878027 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.877934 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d137459e-9920-4c27-a910-0216cc6a7168" containerName="istio-proxy" Oct 31 13:38:18.878027 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.877950 5693 cpu_manager.go:414] "containerMap: removing container" podUID="30cc0f1f-cfc3-475d-8d48-a98f210b91f6" containerName="payload-processing" Oct 31 13:38:18.878027 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.877958 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30cc0f1f-cfc3-475d-8d48-a98f210b91f6" containerName="payload-processing" Oct 31 13:38:18.878027 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.877981 5693 cpu_manager.go:414] "containerMap: removing container" podUID="497f3ecd-3d10-4ffd-ae25-490c570c4b1f" containerName="payload-pre-processing" Oct 31 13:38:18.878027 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.877988 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="497f3ecd-3d10-4ffd-ae25-490c570c4b1f" containerName="payload-pre-processing" Oct 31 13:38:18.878027 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.878007 5693 cpu_manager.go:414] "containerMap: removing container" podUID="05078422-1fc4-4bf1-ab82-673c654c7b0a" containerName="payload-processing" Oct 31 13:38:18.878027 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.878014 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05078422-1fc4-4bf1-ab82-673c654c7b0a" containerName="payload-processing" Oct 31 13:38:18.878027 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.878032 5693 cpu_manager.go:414] "containerMap: removing container" podUID="0e759d3e-611f-43e3-b520-2a5d34517d91" containerName="payload-pre-processing" Oct 31 13:38:18.878986 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.878039 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e759d3e-611f-43e3-b520-2a5d34517d91" containerName="payload-pre-processing" Oct 31 13:38:18.878986 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.878054 5693 cpu_manager.go:414] "containerMap: removing container" podUID="f368dfe0-177d-4e5f-9d68-f0f440717fe2" containerName="istio-proxy" Oct 31 13:38:18.878986 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.878060 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f368dfe0-177d-4e5f-9d68-f0f440717fe2" containerName="istio-proxy" Oct 31 13:38:18.878986 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.878245 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e759d3e-611f-43e3-b520-2a5d34517d91" containerName="payload-pre-processing" Oct 31 13:38:18.878986 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.878259 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="f368dfe0-177d-4e5f-9d68-f0f440717fe2" containerName="istio-proxy" Oct 31 13:38:18.878986 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.878271 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="efccc572-e9f9-4f4a-ab45-c7458b0e2e58" containerName="istio-proxy" Oct 31 13:38:18.878986 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.878282 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="d92607b7-b885-43e2-9694-7bcb94d4f6b5" containerName="payload-processing" Oct 31 13:38:18.878986 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.878293 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb838e2b-f206-4da9-97bc-a160de56256f" containerName="main" Oct 31 13:38:18.878986 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.878301 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2f6ff04-3b39-4e0e-bb85-f41e4aac55ab" containerName="istio-proxy" Oct 31 13:38:18.878986 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.878311 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="efccc572-e9f9-4f4a-ab45-c7458b0e2e58" containerName="istio-proxy" Oct 31 13:38:18.878986 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.878321 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="77ca7503-e170-4c12-af28-8e78cce800a5" containerName="maas-api" Oct 31 13:38:18.878986 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.878334 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="d137459e-9920-4c27-a910-0216cc6a7168" containerName="istio-proxy" Oct 31 13:38:18.878986 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.878343 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="30cc0f1f-cfc3-475d-8d48-a98f210b91f6" containerName="payload-processing" Oct 31 13:38:18.878986 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.878354 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="497f3ecd-3d10-4ffd-ae25-490c570c4b1f" containerName="payload-pre-processing" Oct 31 13:38:18.878986 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.878365 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="05078422-1fc4-4bf1-ab82-673c654c7b0a" containerName="payload-processing" Oct 31 13:38:18.878986 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.878373 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="d137459e-9920-4c27-a910-0216cc6a7168" containerName="istio-proxy" Oct 31 13:38:18.884389 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.884365 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-1e72f6/rate-test-model-1e72f6-kserve-8454cfd857-tbqhz" Oct 31 13:38:18.896931 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.896886 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-1e72f6\"/\"openshift-service-ca.crt\"" Oct 31 13:38:18.897131 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.897009 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-1e72f6\"/\"kube-root-ca.crt\"" Oct 31 13:38:18.897131 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.897116 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-1e72f6\"/\"rate-test-model-1e72f6-kserve-self-signed-certs\"" Oct 31 13:38:18.897799 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.897776 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-1e72f6\"/\"default-dockercfg-w85bx\"" Oct 31 13:38:18.900539 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:18.900520 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-1e72f6/rate-test-model-1e72f6-kserve-8454cfd857-tbqhz"] Oct 31 13:38:19.022418 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:19.022313 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-model-cache\") pod \"rate-test-model-1e72f6-kserve-8454cfd857-tbqhz\" (UID: \"3cc7b994-8cee-47e2-9a6d-b587e3ac1eef\") " pod="ai-tenant-e2e-shared-b-w4-1e72f6/rate-test-model-1e72f6-kserve-8454cfd857-tbqhz" Oct 31 13:38:19.022418 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:19.022372 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-home\") pod \"rate-test-model-1e72f6-kserve-8454cfd857-tbqhz\" (UID: \"3cc7b994-8cee-47e2-9a6d-b587e3ac1eef\") " pod="ai-tenant-e2e-shared-b-w4-1e72f6/rate-test-model-1e72f6-kserve-8454cfd857-tbqhz" Oct 31 13:38:19.022627 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:19.022546 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-tmp-dir\") pod \"rate-test-model-1e72f6-kserve-8454cfd857-tbqhz\" (UID: \"3cc7b994-8cee-47e2-9a6d-b587e3ac1eef\") " pod="ai-tenant-e2e-shared-b-w4-1e72f6/rate-test-model-1e72f6-kserve-8454cfd857-tbqhz" Oct 31 13:38:19.022627 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:19.022595 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpj7l\" (UniqueName: \"kubernetes.io/projected/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-kube-api-access-mpj7l\") pod \"rate-test-model-1e72f6-kserve-8454cfd857-tbqhz\" (UID: \"3cc7b994-8cee-47e2-9a6d-b587e3ac1eef\") " pod="ai-tenant-e2e-shared-b-w4-1e72f6/rate-test-model-1e72f6-kserve-8454cfd857-tbqhz" Oct 31 13:38:19.022770 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:19.022748 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-tls-certs\") pod \"rate-test-model-1e72f6-kserve-8454cfd857-tbqhz\" (UID: \"3cc7b994-8cee-47e2-9a6d-b587e3ac1eef\") " pod="ai-tenant-e2e-shared-b-w4-1e72f6/rate-test-model-1e72f6-kserve-8454cfd857-tbqhz" Oct 31 13:38:19.022836 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:19.022822 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-dshm\") pod \"rate-test-model-1e72f6-kserve-8454cfd857-tbqhz\" (UID: \"3cc7b994-8cee-47e2-9a6d-b587e3ac1eef\") " pod="ai-tenant-e2e-shared-b-w4-1e72f6/rate-test-model-1e72f6-kserve-8454cfd857-tbqhz" Oct 31 13:38:19.125427 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:19.125382 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-dshm\") pod \"rate-test-model-1e72f6-kserve-8454cfd857-tbqhz\" (UID: \"3cc7b994-8cee-47e2-9a6d-b587e3ac1eef\") " pod="ai-tenant-e2e-shared-b-w4-1e72f6/rate-test-model-1e72f6-kserve-8454cfd857-tbqhz" Oct 31 13:38:19.125756 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:19.125563 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-model-cache\") pod \"rate-test-model-1e72f6-kserve-8454cfd857-tbqhz\" (UID: \"3cc7b994-8cee-47e2-9a6d-b587e3ac1eef\") " pod="ai-tenant-e2e-shared-b-w4-1e72f6/rate-test-model-1e72f6-kserve-8454cfd857-tbqhz" Oct 31 13:38:19.125756 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:19.125606 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-home\") pod \"rate-test-model-1e72f6-kserve-8454cfd857-tbqhz\" (UID: \"3cc7b994-8cee-47e2-9a6d-b587e3ac1eef\") " pod="ai-tenant-e2e-shared-b-w4-1e72f6/rate-test-model-1e72f6-kserve-8454cfd857-tbqhz" Oct 31 13:38:19.125906 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:19.125824 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-tmp-dir\") pod \"rate-test-model-1e72f6-kserve-8454cfd857-tbqhz\" (UID: \"3cc7b994-8cee-47e2-9a6d-b587e3ac1eef\") " pod="ai-tenant-e2e-shared-b-w4-1e72f6/rate-test-model-1e72f6-kserve-8454cfd857-tbqhz" Oct 31 13:38:19.125906 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:19.125877 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpj7l\" (UniqueName: \"kubernetes.io/projected/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-kube-api-access-mpj7l\") pod \"rate-test-model-1e72f6-kserve-8454cfd857-tbqhz\" (UID: \"3cc7b994-8cee-47e2-9a6d-b587e3ac1eef\") " pod="ai-tenant-e2e-shared-b-w4-1e72f6/rate-test-model-1e72f6-kserve-8454cfd857-tbqhz" Oct 31 13:38:19.126078 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:19.125921 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-tls-certs\") pod \"rate-test-model-1e72f6-kserve-8454cfd857-tbqhz\" (UID: \"3cc7b994-8cee-47e2-9a6d-b587e3ac1eef\") " pod="ai-tenant-e2e-shared-b-w4-1e72f6/rate-test-model-1e72f6-kserve-8454cfd857-tbqhz" Oct 31 13:38:19.126315 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:19.126291 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-home\") pod \"rate-test-model-1e72f6-kserve-8454cfd857-tbqhz\" (UID: \"3cc7b994-8cee-47e2-9a6d-b587e3ac1eef\") " pod="ai-tenant-e2e-shared-b-w4-1e72f6/rate-test-model-1e72f6-kserve-8454cfd857-tbqhz" Oct 31 13:38:19.127008 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:19.126982 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-tmp-dir\") pod \"rate-test-model-1e72f6-kserve-8454cfd857-tbqhz\" (UID: \"3cc7b994-8cee-47e2-9a6d-b587e3ac1eef\") " pod="ai-tenant-e2e-shared-b-w4-1e72f6/rate-test-model-1e72f6-kserve-8454cfd857-tbqhz" Oct 31 13:38:19.127137 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:19.127068 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-model-cache\") pod \"rate-test-model-1e72f6-kserve-8454cfd857-tbqhz\" (UID: \"3cc7b994-8cee-47e2-9a6d-b587e3ac1eef\") " pod="ai-tenant-e2e-shared-b-w4-1e72f6/rate-test-model-1e72f6-kserve-8454cfd857-tbqhz" Oct 31 13:38:19.130908 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:19.130879 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-tls-certs\") pod \"rate-test-model-1e72f6-kserve-8454cfd857-tbqhz\" (UID: \"3cc7b994-8cee-47e2-9a6d-b587e3ac1eef\") " pod="ai-tenant-e2e-shared-b-w4-1e72f6/rate-test-model-1e72f6-kserve-8454cfd857-tbqhz" Oct 31 13:38:19.131246 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:19.131222 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-dshm\") pod \"rate-test-model-1e72f6-kserve-8454cfd857-tbqhz\" (UID: \"3cc7b994-8cee-47e2-9a6d-b587e3ac1eef\") " pod="ai-tenant-e2e-shared-b-w4-1e72f6/rate-test-model-1e72f6-kserve-8454cfd857-tbqhz" Oct 31 13:38:19.144778 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:19.144748 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpj7l\" (UniqueName: \"kubernetes.io/projected/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-kube-api-access-mpj7l\") pod \"rate-test-model-1e72f6-kserve-8454cfd857-tbqhz\" (UID: \"3cc7b994-8cee-47e2-9a6d-b587e3ac1eef\") " pod="ai-tenant-e2e-shared-b-w4-1e72f6/rate-test-model-1e72f6-kserve-8454cfd857-tbqhz" Oct 31 13:38:19.208234 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:19.208201 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-1e72f6/rate-test-model-1e72f6-kserve-8454cfd857-tbqhz" Oct 31 13:38:19.524905 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:19.524873 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-1e72f6/rate-test-model-1e72f6-kserve-8454cfd857-tbqhz"] Oct 31 13:38:19.532686 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:38:19.531859 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3cc7b994_8cee_47e2_9a6d_b587e3ac1eef.slice/crio-8c182f62fea885448bc80335ae18ba6949168dd1632c1d51387cb1f8e33c25b0 WatchSource:0}: Error finding container 8c182f62fea885448bc80335ae18ba6949168dd1632c1d51387cb1f8e33c25b0: Status 404 returned error can't find the container with id 8c182f62fea885448bc80335ae18ba6949168dd1632c1d51387cb1f8e33c25b0 Oct 31 13:38:19.996900 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:19.996865 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-1e72f6/rate-test-model-1e72f6-kserve-8454cfd857-tbqhz" event={"ID":"3cc7b994-8cee-47e2-9a6d-b587e3ac1eef","Type":"ContainerStarted","Data":"f997a307b48686a73d703cf54fe1995c13494529a4bb63c6868f269d6e39dbce"} Oct 31 13:38:19.997442 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:19.996916 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-1e72f6/rate-test-model-1e72f6-kserve-8454cfd857-tbqhz" event={"ID":"3cc7b994-8cee-47e2-9a6d-b587e3ac1eef","Type":"ContainerStarted","Data":"8c182f62fea885448bc80335ae18ba6949168dd1632c1d51387cb1f8e33c25b0"} Oct 31 13:38:20.019399 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:20.019333 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-1e72f6/rate-test-model-1e72f6-kserve-8454cfd857-tbqhz" podStartSLOduration=2.019316465 podStartE2EDuration="2.019316465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:38:20.018628061 +0000 UTC m=+3950.726214355" watchObservedRunningTime="2026-10-31 13:38:20.019316465 +0000 UTC m=+3950.726902764" Oct 31 13:38:20.209016 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:20.208985 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-1e72f6/rate-test-model-1e72f6-kserve-8454cfd857-tbqhz" Oct 31 13:38:25.604635 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:25.604600 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-50b4aa-54d6cd77df-zsv8v"] Oct 31 13:38:25.633223 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:25.633179 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-50b4aa-58bd97d4c5-mgqbp"] Oct 31 13:38:29.209192 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:29.209157 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-1e72f6/rate-test-model-1e72f6-kserve-8454cfd857-tbqhz" Oct 31 13:38:29.221890 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:29.221863 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-1e72f6/rate-test-model-1e72f6-kserve-8454cfd857-tbqhz" Oct 31 13:38:30.090888 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:30.090854 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-1e72f6/rate-test-model-1e72f6-kserve-8454cfd857-tbqhz" Oct 31 13:38:30.605143 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:30.605097 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-50b4aa-54d6cd77df-zsv8v" podUID="6f93d32d-fa13-4c7b-a12f-10a709bed671" containerName="payload-processing" containerID="cri-o://37b25fd236a25d96cf66cfe2a4021918d097dbfd516b8325c3c511956aaaeca2" gracePeriod=25 Oct 31 13:38:30.634000 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:30.633941 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-50b4aa-58bd97d4c5-mgqbp" podUID="d47d728a-192c-4622-99c9-e3f018a8cb8f" containerName="payload-pre-processing" containerID="cri-o://2728d51cd12169f51c0533d81884e2d717f671b0200d72a3949fc471f8bcc891" gracePeriod=25 Oct 31 13:38:30.920935 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:30.920908 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-50b4aa-54d6cd77df-zsv8v" Oct 31 13:38:31.037869 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:31.037838 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f93d32d-fa13-4c7b-a12f-10a709bed671-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f93d32d-fa13-4c7b-a12f-10a709bed671-plugins-config-volume\") pod \"6f93d32d-fa13-4c7b-a12f-10a709bed671\" (UID: \"6f93d32d-fa13-4c7b-a12f-10a709bed671\") " Oct 31 13:38:31.038040 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:31.038017 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f93d32d-fa13-4c7b-a12f-10a709bed671-kube-api-access-rrtl6\" (UniqueName: \"kubernetes.io/projected/6f93d32d-fa13-4c7b-a12f-10a709bed671-kube-api-access-rrtl6\") pod \"6f93d32d-fa13-4c7b-a12f-10a709bed671\" (UID: \"6f93d32d-fa13-4c7b-a12f-10a709bed671\") " Oct 31 13:38:31.038227 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:31.038204 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f93d32d-fa13-4c7b-a12f-10a709bed671-plugins-config-volume" pod "6f93d32d-fa13-4c7b-a12f-10a709bed671" (UID: "6f93d32d-fa13-4c7b-a12f-10a709bed671"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:38:31.038565 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:31.038547 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f93d32d-fa13-4c7b-a12f-10a709bed671-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:31.040184 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:31.040159 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f93d32d-fa13-4c7b-a12f-10a709bed671-kube-api-access-rrtl6" pod "6f93d32d-fa13-4c7b-a12f-10a709bed671" (UID: "6f93d32d-fa13-4c7b-a12f-10a709bed671"). InnerVolumeSpecName "kube-api-access-rrtl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:38:31.087724 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:31.087697 5693 generic.go:356] "Generic (PLEG): container finished" podID="6f93d32d-fa13-4c7b-a12f-10a709bed671" containerID="37b25fd236a25d96cf66cfe2a4021918d097dbfd516b8325c3c511956aaaeca2" exitCode=0 Oct 31 13:38:31.087898 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:31.087745 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-50b4aa-54d6cd77df-zsv8v" Oct 31 13:38:31.087898 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:31.087783 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-50b4aa-54d6cd77df-zsv8v" event={"ID":"6f93d32d-fa13-4c7b-a12f-10a709bed671","Type":"ContainerDied","Data":"37b25fd236a25d96cf66cfe2a4021918d097dbfd516b8325c3c511956aaaeca2"} Oct 31 13:38:31.087898 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:31.087840 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-50b4aa-54d6cd77df-zsv8v" event={"ID":"6f93d32d-fa13-4c7b-a12f-10a709bed671","Type":"ContainerDied","Data":"e17eaf6b2e3ffc37064d99dd04e4197080fc0e50b48c41910e3592eec646b1d8"} Oct 31 13:38:31.087898 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:31.087870 5693 scope.go:122] "RemoveContainer" containerID="37b25fd236a25d96cf66cfe2a4021918d097dbfd516b8325c3c511956aaaeca2" Oct 31 13:38:31.099870 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:31.099847 5693 scope.go:122] "RemoveContainer" containerID="37b25fd236a25d96cf66cfe2a4021918d097dbfd516b8325c3c511956aaaeca2" Oct 31 13:38:31.100132 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:38:31.100113 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37b25fd236a25d96cf66cfe2a4021918d097dbfd516b8325c3c511956aaaeca2\": container with ID starting with 37b25fd236a25d96cf66cfe2a4021918d097dbfd516b8325c3c511956aaaeca2 not found: ID does not exist" containerID="37b25fd236a25d96cf66cfe2a4021918d097dbfd516b8325c3c511956aaaeca2" Oct 31 13:38:31.100197 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:31.100140 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37b25fd236a25d96cf66cfe2a4021918d097dbfd516b8325c3c511956aaaeca2"} err="failed to get container status \"37b25fd236a25d96cf66cfe2a4021918d097dbfd516b8325c3c511956aaaeca2\": rpc error: code = NotFound desc = could not find container \"37b25fd236a25d96cf66cfe2a4021918d097dbfd516b8325c3c511956aaaeca2\": container with ID starting with 37b25fd236a25d96cf66cfe2a4021918d097dbfd516b8325c3c511956aaaeca2 not found: ID does not exist" Oct 31 13:38:31.124440 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:31.124405 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-50b4aa-54d6cd77df-zsv8v"] Oct 31 13:38:31.132807 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:31.132785 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-50b4aa-54d6cd77df-zsv8v"] Oct 31 13:38:31.139786 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:31.139724 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrtl6\" (UniqueName: \"kubernetes.io/projected/6f93d32d-fa13-4c7b-a12f-10a709bed671-kube-api-access-rrtl6\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:31.410408 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:31.410377 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f93d32d-fa13-4c7b-a12f-10a709bed671" path="/var/lib/kubelet/pods/6f93d32d-fa13-4c7b-a12f-10a709bed671/volumes" Oct 31 13:38:31.740872 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:31.740796 5693 scope.go:122] "RemoveContainer" containerID="d2dd6fd139a26292b848500cbbcf6fc34f5521b344e17e70f731d273dae272d3" Oct 31 13:38:37.243447 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:37.243404 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-50b4aa-58bd97d4c5-mgqbp" podUID="d47d728a-192c-4622-99c9-e3f018a8cb8f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:38:39.927934 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:39.927841 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl"] Oct 31 13:38:39.928400 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:39.928146 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl" podUID="a02b81f9-b4ba-4a1e-a270-62bd75804349" containerName="maas-api" containerID="cri-o://dbe04374ad747eb66d0eb602e8090f70e9ca7b1181e5ed1f99c8ebd95591dfc7" gracePeriod=30 Oct 31 13:38:40.229484 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:40.229235 5693 generic.go:356] "Generic (PLEG): container finished" podID="a02b81f9-b4ba-4a1e-a270-62bd75804349" containerID="dbe04374ad747eb66d0eb602e8090f70e9ca7b1181e5ed1f99c8ebd95591dfc7" exitCode=0 Oct 31 13:38:40.229484 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:40.229366 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl" event={"ID":"a02b81f9-b4ba-4a1e-a270-62bd75804349","Type":"ContainerDied","Data":"dbe04374ad747eb66d0eb602e8090f70e9ca7b1181e5ed1f99c8ebd95591dfc7"} Oct 31 13:38:40.357394 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:40.357369 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl" Oct 31 13:38:40.480401 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:40.480308 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a02b81f9-b4ba-4a1e-a270-62bd75804349-kube-api-access-nqg88\" (UniqueName: \"kubernetes.io/projected/a02b81f9-b4ba-4a1e-a270-62bd75804349-kube-api-access-nqg88\") pod \"a02b81f9-b4ba-4a1e-a270-62bd75804349\" (UID: \"a02b81f9-b4ba-4a1e-a270-62bd75804349\") " Oct 31 13:38:40.480586 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:40.480440 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a02b81f9-b4ba-4a1e-a270-62bd75804349-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a02b81f9-b4ba-4a1e-a270-62bd75804349-openshift-service-ca\") pod \"a02b81f9-b4ba-4a1e-a270-62bd75804349\" (UID: \"a02b81f9-b4ba-4a1e-a270-62bd75804349\") " Oct 31 13:38:40.480586 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:40.480552 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a02b81f9-b4ba-4a1e-a270-62bd75804349-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a02b81f9-b4ba-4a1e-a270-62bd75804349-metrics-certs\") pod \"a02b81f9-b4ba-4a1e-a270-62bd75804349\" (UID: \"a02b81f9-b4ba-4a1e-a270-62bd75804349\") " Oct 31 13:38:40.480752 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:40.480695 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a02b81f9-b4ba-4a1e-a270-62bd75804349-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a02b81f9-b4ba-4a1e-a270-62bd75804349-maas-api-tls\") pod \"a02b81f9-b4ba-4a1e-a270-62bd75804349\" (UID: \"a02b81f9-b4ba-4a1e-a270-62bd75804349\") " Oct 31 13:38:40.480901 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:40.480878 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a02b81f9-b4ba-4a1e-a270-62bd75804349-openshift-service-ca" pod "a02b81f9-b4ba-4a1e-a270-62bd75804349" (UID: "a02b81f9-b4ba-4a1e-a270-62bd75804349"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:38:40.481253 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:40.481217 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a02b81f9-b4ba-4a1e-a270-62bd75804349-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:40.482814 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:40.482785 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a02b81f9-b4ba-4a1e-a270-62bd75804349-maas-api-tls" pod "a02b81f9-b4ba-4a1e-a270-62bd75804349" (UID: "a02b81f9-b4ba-4a1e-a270-62bd75804349"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:38:40.482916 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:40.482841 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a02b81f9-b4ba-4a1e-a270-62bd75804349-metrics-certs" pod "a02b81f9-b4ba-4a1e-a270-62bd75804349" (UID: "a02b81f9-b4ba-4a1e-a270-62bd75804349"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:38:40.482968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:40.482952 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a02b81f9-b4ba-4a1e-a270-62bd75804349-kube-api-access-nqg88" pod "a02b81f9-b4ba-4a1e-a270-62bd75804349" (UID: "a02b81f9-b4ba-4a1e-a270-62bd75804349"). InnerVolumeSpecName "kube-api-access-nqg88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:38:40.582392 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:40.582356 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a02b81f9-b4ba-4a1e-a270-62bd75804349-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:40.582392 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:40.582386 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a02b81f9-b4ba-4a1e-a270-62bd75804349-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:40.582597 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:40.582396 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqg88\" (UniqueName: \"kubernetes.io/projected/a02b81f9-b4ba-4a1e-a270-62bd75804349-kube-api-access-nqg88\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:41.239180 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:41.239149 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl" Oct 31 13:38:41.239608 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:41.239175 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl" event={"ID":"a02b81f9-b4ba-4a1e-a270-62bd75804349","Type":"ContainerDied","Data":"35be075eb4ed12d6f894383c2e50f24dfd3f8a24db0c29156a0ae938ccd23fcf"} Oct 31 13:38:41.239608 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:41.239240 5693 scope.go:122] "RemoveContainer" containerID="dbe04374ad747eb66d0eb602e8090f70e9ca7b1181e5ed1f99c8ebd95591dfc7" Oct 31 13:38:41.263718 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:41.263689 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl"] Oct 31 13:38:41.268426 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:41.268403 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-50b4aa-665b74fb98-96krl"] Oct 31 13:38:41.411238 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:41.411206 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a02b81f9-b4ba-4a1e-a270-62bd75804349" path="/var/lib/kubelet/pods/a02b81f9-b4ba-4a1e-a270-62bd75804349/volumes" Oct 31 13:38:47.243799 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:47.243744 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-50b4aa-58bd97d4c5-mgqbp" podUID="d47d728a-192c-4622-99c9-e3f018a8cb8f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:38:48.691010 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:38:48.690896 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:38:51.403314 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:51.385233 5693 generic.go:356] "Generic (PLEG): container finished" podID="d47d728a-192c-4622-99c9-e3f018a8cb8f" containerID="2728d51cd12169f51c0533d81884e2d717f671b0200d72a3949fc471f8bcc891" exitCode=0 Oct 31 13:38:51.403314 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:51.385466 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-50b4aa-58bd97d4c5-mgqbp" event={"ID":"d47d728a-192c-4622-99c9-e3f018a8cb8f","Type":"ContainerDied","Data":"2728d51cd12169f51c0533d81884e2d717f671b0200d72a3949fc471f8bcc891"} Oct 31 13:38:51.423939 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:51.423909 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-50b4aa-58bd97d4c5-mgqbp" Oct 31 13:38:51.544731 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:51.541913 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d47d728a-192c-4622-99c9-e3f018a8cb8f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d47d728a-192c-4622-99c9-e3f018a8cb8f-plugins-config-volume\") pod \"d47d728a-192c-4622-99c9-e3f018a8cb8f\" (UID: \"d47d728a-192c-4622-99c9-e3f018a8cb8f\") " Oct 31 13:38:51.544731 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:51.542547 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d47d728a-192c-4622-99c9-e3f018a8cb8f-plugins-config-volume" pod "d47d728a-192c-4622-99c9-e3f018a8cb8f" (UID: "d47d728a-192c-4622-99c9-e3f018a8cb8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:38:51.544731 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:51.542693 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d47d728a-192c-4622-99c9-e3f018a8cb8f-kube-api-access-tc8l8\" (UniqueName: \"kubernetes.io/projected/d47d728a-192c-4622-99c9-e3f018a8cb8f-kube-api-access-tc8l8\") pod \"d47d728a-192c-4622-99c9-e3f018a8cb8f\" (UID: \"d47d728a-192c-4622-99c9-e3f018a8cb8f\") " Oct 31 13:38:51.549611 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:51.549385 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d47d728a-192c-4622-99c9-e3f018a8cb8f-kube-api-access-tc8l8" pod "d47d728a-192c-4622-99c9-e3f018a8cb8f" (UID: "d47d728a-192c-4622-99c9-e3f018a8cb8f"). InnerVolumeSpecName "kube-api-access-tc8l8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:38:51.549611 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:51.549605 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d47d728a-192c-4622-99c9-e3f018a8cb8f-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:51.652067 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:51.652033 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc8l8\" (UniqueName: \"kubernetes.io/projected/d47d728a-192c-4622-99c9-e3f018a8cb8f-kube-api-access-tc8l8\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:52.418857 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:52.418825 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-50b4aa-58bd97d4c5-mgqbp" Oct 31 13:38:52.419332 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:52.418882 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-50b4aa-58bd97d4c5-mgqbp" event={"ID":"d47d728a-192c-4622-99c9-e3f018a8cb8f","Type":"ContainerDied","Data":"32c4addea235dfe178b226bec4fbc220007b4e414a9f21834a7c9668c86c5e5a"} Oct 31 13:38:52.419332 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:52.418956 5693 scope.go:122] "RemoveContainer" containerID="2728d51cd12169f51c0533d81884e2d717f671b0200d72a3949fc471f8bcc891" Oct 31 13:38:52.456577 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:52.456536 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-50b4aa-58bd97d4c5-mgqbp"] Oct 31 13:38:52.463956 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:52.463928 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-50b4aa-58bd97d4c5-mgqbp"] Oct 31 13:38:53.411320 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:53.411285 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d47d728a-192c-4622-99c9-e3f018a8cb8f" path="/var/lib/kubelet/pods/d47d728a-192c-4622-99c9-e3f018a8cb8f/volumes" Oct 31 13:38:56.003625 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:56.003590 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-4574ec/rate-test-model-4574ec-kserve-66686cb98f-lzbn9"] Oct 31 13:38:56.004377 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:56.004360 5693 cpu_manager.go:414] "containerMap: removing container" podUID="d137459e-9920-4c27-a910-0216cc6a7168" containerName="istio-proxy" Oct 31 13:38:56.004377 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:56.004376 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d137459e-9920-4c27-a910-0216cc6a7168" containerName="istio-proxy" Oct 31 13:38:56.004551 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:56.004403 5693 cpu_manager.go:414] "containerMap: removing container" podUID="d47d728a-192c-4622-99c9-e3f018a8cb8f" containerName="payload-pre-processing" Oct 31 13:38:56.004551 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:56.004408 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d47d728a-192c-4622-99c9-e3f018a8cb8f" containerName="payload-pre-processing" Oct 31 13:38:56.004551 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:56.004422 5693 cpu_manager.go:414] "containerMap: removing container" podUID="a02b81f9-b4ba-4a1e-a270-62bd75804349" containerName="maas-api" Oct 31 13:38:56.004551 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:56.004427 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a02b81f9-b4ba-4a1e-a270-62bd75804349" containerName="maas-api" Oct 31 13:38:56.004551 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:56.004443 5693 cpu_manager.go:414] "containerMap: removing container" podUID="6f93d32d-fa13-4c7b-a12f-10a709bed671" containerName="payload-processing" Oct 31 13:38:56.004551 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:56.004448 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f93d32d-fa13-4c7b-a12f-10a709bed671" containerName="payload-processing" Oct 31 13:38:56.004551 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:56.004461 5693 cpu_manager.go:414] "containerMap: removing container" podUID="efccc572-e9f9-4f4a-ab45-c7458b0e2e58" containerName="istio-proxy" Oct 31 13:38:56.004551 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:56.004465 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efccc572-e9f9-4f4a-ab45-c7458b0e2e58" containerName="istio-proxy" Oct 31 13:38:56.004865 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:56.004576 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="d47d728a-192c-4622-99c9-e3f018a8cb8f" containerName="payload-pre-processing" Oct 31 13:38:56.004865 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:56.004588 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="a02b81f9-b4ba-4a1e-a270-62bd75804349" containerName="maas-api" Oct 31 13:38:56.004865 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:56.004596 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f93d32d-fa13-4c7b-a12f-10a709bed671" containerName="payload-processing" Oct 31 13:38:56.009214 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:56.009195 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4574ec/rate-test-model-4574ec-kserve-66686cb98f-lzbn9" Oct 31 13:38:56.029863 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:56.029833 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-4574ec\"/\"openshift-service-ca.crt\"" Oct 31 13:38:56.030067 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:56.029841 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-4574ec\"/\"default-dockercfg-2qql5\"" Oct 31 13:38:56.030380 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:56.029985 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-4574ec\"/\"kube-root-ca.crt\"" Oct 31 13:38:56.030544 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:56.030078 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-4574ec\"/\"rate-test-model-4574ec-kserve-self-signed-certs\"" Oct 31 13:38:56.039223 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:56.038469 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4574ec/rate-test-model-4574ec-kserve-66686cb98f-lzbn9"] Oct 31 13:38:56.048680 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:56.048632 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0191a8b7-5eee-4437-aa2b-c228014966cd-tls-certs\") pod \"rate-test-model-4574ec-kserve-66686cb98f-lzbn9\" (UID: \"0191a8b7-5eee-4437-aa2b-c228014966cd\") " pod="ai-tenant-e2e-shared-a-w3-4574ec/rate-test-model-4574ec-kserve-66686cb98f-lzbn9" Oct 31 13:38:56.048890 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:56.048867 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqhrj\" (UniqueName: \"kubernetes.io/projected/0191a8b7-5eee-4437-aa2b-c228014966cd-kube-api-access-lqhrj\") pod \"rate-test-model-4574ec-kserve-66686cb98f-lzbn9\" (UID: \"0191a8b7-5eee-4437-aa2b-c228014966cd\") " pod="ai-tenant-e2e-shared-a-w3-4574ec/rate-test-model-4574ec-kserve-66686cb98f-lzbn9" Oct 31 13:38:56.049029 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:56.048941 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0191a8b7-5eee-4437-aa2b-c228014966cd-dshm\") pod \"rate-test-model-4574ec-kserve-66686cb98f-lzbn9\" (UID: \"0191a8b7-5eee-4437-aa2b-c228014966cd\") " pod="ai-tenant-e2e-shared-a-w3-4574ec/rate-test-model-4574ec-kserve-66686cb98f-lzbn9" Oct 31 13:38:56.049029 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:56.048964 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0191a8b7-5eee-4437-aa2b-c228014966cd-home\") pod \"rate-test-model-4574ec-kserve-66686cb98f-lzbn9\" (UID: \"0191a8b7-5eee-4437-aa2b-c228014966cd\") " pod="ai-tenant-e2e-shared-a-w3-4574ec/rate-test-model-4574ec-kserve-66686cb98f-lzbn9" Oct 31 13:38:56.049029 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:56.048994 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0191a8b7-5eee-4437-aa2b-c228014966cd-model-cache\") pod \"rate-test-model-4574ec-kserve-66686cb98f-lzbn9\" (UID: \"0191a8b7-5eee-4437-aa2b-c228014966cd\") " pod="ai-tenant-e2e-shared-a-w3-4574ec/rate-test-model-4574ec-kserve-66686cb98f-lzbn9" Oct 31 13:38:56.049234 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:56.049111 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0191a8b7-5eee-4437-aa2b-c228014966cd-tmp-dir\") pod \"rate-test-model-4574ec-kserve-66686cb98f-lzbn9\" (UID: \"0191a8b7-5eee-4437-aa2b-c228014966cd\") " pod="ai-tenant-e2e-shared-a-w3-4574ec/rate-test-model-4574ec-kserve-66686cb98f-lzbn9" Oct 31 13:38:56.150709 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:56.150675 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0191a8b7-5eee-4437-aa2b-c228014966cd-tmp-dir\") pod \"rate-test-model-4574ec-kserve-66686cb98f-lzbn9\" (UID: \"0191a8b7-5eee-4437-aa2b-c228014966cd\") " pod="ai-tenant-e2e-shared-a-w3-4574ec/rate-test-model-4574ec-kserve-66686cb98f-lzbn9" Oct 31 13:38:56.150908 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:56.150886 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0191a8b7-5eee-4437-aa2b-c228014966cd-tls-certs\") pod \"rate-test-model-4574ec-kserve-66686cb98f-lzbn9\" (UID: \"0191a8b7-5eee-4437-aa2b-c228014966cd\") " pod="ai-tenant-e2e-shared-a-w3-4574ec/rate-test-model-4574ec-kserve-66686cb98f-lzbn9" Oct 31 13:38:56.150979 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:56.150922 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqhrj\" (UniqueName: \"kubernetes.io/projected/0191a8b7-5eee-4437-aa2b-c228014966cd-kube-api-access-lqhrj\") pod \"rate-test-model-4574ec-kserve-66686cb98f-lzbn9\" (UID: \"0191a8b7-5eee-4437-aa2b-c228014966cd\") " pod="ai-tenant-e2e-shared-a-w3-4574ec/rate-test-model-4574ec-kserve-66686cb98f-lzbn9" Oct 31 13:38:56.151040 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:56.150996 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0191a8b7-5eee-4437-aa2b-c228014966cd-dshm\") pod \"rate-test-model-4574ec-kserve-66686cb98f-lzbn9\" (UID: \"0191a8b7-5eee-4437-aa2b-c228014966cd\") " pod="ai-tenant-e2e-shared-a-w3-4574ec/rate-test-model-4574ec-kserve-66686cb98f-lzbn9" Oct 31 13:38:56.151040 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:56.151022 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0191a8b7-5eee-4437-aa2b-c228014966cd-home\") pod \"rate-test-model-4574ec-kserve-66686cb98f-lzbn9\" (UID: \"0191a8b7-5eee-4437-aa2b-c228014966cd\") " pod="ai-tenant-e2e-shared-a-w3-4574ec/rate-test-model-4574ec-kserve-66686cb98f-lzbn9" Oct 31 13:38:56.151170 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:56.151140 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0191a8b7-5eee-4437-aa2b-c228014966cd-tmp-dir\") pod \"rate-test-model-4574ec-kserve-66686cb98f-lzbn9\" (UID: \"0191a8b7-5eee-4437-aa2b-c228014966cd\") " pod="ai-tenant-e2e-shared-a-w3-4574ec/rate-test-model-4574ec-kserve-66686cb98f-lzbn9" Oct 31 13:38:56.151170 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:56.151146 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0191a8b7-5eee-4437-aa2b-c228014966cd-model-cache\") pod \"rate-test-model-4574ec-kserve-66686cb98f-lzbn9\" (UID: \"0191a8b7-5eee-4437-aa2b-c228014966cd\") " pod="ai-tenant-e2e-shared-a-w3-4574ec/rate-test-model-4574ec-kserve-66686cb98f-lzbn9" Oct 31 13:38:56.151292 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:56.151279 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0191a8b7-5eee-4437-aa2b-c228014966cd-home\") pod \"rate-test-model-4574ec-kserve-66686cb98f-lzbn9\" (UID: \"0191a8b7-5eee-4437-aa2b-c228014966cd\") " pod="ai-tenant-e2e-shared-a-w3-4574ec/rate-test-model-4574ec-kserve-66686cb98f-lzbn9" Oct 31 13:38:56.151491 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:56.151469 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0191a8b7-5eee-4437-aa2b-c228014966cd-model-cache\") pod \"rate-test-model-4574ec-kserve-66686cb98f-lzbn9\" (UID: \"0191a8b7-5eee-4437-aa2b-c228014966cd\") " pod="ai-tenant-e2e-shared-a-w3-4574ec/rate-test-model-4574ec-kserve-66686cb98f-lzbn9" Oct 31 13:38:56.153582 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:56.153559 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0191a8b7-5eee-4437-aa2b-c228014966cd-dshm\") pod \"rate-test-model-4574ec-kserve-66686cb98f-lzbn9\" (UID: \"0191a8b7-5eee-4437-aa2b-c228014966cd\") " pod="ai-tenant-e2e-shared-a-w3-4574ec/rate-test-model-4574ec-kserve-66686cb98f-lzbn9" Oct 31 13:38:56.153861 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:56.153844 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0191a8b7-5eee-4437-aa2b-c228014966cd-tls-certs\") pod \"rate-test-model-4574ec-kserve-66686cb98f-lzbn9\" (UID: \"0191a8b7-5eee-4437-aa2b-c228014966cd\") " pod="ai-tenant-e2e-shared-a-w3-4574ec/rate-test-model-4574ec-kserve-66686cb98f-lzbn9" Oct 31 13:38:56.177762 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:56.177738 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqhrj\" (UniqueName: \"kubernetes.io/projected/0191a8b7-5eee-4437-aa2b-c228014966cd-kube-api-access-lqhrj\") pod \"rate-test-model-4574ec-kserve-66686cb98f-lzbn9\" (UID: \"0191a8b7-5eee-4437-aa2b-c228014966cd\") " pod="ai-tenant-e2e-shared-a-w3-4574ec/rate-test-model-4574ec-kserve-66686cb98f-lzbn9" Oct 31 13:38:56.322856 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:56.322760 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4574ec/rate-test-model-4574ec-kserve-66686cb98f-lzbn9" Oct 31 13:38:56.691867 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:56.691835 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4574ec/rate-test-model-4574ec-kserve-66686cb98f-lzbn9"] Oct 31 13:38:56.696352 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:38:56.696325 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0191a8b7_5eee_4437_aa2b_c228014966cd.slice/crio-9a8978728ba77cce6c1bc785ee86d1c802e46360f3ad0e65806e068d5a98d685 WatchSource:0}: Error finding container 9a8978728ba77cce6c1bc785ee86d1c802e46360f3ad0e65806e068d5a98d685: Status 404 returned error can't find the container with id 9a8978728ba77cce6c1bc785ee86d1c802e46360f3ad0e65806e068d5a98d685 Oct 31 13:38:57.464687 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:57.464633 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4574ec/rate-test-model-4574ec-kserve-66686cb98f-lzbn9" event={"ID":"0191a8b7-5eee-4437-aa2b-c228014966cd","Type":"ContainerStarted","Data":"cc217b88d77ea8ab3c6694459e996d77c2813de173dc680a30469ded26cffa4e"} Oct 31 13:38:57.465122 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:57.464699 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4574ec/rate-test-model-4574ec-kserve-66686cb98f-lzbn9" event={"ID":"0191a8b7-5eee-4437-aa2b-c228014966cd","Type":"ContainerStarted","Data":"9a8978728ba77cce6c1bc785ee86d1c802e46360f3ad0e65806e068d5a98d685"} Oct 31 13:38:57.487441 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:57.487391 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-4574ec/rate-test-model-4574ec-kserve-66686cb98f-lzbn9" podStartSLOduration=2.487360382 podStartE2EDuration="2.487360382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:38:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:38:57.48639358 +0000 UTC m=+3988.193979877" watchObservedRunningTime="2026-10-31 13:38:57.487360382 +0000 UTC m=+3988.194946676" Oct 31 13:38:58.323747 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:38:58.323708 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-4574ec/rate-test-model-4574ec-kserve-66686cb98f-lzbn9" Oct 31 13:39:06.323523 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:06.323489 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-4574ec/rate-test-model-4574ec-kserve-66686cb98f-lzbn9" Oct 31 13:39:06.339342 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:06.339313 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-4574ec/rate-test-model-4574ec-kserve-66686cb98f-lzbn9" Oct 31 13:39:06.584272 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:06.584184 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-4574ec/rate-test-model-4574ec-kserve-66686cb98f-lzbn9" Oct 31 13:39:16.562866 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:16.562832 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9ef744-75b9b8c75c-hzlhr"] Oct 31 13:39:16.582657 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:16.582604 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5"] Oct 31 13:39:16.590156 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:16.590129 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9ef744-79656647f-g5cd2"] Oct 31 13:39:16.590347 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:16.590138 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" Oct 31 13:39:16.625208 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:16.625170 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5"] Oct 31 13:39:16.628088 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:16.628065 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-45bcad-openshift-default-dockercfg-gfb47\"" Oct 31 13:39:16.717903 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:16.717865 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-credential-socket\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5\" (UID: \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" Oct 31 13:39:16.718089 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:16.718012 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-istiod-ca-cert\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5\" (UID: \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" Oct 31 13:39:16.718140 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:16.718081 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-workload-socket\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5\" (UID: \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" Oct 31 13:39:16.718140 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:16.718133 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-istio-data\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5\" (UID: \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" Oct 31 13:39:16.718225 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:16.718162 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-istio-podinfo\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5\" (UID: \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" Oct 31 13:39:16.718410 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:16.718384 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfltj\" (UniqueName: \"kubernetes.io/projected/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-kube-api-access-nfltj\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5\" (UID: \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" Oct 31 13:39:16.718524 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:16.718507 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-istio-token\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5\" (UID: \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" Oct 31 13:39:16.718569 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:16.718545 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-workload-certs\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5\" (UID: \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" Oct 31 13:39:16.718612 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:16.718576 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-istio-envoy\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5\" (UID: \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" Oct 31 13:39:16.820108 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:16.820021 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-credential-socket\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5\" (UID: \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" Oct 31 13:39:16.820280 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:16.820110 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-istiod-ca-cert\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5\" (UID: \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" Oct 31 13:39:16.820280 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:16.820138 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-workload-socket\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5\" (UID: \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" Oct 31 13:39:16.820280 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:16.820240 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-istio-data\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5\" (UID: \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" Oct 31 13:39:16.820422 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:16.820293 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-istio-podinfo\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5\" (UID: \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" Oct 31 13:39:16.820476 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:16.820417 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfltj\" (UniqueName: \"kubernetes.io/projected/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-kube-api-access-nfltj\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5\" (UID: \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" Oct 31 13:39:16.820534 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:16.820487 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-credential-socket\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5\" (UID: \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" Oct 31 13:39:16.820534 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:16.820515 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-workload-socket\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5\" (UID: \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" Oct 31 13:39:16.820534 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:16.820516 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-istio-token\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5\" (UID: \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" Oct 31 13:39:16.820722 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:16.820564 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-workload-certs\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5\" (UID: \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" Oct 31 13:39:16.820722 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:16.820592 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-istio-envoy\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5\" (UID: \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" Oct 31 13:39:16.820722 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:16.820698 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-istio-data\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5\" (UID: \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" Oct 31 13:39:16.821284 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:16.820910 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-workload-certs\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5\" (UID: \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" Oct 31 13:39:16.821284 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:16.821181 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-istiod-ca-cert\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5\" (UID: \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" Oct 31 13:39:16.823099 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:16.823078 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-istio-envoy\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5\" (UID: \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" Oct 31 13:39:16.823255 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:16.823236 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-istio-podinfo\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5\" (UID: \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" Oct 31 13:39:16.841296 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:16.841268 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-istio-token\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5\" (UID: \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" Oct 31 13:39:16.841608 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:16.841590 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfltj\" (UniqueName: \"kubernetes.io/projected/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-kube-api-access-nfltj\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5\" (UID: \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" Oct 31 13:39:16.928104 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:16.926843 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" Oct 31 13:39:17.360017 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:39:17.359983 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5d9b1d1_f122_4108_9e23_d8d6eff64bc7.slice/crio-9f31eaba8964dc82463aec90a27c58e72f32bc23576380a25b0351f403a91e54 WatchSource:0}: Error finding container 9f31eaba8964dc82463aec90a27c58e72f32bc23576380a25b0351f403a91e54: Status 404 returned error can't find the container with id 9f31eaba8964dc82463aec90a27c58e72f32bc23576380a25b0351f403a91e54 Oct 31 13:39:17.360221 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:17.360204 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5"] Oct 31 13:39:17.362848 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:17.362812 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:39:17.363012 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:17.362893 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:39:17.363012 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:17.362988 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:39:17.638188 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:17.638098 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" event={"ID":"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7","Type":"ContainerStarted","Data":"c811a756f24a3eef88e1b5f3033cf051983f59e756b5f32363dc07ac949d0e2f"} Oct 31 13:39:17.638188 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:17.638157 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" event={"ID":"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7","Type":"ContainerStarted","Data":"9f31eaba8964dc82463aec90a27c58e72f32bc23576380a25b0351f403a91e54"} Oct 31 13:39:17.667616 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:17.667564 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" podStartSLOduration=1.667549403 podStartE2EDuration="1.667549403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:39:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:39:17.662441979 +0000 UTC m=+4008.370028268" watchObservedRunningTime="2026-10-31 13:39:17.667549403 +0000 UTC m=+4008.375135699" Oct 31 13:39:17.928674 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:17.928615 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" Oct 31 13:39:18.688741 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:39:18.688706 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:39:18.930121 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:18.930090 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" start-of-body= Oct 31 13:39:18.930291 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:18.930147 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" podUID="f5d9b1d1-f122-4108-9e23-d8d6eff64bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" Oct 31 13:39:19.929004 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:19.928968 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" start-of-body= Oct 31 13:39:19.929404 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:19.929021 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" podUID="f5d9b1d1-f122-4108-9e23-d8d6eff64bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" Oct 31 13:39:20.928789 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:20.928750 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" start-of-body= Oct 31 13:39:20.929006 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:20.928821 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" podUID="f5d9b1d1-f122-4108-9e23-d8d6eff64bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" Oct 31 13:39:21.563666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:21.563604 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-9ef744-75b9b8c75c-hzlhr" podUID="430e7981-8421-4898-8bef-fba2efcb1676" containerName="payload-processing" containerID="cri-o://01e5d6e29d41a9b9417fe7ced7bf79c95b660975d2d5bc429125bd6a577ce467" gracePeriod=25 Oct 31 13:39:21.600778 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:21.597735 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ef744-79656647f-g5cd2" podUID="bf6026f7-85c0-4659-8888-e4532b55f73a" containerName="payload-pre-processing" containerID="cri-o://97bfe515a0ec732f88b382caaf2e7ec62632bb226654fbd3913f940da26f911d" gracePeriod=25 Oct 31 13:39:21.928994 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:21.928954 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" start-of-body= Oct 31 13:39:21.929184 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:21.929025 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" podUID="f5d9b1d1-f122-4108-9e23-d8d6eff64bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" Oct 31 13:39:22.928400 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:22.928363 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" start-of-body= Oct 31 13:39:22.928870 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:22.928417 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" podUID="f5d9b1d1-f122-4108-9e23-d8d6eff64bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" Oct 31 13:39:23.928852 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:23.928820 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" start-of-body= Oct 31 13:39:23.929257 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:23.928886 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" podUID="f5d9b1d1-f122-4108-9e23-d8d6eff64bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" Oct 31 13:39:24.934227 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:24.934192 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" Oct 31 13:39:24.934744 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:24.934621 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" Oct 31 13:39:24.935623 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:24.935594 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" Oct 31 13:39:30.891091 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:30.891049 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ef744-79656647f-g5cd2" podUID="bf6026f7-85c0-4659-8888-e4532b55f73a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:39:30.891509 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:30.891065 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-9ef744-75b9b8c75c-hzlhr" podUID="430e7981-8421-4898-8bef-fba2efcb1676" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:39:40.891252 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:40.891208 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ef744-79656647f-g5cd2" podUID="bf6026f7-85c0-4659-8888-e4532b55f73a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:39:40.891721 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:40.891208 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-9ef744-75b9b8c75c-hzlhr" podUID="430e7981-8421-4898-8bef-fba2efcb1676" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:39:41.594703 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:41.594669 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-29e3a0-59874b6cdf-7hvmk"] Oct 31 13:39:41.608851 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:41.608818 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-29e3a0-b4ccb7d5f-2z6dt"] Oct 31 13:39:41.945664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:41.927897 5693 generic.go:356] "Generic (PLEG): container finished" podID="bf6026f7-85c0-4659-8888-e4532b55f73a" containerID="97bfe515a0ec732f88b382caaf2e7ec62632bb226654fbd3913f940da26f911d" exitCode=0 Oct 31 13:39:41.945664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:41.928056 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ef744-79656647f-g5cd2" event={"ID":"bf6026f7-85c0-4659-8888-e4532b55f73a","Type":"ContainerDied","Data":"97bfe515a0ec732f88b382caaf2e7ec62632bb226654fbd3913f940da26f911d"} Oct 31 13:39:41.965815 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:41.965376 5693 generic.go:356] "Generic (PLEG): container finished" podID="430e7981-8421-4898-8bef-fba2efcb1676" containerID="01e5d6e29d41a9b9417fe7ced7bf79c95b660975d2d5bc429125bd6a577ce467" exitCode=0 Oct 31 13:39:41.965815 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:41.965691 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9ef744-75b9b8c75c-hzlhr" event={"ID":"430e7981-8421-4898-8bef-fba2efcb1676","Type":"ContainerDied","Data":"01e5d6e29d41a9b9417fe7ced7bf79c95b660975d2d5bc429125bd6a577ce467"} Oct 31 13:39:41.971591 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:41.971253 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9ef744-75b9b8c75c-hzlhr" Oct 31 13:39:42.106332 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:42.106305 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ef744-79656647f-g5cd2" Oct 31 13:39:42.112245 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:42.112217 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/430e7981-8421-4898-8bef-fba2efcb1676-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/430e7981-8421-4898-8bef-fba2efcb1676-plugins-config-volume\") pod \"430e7981-8421-4898-8bef-fba2efcb1676\" (UID: \"430e7981-8421-4898-8bef-fba2efcb1676\") " Oct 31 13:39:42.112388 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:42.112278 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/430e7981-8421-4898-8bef-fba2efcb1676-kube-api-access-4tq8m\" (UniqueName: \"kubernetes.io/projected/430e7981-8421-4898-8bef-fba2efcb1676-kube-api-access-4tq8m\") pod \"430e7981-8421-4898-8bef-fba2efcb1676\" (UID: \"430e7981-8421-4898-8bef-fba2efcb1676\") " Oct 31 13:39:42.112677 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:42.112633 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/430e7981-8421-4898-8bef-fba2efcb1676-plugins-config-volume" pod "430e7981-8421-4898-8bef-fba2efcb1676" (UID: "430e7981-8421-4898-8bef-fba2efcb1676"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:39:42.112850 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:42.112831 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/430e7981-8421-4898-8bef-fba2efcb1676-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:39:42.114757 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:42.114736 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/430e7981-8421-4898-8bef-fba2efcb1676-kube-api-access-4tq8m" pod "430e7981-8421-4898-8bef-fba2efcb1676" (UID: "430e7981-8421-4898-8bef-fba2efcb1676"). InnerVolumeSpecName "kube-api-access-4tq8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:39:42.214023 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:42.213991 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf6026f7-85c0-4659-8888-e4532b55f73a-kube-api-access-gsj4j\" (UniqueName: \"kubernetes.io/projected/bf6026f7-85c0-4659-8888-e4532b55f73a-kube-api-access-gsj4j\") pod \"bf6026f7-85c0-4659-8888-e4532b55f73a\" (UID: \"bf6026f7-85c0-4659-8888-e4532b55f73a\") " Oct 31 13:39:42.214216 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:42.214198 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf6026f7-85c0-4659-8888-e4532b55f73a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf6026f7-85c0-4659-8888-e4532b55f73a-plugins-config-volume\") pod \"bf6026f7-85c0-4659-8888-e4532b55f73a\" (UID: \"bf6026f7-85c0-4659-8888-e4532b55f73a\") " Oct 31 13:39:42.214590 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:42.214563 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf6026f7-85c0-4659-8888-e4532b55f73a-plugins-config-volume" pod "bf6026f7-85c0-4659-8888-e4532b55f73a" (UID: "bf6026f7-85c0-4659-8888-e4532b55f73a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:39:42.214825 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:42.214807 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf6026f7-85c0-4659-8888-e4532b55f73a-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:39:42.214894 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:42.214830 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tq8m\" (UniqueName: \"kubernetes.io/projected/430e7981-8421-4898-8bef-fba2efcb1676-kube-api-access-4tq8m\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:39:42.216472 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:42.216451 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf6026f7-85c0-4659-8888-e4532b55f73a-kube-api-access-gsj4j" pod "bf6026f7-85c0-4659-8888-e4532b55f73a" (UID: "bf6026f7-85c0-4659-8888-e4532b55f73a"). InnerVolumeSpecName "kube-api-access-gsj4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:39:42.316273 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:42.316230 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsj4j\" (UniqueName: \"kubernetes.io/projected/bf6026f7-85c0-4659-8888-e4532b55f73a-kube-api-access-gsj4j\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:39:42.975266 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:42.975240 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ef744-79656647f-g5cd2" Oct 31 13:39:42.975266 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:42.975249 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ef744-79656647f-g5cd2" event={"ID":"bf6026f7-85c0-4659-8888-e4532b55f73a","Type":"ContainerDied","Data":"154a47949befe794a7c91824f2158e616b8fc398abb85861e86252255c27b743"} Oct 31 13:39:42.975853 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:42.975318 5693 scope.go:122] "RemoveContainer" containerID="97bfe515a0ec732f88b382caaf2e7ec62632bb226654fbd3913f940da26f911d" Oct 31 13:39:42.977285 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:42.977258 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9ef744-75b9b8c75c-hzlhr" Oct 31 13:39:42.977403 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:42.977290 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9ef744-75b9b8c75c-hzlhr" event={"ID":"430e7981-8421-4898-8bef-fba2efcb1676","Type":"ContainerDied","Data":"2f75eff167c10f97af31400fa88bddee77e7be66d6469ee8d520ba24682e2869"} Oct 31 13:39:42.988580 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:42.988559 5693 scope.go:122] "RemoveContainer" containerID="01e5d6e29d41a9b9417fe7ced7bf79c95b660975d2d5bc429125bd6a577ce467" Oct 31 13:39:43.004864 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:43.004826 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9ef744-79656647f-g5cd2"] Oct 31 13:39:43.009400 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:43.009373 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9ef744-79656647f-g5cd2"] Oct 31 13:39:43.021853 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:43.021827 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9ef744-75b9b8c75c-hzlhr"] Oct 31 13:39:43.025014 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:43.024995 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9ef744-75b9b8c75c-hzlhr"] Oct 31 13:39:43.412750 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:43.412716 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="430e7981-8421-4898-8bef-fba2efcb1676" path="/var/lib/kubelet/pods/430e7981-8421-4898-8bef-fba2efcb1676/volumes" Oct 31 13:39:43.418957 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:43.418923 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf6026f7-85c0-4659-8888-e4532b55f73a" path="/var/lib/kubelet/pods/bf6026f7-85c0-4659-8888-e4532b55f73a/volumes" Oct 31 13:39:46.595730 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:46.595694 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-29e3a0-59874b6cdf-7hvmk" podUID="5458e7f2-d785-4a4e-9b45-e5a9736b6758" containerName="payload-processing" containerID="cri-o://1f8334b7769d22a882a0c98a3775133bd6b8ffbdc89ec526a5e6baaa59d93770" gracePeriod=25 Oct 31 13:39:46.609983 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:46.609953 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-29e3a0-b4ccb7d5f-2z6dt" podUID="e18dd291-4694-41ad-922c-8852d7c12960" containerName="payload-pre-processing" containerID="cri-o://01ddeac2d9492d695ddba4145d70f9361789604246fa9c985b9d40c4685ec1e9" gracePeriod=25 Oct 31 13:39:47.737960 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:47.737918 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-29e3a0-b4ccb7d5f-2z6dt" podUID="e18dd291-4694-41ad-922c-8852d7c12960" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:39:47.738383 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:47.737966 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-29e3a0-59874b6cdf-7hvmk" podUID="5458e7f2-d785-4a4e-9b45-e5a9736b6758" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:39:48.688668 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:39:48.688614 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:39:57.738733 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:57.738689 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-29e3a0-b4ccb7d5f-2z6dt" podUID="e18dd291-4694-41ad-922c-8852d7c12960" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:39:57.739233 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:39:57.738731 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-29e3a0-59874b6cdf-7hvmk" podUID="5458e7f2-d785-4a4e-9b45-e5a9736b6758" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:40:05.644803 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:05.644770 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-7cdec6-5969765647-58sz8"] Oct 31 13:40:06.658001 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:40:06.657965 5693 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode18dd291_4694_41ad_922c_8852d7c12960.slice/crio-conmon-01ddeac2d9492d695ddba4145d70f9361789604246fa9c985b9d40c4685ec1e9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5458e7f2_d785_4a4e_9b45_e5a9736b6758.slice/crio-1f8334b7769d22a882a0c98a3775133bd6b8ffbdc89ec526a5e6baaa59d93770.scope\": RecentStats: unable to find data in memory cache]" Oct 31 13:40:06.917550 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:06.917525 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-29e3a0-59874b6cdf-7hvmk" Oct 31 13:40:06.955259 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:06.955234 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-29e3a0-b4ccb7d5f-2z6dt" Oct 31 13:40:06.989400 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:06.989365 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5458e7f2-d785-4a4e-9b45-e5a9736b6758-kube-api-access-qtlj4\" (UniqueName: \"kubernetes.io/projected/5458e7f2-d785-4a4e-9b45-e5a9736b6758-kube-api-access-qtlj4\") pod \"5458e7f2-d785-4a4e-9b45-e5a9736b6758\" (UID: \"5458e7f2-d785-4a4e-9b45-e5a9736b6758\") " Oct 31 13:40:06.989587 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:06.989421 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5458e7f2-d785-4a4e-9b45-e5a9736b6758-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5458e7f2-d785-4a4e-9b45-e5a9736b6758-plugins-config-volume\") pod \"5458e7f2-d785-4a4e-9b45-e5a9736b6758\" (UID: \"5458e7f2-d785-4a4e-9b45-e5a9736b6758\") " Oct 31 13:40:06.989873 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:06.989851 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5458e7f2-d785-4a4e-9b45-e5a9736b6758-plugins-config-volume" pod "5458e7f2-d785-4a4e-9b45-e5a9736b6758" (UID: "5458e7f2-d785-4a4e-9b45-e5a9736b6758"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:40:06.990304 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:06.990277 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5458e7f2-d785-4a4e-9b45-e5a9736b6758-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:06.992009 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:06.991983 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5458e7f2-d785-4a4e-9b45-e5a9736b6758-kube-api-access-qtlj4" pod "5458e7f2-d785-4a4e-9b45-e5a9736b6758" (UID: "5458e7f2-d785-4a4e-9b45-e5a9736b6758"). InnerVolumeSpecName "kube-api-access-qtlj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:40:07.091148 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:07.091110 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e18dd291-4694-41ad-922c-8852d7c12960-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e18dd291-4694-41ad-922c-8852d7c12960-plugins-config-volume\") pod \"e18dd291-4694-41ad-922c-8852d7c12960\" (UID: \"e18dd291-4694-41ad-922c-8852d7c12960\") " Oct 31 13:40:07.091423 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:07.091394 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e18dd291-4694-41ad-922c-8852d7c12960-kube-api-access-6zksb\" (UniqueName: \"kubernetes.io/projected/e18dd291-4694-41ad-922c-8852d7c12960-kube-api-access-6zksb\") pod \"e18dd291-4694-41ad-922c-8852d7c12960\" (UID: \"e18dd291-4694-41ad-922c-8852d7c12960\") " Oct 31 13:40:07.091633 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:07.091605 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e18dd291-4694-41ad-922c-8852d7c12960-plugins-config-volume" pod "e18dd291-4694-41ad-922c-8852d7c12960" (UID: "e18dd291-4694-41ad-922c-8852d7c12960"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:40:07.092086 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:07.092068 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtlj4\" (UniqueName: \"kubernetes.io/projected/5458e7f2-d785-4a4e-9b45-e5a9736b6758-kube-api-access-qtlj4\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:07.092135 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:07.092090 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e18dd291-4694-41ad-922c-8852d7c12960-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:07.093820 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:07.093802 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e18dd291-4694-41ad-922c-8852d7c12960-kube-api-access-6zksb" pod "e18dd291-4694-41ad-922c-8852d7c12960" (UID: "e18dd291-4694-41ad-922c-8852d7c12960"). InnerVolumeSpecName "kube-api-access-6zksb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:40:07.160973 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:07.160943 5693 generic.go:356] "Generic (PLEG): container finished" podID="5458e7f2-d785-4a4e-9b45-e5a9736b6758" containerID="1f8334b7769d22a882a0c98a3775133bd6b8ffbdc89ec526a5e6baaa59d93770" exitCode=0 Oct 31 13:40:07.161138 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:07.160992 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-29e3a0-59874b6cdf-7hvmk" Oct 31 13:40:07.161138 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:07.161033 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-29e3a0-59874b6cdf-7hvmk" event={"ID":"5458e7f2-d785-4a4e-9b45-e5a9736b6758","Type":"ContainerDied","Data":"1f8334b7769d22a882a0c98a3775133bd6b8ffbdc89ec526a5e6baaa59d93770"} Oct 31 13:40:07.161138 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:07.161130 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-29e3a0-59874b6cdf-7hvmk" event={"ID":"5458e7f2-d785-4a4e-9b45-e5a9736b6758","Type":"ContainerDied","Data":"a53aaed21c1374659e5917b16f912b42c6b096f98032193194c7ac078dfb568d"} Oct 31 13:40:07.161341 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:07.161162 5693 scope.go:122] "RemoveContainer" containerID="1f8334b7769d22a882a0c98a3775133bd6b8ffbdc89ec526a5e6baaa59d93770" Oct 31 13:40:07.162463 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:07.162438 5693 generic.go:356] "Generic (PLEG): container finished" podID="e18dd291-4694-41ad-922c-8852d7c12960" containerID="01ddeac2d9492d695ddba4145d70f9361789604246fa9c985b9d40c4685ec1e9" exitCode=0 Oct 31 13:40:07.162578 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:07.162565 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-29e3a0-b4ccb7d5f-2z6dt" event={"ID":"e18dd291-4694-41ad-922c-8852d7c12960","Type":"ContainerDied","Data":"01ddeac2d9492d695ddba4145d70f9361789604246fa9c985b9d40c4685ec1e9"} Oct 31 13:40:07.162666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:07.162597 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-29e3a0-b4ccb7d5f-2z6dt" event={"ID":"e18dd291-4694-41ad-922c-8852d7c12960","Type":"ContainerDied","Data":"4f6263121667e2914c5ef6834ab71cbf5721f2ac4cc16320483637e7237f6e55"} Oct 31 13:40:07.162666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:07.162667 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-29e3a0-b4ccb7d5f-2z6dt" Oct 31 13:40:07.173315 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:07.173281 5693 scope.go:122] "RemoveContainer" containerID="1f8334b7769d22a882a0c98a3775133bd6b8ffbdc89ec526a5e6baaa59d93770" Oct 31 13:40:07.173601 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:40:07.173580 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f8334b7769d22a882a0c98a3775133bd6b8ffbdc89ec526a5e6baaa59d93770\": container with ID starting with 1f8334b7769d22a882a0c98a3775133bd6b8ffbdc89ec526a5e6baaa59d93770 not found: ID does not exist" containerID="1f8334b7769d22a882a0c98a3775133bd6b8ffbdc89ec526a5e6baaa59d93770" Oct 31 13:40:07.173710 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:07.173606 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f8334b7769d22a882a0c98a3775133bd6b8ffbdc89ec526a5e6baaa59d93770"} err="failed to get container status \"1f8334b7769d22a882a0c98a3775133bd6b8ffbdc89ec526a5e6baaa59d93770\": rpc error: code = NotFound desc = could not find container \"1f8334b7769d22a882a0c98a3775133bd6b8ffbdc89ec526a5e6baaa59d93770\": container with ID starting with 1f8334b7769d22a882a0c98a3775133bd6b8ffbdc89ec526a5e6baaa59d93770 not found: ID does not exist" Oct 31 13:40:07.173710 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:07.173624 5693 scope.go:122] "RemoveContainer" containerID="01ddeac2d9492d695ddba4145d70f9361789604246fa9c985b9d40c4685ec1e9" Oct 31 13:40:07.189532 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:07.189504 5693 scope.go:122] "RemoveContainer" containerID="01ddeac2d9492d695ddba4145d70f9361789604246fa9c985b9d40c4685ec1e9" Oct 31 13:40:07.192829 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:40:07.192797 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01ddeac2d9492d695ddba4145d70f9361789604246fa9c985b9d40c4685ec1e9\": container with ID starting with 01ddeac2d9492d695ddba4145d70f9361789604246fa9c985b9d40c4685ec1e9 not found: ID does not exist" containerID="01ddeac2d9492d695ddba4145d70f9361789604246fa9c985b9d40c4685ec1e9" Oct 31 13:40:07.192936 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:07.192838 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01ddeac2d9492d695ddba4145d70f9361789604246fa9c985b9d40c4685ec1e9"} err="failed to get container status \"01ddeac2d9492d695ddba4145d70f9361789604246fa9c985b9d40c4685ec1e9\": rpc error: code = NotFound desc = could not find container \"01ddeac2d9492d695ddba4145d70f9361789604246fa9c985b9d40c4685ec1e9\": container with ID starting with 01ddeac2d9492d695ddba4145d70f9361789604246fa9c985b9d40c4685ec1e9 not found: ID does not exist" Oct 31 13:40:07.193417 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:07.193396 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zksb\" (UniqueName: \"kubernetes.io/projected/e18dd291-4694-41ad-922c-8852d7c12960-kube-api-access-6zksb\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:07.194485 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:07.194465 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-29e3a0-59874b6cdf-7hvmk"] Oct 31 13:40:07.197719 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:07.197701 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-29e3a0-59874b6cdf-7hvmk"] Oct 31 13:40:07.205582 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:07.205560 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-29e3a0-b4ccb7d5f-2z6dt"] Oct 31 13:40:07.211331 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:07.211310 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-29e3a0-b4ccb7d5f-2z6dt"] Oct 31 13:40:07.410163 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:07.410134 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5458e7f2-d785-4a4e-9b45-e5a9736b6758" path="/var/lib/kubelet/pods/5458e7f2-d785-4a4e-9b45-e5a9736b6758/volumes" Oct 31 13:40:07.410574 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:07.410560 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e18dd291-4694-41ad-922c-8852d7c12960" path="/var/lib/kubelet/pods/e18dd291-4694-41ad-922c-8852d7c12960/volumes" Oct 31 13:40:10.645498 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:10.645418 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-7cdec6-5969765647-58sz8" podUID="4de1aea2-a2ee-4257-8255-e63c353af739" containerName="payload-pre-processing" containerID="cri-o://a56fc10c3e41c1ff6df791eb819bea017bd0ab07f913c98d491145e7a20d8218" gracePeriod=25 Oct 31 13:40:13.033003 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:13.032968 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-7cdec6-5969765647-58sz8" podUID="4de1aea2-a2ee-4257-8255-e63c353af739" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:40:13.940264 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:13.940228 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p"] Oct 31 13:40:13.940489 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:13.940434 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" podUID="5455d8fe-86dd-413e-a001-80b5e91c7414" containerName="istio-proxy" containerID="cri-o://a8c49580c1a5b520cfafd57e24b0bf03b80a995d4e27fda418772adf7252cfdb" gracePeriod=30 Oct 31 13:40:18.258083 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:18.258048 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:40:18.258518 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:18.258109 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" podUID="5455d8fe-86dd-413e-a001-80b5e91c7414" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:40:18.688537 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:40:18.688493 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:40:19.287532 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:19.287155 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" Oct 31 13:40:19.291223 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:19.291194 5693 generic.go:356] "Generic (PLEG): container finished" podID="5455d8fe-86dd-413e-a001-80b5e91c7414" containerID="a8c49580c1a5b520cfafd57e24b0bf03b80a995d4e27fda418772adf7252cfdb" exitCode=0 Oct 31 13:40:19.291852 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:19.291808 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" Oct 31 13:40:19.292042 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:19.292021 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" event={"ID":"5455d8fe-86dd-413e-a001-80b5e91c7414","Type":"ContainerDied","Data":"a8c49580c1a5b520cfafd57e24b0bf03b80a995d4e27fda418772adf7252cfdb"} Oct 31 13:40:19.292118 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:19.292078 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p" event={"ID":"5455d8fe-86dd-413e-a001-80b5e91c7414","Type":"ContainerDied","Data":"f73bac3b4ad5a0768b43b2123d37d477e4df3d02f09c57dea6affc387edc1b9b"} Oct 31 13:40:19.292118 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:19.292110 5693 scope.go:122] "RemoveContainer" containerID="a8c49580c1a5b520cfafd57e24b0bf03b80a995d4e27fda418772adf7252cfdb" Oct 31 13:40:19.304226 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:19.304204 5693 scope.go:122] "RemoveContainer" containerID="a8c49580c1a5b520cfafd57e24b0bf03b80a995d4e27fda418772adf7252cfdb" Oct 31 13:40:19.304535 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:40:19.304516 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8c49580c1a5b520cfafd57e24b0bf03b80a995d4e27fda418772adf7252cfdb\": container with ID starting with a8c49580c1a5b520cfafd57e24b0bf03b80a995d4e27fda418772adf7252cfdb not found: ID does not exist" containerID="a8c49580c1a5b520cfafd57e24b0bf03b80a995d4e27fda418772adf7252cfdb" Oct 31 13:40:19.304650 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:19.304544 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8c49580c1a5b520cfafd57e24b0bf03b80a995d4e27fda418772adf7252cfdb"} err="failed to get container status \"a8c49580c1a5b520cfafd57e24b0bf03b80a995d4e27fda418772adf7252cfdb\": rpc error: code = NotFound desc = could not find container \"a8c49580c1a5b520cfafd57e24b0bf03b80a995d4e27fda418772adf7252cfdb\": container with ID starting with a8c49580c1a5b520cfafd57e24b0bf03b80a995d4e27fda418772adf7252cfdb not found: ID does not exist" Oct 31 13:40:19.425028 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:19.424999 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5455d8fe-86dd-413e-a001-80b5e91c7414-istio-token\" (UniqueName: \"kubernetes.io/projected/5455d8fe-86dd-413e-a001-80b5e91c7414-istio-token\") pod \"5455d8fe-86dd-413e-a001-80b5e91c7414\" (UID: \"5455d8fe-86dd-413e-a001-80b5e91c7414\") " Oct 31 13:40:19.425208 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:19.425049 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5455d8fe-86dd-413e-a001-80b5e91c7414-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5455d8fe-86dd-413e-a001-80b5e91c7414-workload-socket\") pod \"5455d8fe-86dd-413e-a001-80b5e91c7414\" (UID: \"5455d8fe-86dd-413e-a001-80b5e91c7414\") " Oct 31 13:40:19.425268 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:19.425245 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5455d8fe-86dd-413e-a001-80b5e91c7414-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5455d8fe-86dd-413e-a001-80b5e91c7414-istio-podinfo\") pod \"5455d8fe-86dd-413e-a001-80b5e91c7414\" (UID: \"5455d8fe-86dd-413e-a001-80b5e91c7414\") " Oct 31 13:40:19.425326 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:19.425278 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5455d8fe-86dd-413e-a001-80b5e91c7414-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5455d8fe-86dd-413e-a001-80b5e91c7414-credential-socket\") pod \"5455d8fe-86dd-413e-a001-80b5e91c7414\" (UID: \"5455d8fe-86dd-413e-a001-80b5e91c7414\") " Oct 31 13:40:19.425326 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:19.425309 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5455d8fe-86dd-413e-a001-80b5e91c7414-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5455d8fe-86dd-413e-a001-80b5e91c7414-istiod-ca-cert\") pod \"5455d8fe-86dd-413e-a001-80b5e91c7414\" (UID: \"5455d8fe-86dd-413e-a001-80b5e91c7414\") " Oct 31 13:40:19.425451 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:19.425361 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5455d8fe-86dd-413e-a001-80b5e91c7414-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5455d8fe-86dd-413e-a001-80b5e91c7414-istio-data\") pod \"5455d8fe-86dd-413e-a001-80b5e91c7414\" (UID: \"5455d8fe-86dd-413e-a001-80b5e91c7414\") " Oct 31 13:40:19.425451 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:19.425396 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5455d8fe-86dd-413e-a001-80b5e91c7414-workload-socket" pod "5455d8fe-86dd-413e-a001-80b5e91c7414" (UID: "5455d8fe-86dd-413e-a001-80b5e91c7414"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:40:19.425451 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:19.425416 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5455d8fe-86dd-413e-a001-80b5e91c7414-kube-api-access-gghgv\" (UniqueName: \"kubernetes.io/projected/5455d8fe-86dd-413e-a001-80b5e91c7414-kube-api-access-gghgv\") pod \"5455d8fe-86dd-413e-a001-80b5e91c7414\" (UID: \"5455d8fe-86dd-413e-a001-80b5e91c7414\") " Oct 31 13:40:19.425451 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:19.425440 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5455d8fe-86dd-413e-a001-80b5e91c7414-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5455d8fe-86dd-413e-a001-80b5e91c7414-istio-envoy\") pod \"5455d8fe-86dd-413e-a001-80b5e91c7414\" (UID: \"5455d8fe-86dd-413e-a001-80b5e91c7414\") " Oct 31 13:40:19.425686 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:19.425478 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5455d8fe-86dd-413e-a001-80b5e91c7414-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5455d8fe-86dd-413e-a001-80b5e91c7414-workload-certs\") pod \"5455d8fe-86dd-413e-a001-80b5e91c7414\" (UID: \"5455d8fe-86dd-413e-a001-80b5e91c7414\") " Oct 31 13:40:19.425686 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:19.425486 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5455d8fe-86dd-413e-a001-80b5e91c7414-credential-socket" pod "5455d8fe-86dd-413e-a001-80b5e91c7414" (UID: "5455d8fe-86dd-413e-a001-80b5e91c7414"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:40:19.425686 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:19.425623 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5455d8fe-86dd-413e-a001-80b5e91c7414-istio-data" pod "5455d8fe-86dd-413e-a001-80b5e91c7414" (UID: "5455d8fe-86dd-413e-a001-80b5e91c7414"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:40:19.426167 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:19.426075 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5455d8fe-86dd-413e-a001-80b5e91c7414-workload-certs" pod "5455d8fe-86dd-413e-a001-80b5e91c7414" (UID: "5455d8fe-86dd-413e-a001-80b5e91c7414"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:40:19.426308 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:19.426287 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5455d8fe-86dd-413e-a001-80b5e91c7414-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:19.426308 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:19.426308 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5455d8fe-86dd-413e-a001-80b5e91c7414-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:19.426529 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:19.426323 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5455d8fe-86dd-413e-a001-80b5e91c7414-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:19.426529 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:19.426376 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5455d8fe-86dd-413e-a001-80b5e91c7414-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:19.426529 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:19.426422 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5455d8fe-86dd-413e-a001-80b5e91c7414-istiod-ca-cert" pod "5455d8fe-86dd-413e-a001-80b5e91c7414" (UID: "5455d8fe-86dd-413e-a001-80b5e91c7414"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:40:19.427728 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:19.427706 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5455d8fe-86dd-413e-a001-80b5e91c7414-istio-podinfo" pod "5455d8fe-86dd-413e-a001-80b5e91c7414" (UID: "5455d8fe-86dd-413e-a001-80b5e91c7414"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:40:19.427917 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:19.427901 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5455d8fe-86dd-413e-a001-80b5e91c7414-kube-api-access-gghgv" pod "5455d8fe-86dd-413e-a001-80b5e91c7414" (UID: "5455d8fe-86dd-413e-a001-80b5e91c7414"). InnerVolumeSpecName "kube-api-access-gghgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:40:19.427989 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:19.427925 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5455d8fe-86dd-413e-a001-80b5e91c7414-istio-token" pod "5455d8fe-86dd-413e-a001-80b5e91c7414" (UID: "5455d8fe-86dd-413e-a001-80b5e91c7414"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:40:19.428190 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:19.428165 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5455d8fe-86dd-413e-a001-80b5e91c7414-istio-envoy" pod "5455d8fe-86dd-413e-a001-80b5e91c7414" (UID: "5455d8fe-86dd-413e-a001-80b5e91c7414"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:40:19.528135 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:19.528035 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gghgv\" (UniqueName: \"kubernetes.io/projected/5455d8fe-86dd-413e-a001-80b5e91c7414-kube-api-access-gghgv\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:19.528135 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:19.528068 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5455d8fe-86dd-413e-a001-80b5e91c7414-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:19.528135 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:19.528085 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5455d8fe-86dd-413e-a001-80b5e91c7414-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:19.528135 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:19.528094 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5455d8fe-86dd-413e-a001-80b5e91c7414-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:19.528135 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:19.528104 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5455d8fe-86dd-413e-a001-80b5e91c7414-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:19.619214 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:19.619176 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p"] Oct 31 13:40:19.623287 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:19.623262 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-5pz5p"] Oct 31 13:40:21.363220 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:21.363183 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-377f60-c59666d89-b6xbr"] Oct 31 13:40:21.390730 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:21.386726 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-377f60-bdf4746c-r7hng"] Oct 31 13:40:21.423578 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:21.423545 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5455d8fe-86dd-413e-a001-80b5e91c7414" path="/var/lib/kubelet/pods/5455d8fe-86dd-413e-a001-80b5e91c7414/volumes" Oct 31 13:40:23.033404 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:23.033362 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-7cdec6-5969765647-58sz8" podUID="4de1aea2-a2ee-4257-8255-e63c353af739" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:40:26.364312 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:26.364253 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-377f60-c59666d89-b6xbr" podUID="03b3ba33-5c21-487d-bee4-abc978d8cf06" containerName="payload-processing" containerID="cri-o://5fe013ac611b437e3e7d5c90ba1c3cb96e945b7e27e66c10d0f697611e0fccd9" gracePeriod=25 Oct 31 13:40:26.388067 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:26.388017 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-377f60-bdf4746c-r7hng" podUID="c2ca0d7c-7fb8-438d-9c27-09c1602339fa" containerName="payload-pre-processing" containerID="cri-o://03e53c48bb964e273baa0e73e65451c78201e1bb16ff6f30c355bb861cff99d1" gracePeriod=25 Oct 31 13:40:30.512943 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:30.512898 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-377f60-bdf4746c-r7hng" podUID="c2ca0d7c-7fb8-438d-9c27-09c1602339fa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:40:30.513437 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:30.512916 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-377f60-c59666d89-b6xbr" podUID="03b3ba33-5c21-487d-bee4-abc978d8cf06" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:40:30.965992 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:30.965963 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-7cdec6-5969765647-58sz8" Oct 31 13:40:31.034933 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:31.034847 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4de1aea2-a2ee-4257-8255-e63c353af739-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4de1aea2-a2ee-4257-8255-e63c353af739-plugins-config-volume\") pod \"4de1aea2-a2ee-4257-8255-e63c353af739\" (UID: \"4de1aea2-a2ee-4257-8255-e63c353af739\") " Oct 31 13:40:31.035160 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:31.035134 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4de1aea2-a2ee-4257-8255-e63c353af739-kube-api-access-rq2f5\" (UniqueName: \"kubernetes.io/projected/4de1aea2-a2ee-4257-8255-e63c353af739-kube-api-access-rq2f5\") pod \"4de1aea2-a2ee-4257-8255-e63c353af739\" (UID: \"4de1aea2-a2ee-4257-8255-e63c353af739\") " Oct 31 13:40:31.035314 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:31.035285 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4de1aea2-a2ee-4257-8255-e63c353af739-plugins-config-volume" pod "4de1aea2-a2ee-4257-8255-e63c353af739" (UID: "4de1aea2-a2ee-4257-8255-e63c353af739"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:40:31.035748 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:31.035728 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4de1aea2-a2ee-4257-8255-e63c353af739-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:31.037471 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:31.037449 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4de1aea2-a2ee-4257-8255-e63c353af739-kube-api-access-rq2f5" pod "4de1aea2-a2ee-4257-8255-e63c353af739" (UID: "4de1aea2-a2ee-4257-8255-e63c353af739"). InnerVolumeSpecName "kube-api-access-rq2f5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:40:31.137179 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:31.137142 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq2f5\" (UniqueName: \"kubernetes.io/projected/4de1aea2-a2ee-4257-8255-e63c353af739-kube-api-access-rq2f5\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:31.385363 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:31.385261 5693 generic.go:356] "Generic (PLEG): container finished" podID="4de1aea2-a2ee-4257-8255-e63c353af739" containerID="a56fc10c3e41c1ff6df791eb819bea017bd0ab07f913c98d491145e7a20d8218" exitCode=0 Oct 31 13:40:31.385363 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:31.385309 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-7cdec6-5969765647-58sz8" Oct 31 13:40:31.385606 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:31.385349 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-7cdec6-5969765647-58sz8" event={"ID":"4de1aea2-a2ee-4257-8255-e63c353af739","Type":"ContainerDied","Data":"a56fc10c3e41c1ff6df791eb819bea017bd0ab07f913c98d491145e7a20d8218"} Oct 31 13:40:31.385606 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:31.385425 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-7cdec6-5969765647-58sz8" event={"ID":"4de1aea2-a2ee-4257-8255-e63c353af739","Type":"ContainerDied","Data":"6edc01cf32e328ef79fdcfe3da0ad8605990c47b4043f63282160540a12f08f4"} Oct 31 13:40:31.385606 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:31.385455 5693 scope.go:122] "RemoveContainer" containerID="a56fc10c3e41c1ff6df791eb819bea017bd0ab07f913c98d491145e7a20d8218" Oct 31 13:40:31.400128 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:31.400103 5693 scope.go:122] "RemoveContainer" containerID="a56fc10c3e41c1ff6df791eb819bea017bd0ab07f913c98d491145e7a20d8218" Oct 31 13:40:31.400501 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:40:31.400470 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a56fc10c3e41c1ff6df791eb819bea017bd0ab07f913c98d491145e7a20d8218\": container with ID starting with a56fc10c3e41c1ff6df791eb819bea017bd0ab07f913c98d491145e7a20d8218 not found: ID does not exist" containerID="a56fc10c3e41c1ff6df791eb819bea017bd0ab07f913c98d491145e7a20d8218" Oct 31 13:40:31.400657 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:31.400512 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a56fc10c3e41c1ff6df791eb819bea017bd0ab07f913c98d491145e7a20d8218"} err="failed to get container status \"a56fc10c3e41c1ff6df791eb819bea017bd0ab07f913c98d491145e7a20d8218\": rpc error: code = NotFound desc = could not find container \"a56fc10c3e41c1ff6df791eb819bea017bd0ab07f913c98d491145e7a20d8218\": container with ID starting with a56fc10c3e41c1ff6df791eb819bea017bd0ab07f913c98d491145e7a20d8218 not found: ID does not exist" Oct 31 13:40:31.426056 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:31.426022 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-7cdec6-5969765647-58sz8"] Oct 31 13:40:31.429226 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:31.429200 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-7cdec6-5969765647-58sz8"] Oct 31 13:40:33.409873 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:33.409831 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4de1aea2-a2ee-4257-8255-e63c353af739" path="/var/lib/kubelet/pods/4de1aea2-a2ee-4257-8255-e63c353af739/volumes" Oct 31 13:40:40.512599 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:40.512554 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-377f60-c59666d89-b6xbr" podUID="03b3ba33-5c21-487d-bee4-abc978d8cf06" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:40:40.512599 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:40.512575 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-377f60-bdf4746c-r7hng" podUID="c2ca0d7c-7fb8-438d-9c27-09c1602339fa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:40:46.369899 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:40:46.369697 5693 helpers.go:246] readString: Failed to read "/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b3ba33_5c21_487d_bee4_abc978d8cf06.slice/crio-5fe013ac611b437e3e7d5c90ba1c3cb96e945b7e27e66c10d0f697611e0fccd9.scope/memory.swap.max": read /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b3ba33_5c21_487d_bee4_abc978d8cf06.slice/crio-5fe013ac611b437e3e7d5c90ba1c3cb96e945b7e27e66c10d0f697611e0fccd9.scope/memory.swap.max: no such device Oct 31 13:40:46.396865 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:40:46.396830 5693 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b3ba33_5c21_487d_bee4_abc978d8cf06.slice/crio-5fe013ac611b437e3e7d5c90ba1c3cb96e945b7e27e66c10d0f697611e0fccd9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b3ba33_5c21_487d_bee4_abc978d8cf06.slice/crio-conmon-5fe013ac611b437e3e7d5c90ba1c3cb96e945b7e27e66c10d0f697611e0fccd9.scope\": RecentStats: unable to find data in memory cache]" Oct 31 13:40:46.521095 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:46.521056 5693 generic.go:356] "Generic (PLEG): container finished" podID="c2ca0d7c-7fb8-438d-9c27-09c1602339fa" containerID="03e53c48bb964e273baa0e73e65451c78201e1bb16ff6f30c355bb861cff99d1" exitCode=0 Oct 31 13:40:46.521270 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:46.521123 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-377f60-bdf4746c-r7hng" event={"ID":"c2ca0d7c-7fb8-438d-9c27-09c1602339fa","Type":"ContainerDied","Data":"03e53c48bb964e273baa0e73e65451c78201e1bb16ff6f30c355bb861cff99d1"} Oct 31 13:40:46.523495 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:46.523465 5693 generic.go:356] "Generic (PLEG): container finished" podID="03b3ba33-5c21-487d-bee4-abc978d8cf06" containerID="5fe013ac611b437e3e7d5c90ba1c3cb96e945b7e27e66c10d0f697611e0fccd9" exitCode=0 Oct 31 13:40:46.523733 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:46.523536 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-377f60-c59666d89-b6xbr" event={"ID":"03b3ba33-5c21-487d-bee4-abc978d8cf06","Type":"ContainerDied","Data":"5fe013ac611b437e3e7d5c90ba1c3cb96e945b7e27e66c10d0f697611e0fccd9"} Oct 31 13:40:46.707375 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:46.707352 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-377f60-c59666d89-b6xbr" Oct 31 13:40:46.711060 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:46.711032 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-377f60-bdf4746c-r7hng" Oct 31 13:40:46.790048 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:46.790015 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03b3ba33-5c21-487d-bee4-abc978d8cf06-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03b3ba33-5c21-487d-bee4-abc978d8cf06-plugins-config-volume\") pod \"03b3ba33-5c21-487d-bee4-abc978d8cf06\" (UID: \"03b3ba33-5c21-487d-bee4-abc978d8cf06\") " Oct 31 13:40:46.790240 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:46.790221 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2ca0d7c-7fb8-438d-9c27-09c1602339fa-kube-api-access-rjnmm\" (UniqueName: \"kubernetes.io/projected/c2ca0d7c-7fb8-438d-9c27-09c1602339fa-kube-api-access-rjnmm\") pod \"c2ca0d7c-7fb8-438d-9c27-09c1602339fa\" (UID: \"c2ca0d7c-7fb8-438d-9c27-09c1602339fa\") " Oct 31 13:40:46.790328 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:46.790312 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03b3ba33-5c21-487d-bee4-abc978d8cf06-kube-api-access-ldhms\" (UniqueName: \"kubernetes.io/projected/03b3ba33-5c21-487d-bee4-abc978d8cf06-kube-api-access-ldhms\") pod \"03b3ba33-5c21-487d-bee4-abc978d8cf06\" (UID: \"03b3ba33-5c21-487d-bee4-abc978d8cf06\") " Oct 31 13:40:46.790401 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:46.790373 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2ca0d7c-7fb8-438d-9c27-09c1602339fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ca0d7c-7fb8-438d-9c27-09c1602339fa-plugins-config-volume\") pod \"c2ca0d7c-7fb8-438d-9c27-09c1602339fa\" (UID: \"c2ca0d7c-7fb8-438d-9c27-09c1602339fa\") " Oct 31 13:40:46.790496 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:46.790470 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03b3ba33-5c21-487d-bee4-abc978d8cf06-plugins-config-volume" pod "03b3ba33-5c21-487d-bee4-abc978d8cf06" (UID: "03b3ba33-5c21-487d-bee4-abc978d8cf06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:40:46.791019 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:46.790986 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2ca0d7c-7fb8-438d-9c27-09c1602339fa-plugins-config-volume" pod "c2ca0d7c-7fb8-438d-9c27-09c1602339fa" (UID: "c2ca0d7c-7fb8-438d-9c27-09c1602339fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:40:46.791019 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:46.790996 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03b3ba33-5c21-487d-bee4-abc978d8cf06-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:46.793026 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:46.793003 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2ca0d7c-7fb8-438d-9c27-09c1602339fa-kube-api-access-rjnmm" pod "c2ca0d7c-7fb8-438d-9c27-09c1602339fa" (UID: "c2ca0d7c-7fb8-438d-9c27-09c1602339fa"). InnerVolumeSpecName "kube-api-access-rjnmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:40:46.794394 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:46.794373 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03b3ba33-5c21-487d-bee4-abc978d8cf06-kube-api-access-ldhms" pod "03b3ba33-5c21-487d-bee4-abc978d8cf06" (UID: "03b3ba33-5c21-487d-bee4-abc978d8cf06"). InnerVolumeSpecName "kube-api-access-ldhms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:40:46.892838 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:46.892803 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjnmm\" (UniqueName: \"kubernetes.io/projected/c2ca0d7c-7fb8-438d-9c27-09c1602339fa-kube-api-access-rjnmm\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:46.892838 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:46.892833 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldhms\" (UniqueName: \"kubernetes.io/projected/03b3ba33-5c21-487d-bee4-abc978d8cf06-kube-api-access-ldhms\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:46.892838 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:46.892842 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ca0d7c-7fb8-438d-9c27-09c1602339fa-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:47.534502 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:47.534475 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-377f60-c59666d89-b6xbr" Oct 31 13:40:47.534958 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:47.534510 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-377f60-c59666d89-b6xbr" event={"ID":"03b3ba33-5c21-487d-bee4-abc978d8cf06","Type":"ContainerDied","Data":"2e79eeec393f37db8469ae00b8c5846db44617f045a2dd0d68518b11dc39221e"} Oct 31 13:40:47.534958 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:47.534573 5693 scope.go:122] "RemoveContainer" containerID="5fe013ac611b437e3e7d5c90ba1c3cb96e945b7e27e66c10d0f697611e0fccd9" Oct 31 13:40:47.536294 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:47.536186 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-377f60-bdf4746c-r7hng" event={"ID":"c2ca0d7c-7fb8-438d-9c27-09c1602339fa","Type":"ContainerDied","Data":"5eafc09a521875e0eb7b99f892e3846a4a272695d25f30a18b50538ab374c113"} Oct 31 13:40:47.536380 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:47.536328 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-377f60-bdf4746c-r7hng" Oct 31 13:40:47.548181 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:47.548161 5693 scope.go:122] "RemoveContainer" containerID="03e53c48bb964e273baa0e73e65451c78201e1bb16ff6f30c355bb861cff99d1" Oct 31 13:40:47.562521 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:47.562497 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-377f60-bdf4746c-r7hng"] Oct 31 13:40:47.571823 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:47.571798 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-377f60-bdf4746c-r7hng"] Oct 31 13:40:47.612912 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:47.612877 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-377f60-c59666d89-b6xbr"] Oct 31 13:40:47.627256 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:47.627213 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-377f60-c59666d89-b6xbr"] Oct 31 13:40:48.689087 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:40:48.689049 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:40:49.408830 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:49.408798 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03b3ba33-5c21-487d-bee4-abc978d8cf06" path="/var/lib/kubelet/pods/03b3ba33-5c21-487d-bee4-abc978d8cf06/volumes" Oct 31 13:40:49.409282 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:49.409268 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2ca0d7c-7fb8-438d-9c27-09c1602339fa" path="/var/lib/kubelet/pods/c2ca0d7c-7fb8-438d-9c27-09c1602339fa/volumes" Oct 31 13:40:53.412566 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:40:53.412537 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl"] Oct 31 13:41:08.435423 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:41:08.435372 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" podUID="1a2abd3f-fb4f-4990-9a32-5f945a588068" containerName="main" containerID="cri-o://4db86fb4b1d8cd02ab6169d455af04822523b9903ff6ac18dfe7896623d5a0db" gracePeriod=45 Oct 31 13:41:08.686873 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:41:08.686788 5693 generic.go:356] "Generic (PLEG): container finished" podID="1a2abd3f-fb4f-4990-9a32-5f945a588068" containerID="4db86fb4b1d8cd02ab6169d455af04822523b9903ff6ac18dfe7896623d5a0db" exitCode=0 Oct 31 13:41:08.687013 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:41:08.686864 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" event={"ID":"1a2abd3f-fb4f-4990-9a32-5f945a588068","Type":"ContainerDied","Data":"4db86fb4b1d8cd02ab6169d455af04822523b9903ff6ac18dfe7896623d5a0db"} Oct 31 13:41:08.713562 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:41:08.713532 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" Oct 31 13:41:08.838926 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:41:08.838892 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a2abd3f-fb4f-4990-9a32-5f945a588068-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a2abd3f-fb4f-4990-9a32-5f945a588068-tmp-dir\") pod \"1a2abd3f-fb4f-4990-9a32-5f945a588068\" (UID: \"1a2abd3f-fb4f-4990-9a32-5f945a588068\") " Oct 31 13:41:08.839102 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:41:08.839026 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a2abd3f-fb4f-4990-9a32-5f945a588068-home\" (UniqueName: \"kubernetes.io/empty-dir/1a2abd3f-fb4f-4990-9a32-5f945a588068-home\") pod \"1a2abd3f-fb4f-4990-9a32-5f945a588068\" (UID: \"1a2abd3f-fb4f-4990-9a32-5f945a588068\") " Oct 31 13:41:08.839102 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:41:08.839075 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a2abd3f-fb4f-4990-9a32-5f945a588068-tls-certs\" (UniqueName: \"kubernetes.io/secret/1a2abd3f-fb4f-4990-9a32-5f945a588068-tls-certs\") pod \"1a2abd3f-fb4f-4990-9a32-5f945a588068\" (UID: \"1a2abd3f-fb4f-4990-9a32-5f945a588068\") " Oct 31 13:41:08.839102 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:41:08.839099 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a2abd3f-fb4f-4990-9a32-5f945a588068-kube-api-access-tlpkl\" (UniqueName: \"kubernetes.io/projected/1a2abd3f-fb4f-4990-9a32-5f945a588068-kube-api-access-tlpkl\") pod \"1a2abd3f-fb4f-4990-9a32-5f945a588068\" (UID: \"1a2abd3f-fb4f-4990-9a32-5f945a588068\") " Oct 31 13:41:08.839216 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:41:08.839183 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a2abd3f-fb4f-4990-9a32-5f945a588068-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a2abd3f-fb4f-4990-9a32-5f945a588068-dshm\") pod \"1a2abd3f-fb4f-4990-9a32-5f945a588068\" (UID: \"1a2abd3f-fb4f-4990-9a32-5f945a588068\") " Oct 31 13:41:08.839216 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:41:08.839204 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a2abd3f-fb4f-4990-9a32-5f945a588068-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a2abd3f-fb4f-4990-9a32-5f945a588068-model-cache\") pod \"1a2abd3f-fb4f-4990-9a32-5f945a588068\" (UID: \"1a2abd3f-fb4f-4990-9a32-5f945a588068\") " Oct 31 13:41:08.839871 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:41:08.839850 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a2abd3f-fb4f-4990-9a32-5f945a588068-model-cache" pod "1a2abd3f-fb4f-4990-9a32-5f945a588068" (UID: "1a2abd3f-fb4f-4990-9a32-5f945a588068"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:08.840058 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:41:08.840039 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a2abd3f-fb4f-4990-9a32-5f945a588068-tmp-dir" pod "1a2abd3f-fb4f-4990-9a32-5f945a588068" (UID: "1a2abd3f-fb4f-4990-9a32-5f945a588068"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:08.840238 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:41:08.840218 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a2abd3f-fb4f-4990-9a32-5f945a588068-home" pod "1a2abd3f-fb4f-4990-9a32-5f945a588068" (UID: "1a2abd3f-fb4f-4990-9a32-5f945a588068"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:08.853062 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:41:08.853024 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a2abd3f-fb4f-4990-9a32-5f945a588068-tls-certs" pod "1a2abd3f-fb4f-4990-9a32-5f945a588068" (UID: "1a2abd3f-fb4f-4990-9a32-5f945a588068"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:41:08.853225 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:41:08.853097 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a2abd3f-fb4f-4990-9a32-5f945a588068-kube-api-access-tlpkl" pod "1a2abd3f-fb4f-4990-9a32-5f945a588068" (UID: "1a2abd3f-fb4f-4990-9a32-5f945a588068"). InnerVolumeSpecName "kube-api-access-tlpkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:41:08.854524 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:41:08.854501 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a2abd3f-fb4f-4990-9a32-5f945a588068-dshm" pod "1a2abd3f-fb4f-4990-9a32-5f945a588068" (UID: "1a2abd3f-fb4f-4990-9a32-5f945a588068"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:08.942574 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:41:08.942492 5693 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a2abd3f-fb4f-4990-9a32-5f945a588068-tls-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:08.944626 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:41:08.942587 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlpkl\" (UniqueName: \"kubernetes.io/projected/1a2abd3f-fb4f-4990-9a32-5f945a588068-kube-api-access-tlpkl\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:08.944909 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:41:08.944890 5693 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a2abd3f-fb4f-4990-9a32-5f945a588068-dshm\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:08.945018 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:41:08.945006 5693 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a2abd3f-fb4f-4990-9a32-5f945a588068-model-cache\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:08.945115 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:41:08.945105 5693 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a2abd3f-fb4f-4990-9a32-5f945a588068-tmp-dir\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:08.945384 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:41:08.945368 5693 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a2abd3f-fb4f-4990-9a32-5f945a588068-home\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:09.695113 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:41:09.695021 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" Oct 31 13:41:09.695113 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:41:09.695063 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl" event={"ID":"1a2abd3f-fb4f-4990-9a32-5f945a588068","Type":"ContainerDied","Data":"73a8f41938d4846d3b1aad1cd1f57425280b4c04fffbd073539bb4ca986dfa27"} Oct 31 13:41:09.695634 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:41:09.695120 5693 scope.go:122] "RemoveContainer" containerID="4db86fb4b1d8cd02ab6169d455af04822523b9903ff6ac18dfe7896623d5a0db" Oct 31 13:41:09.717835 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:41:09.717803 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl"] Oct 31 13:41:09.720918 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:41:09.720895 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-107955/premium-w0-107955-kserve-69b87d495-dckzl"] Oct 31 13:41:11.410518 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:41:11.410482 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a2abd3f-fb4f-4990-9a32-5f945a588068" path="/var/lib/kubelet/pods/1a2abd3f-fb4f-4990-9a32-5f945a588068/volumes" Oct 31 13:41:18.688576 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:41:18.688543 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:41:47.899260 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:41:47.899167 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0c1fbb-5dcd8b5b65-qlkdw"] Oct 31 13:41:47.920572 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:41:47.920268 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0c1fbb-9c8cfbcf-rthxp"] Oct 31 13:41:48.689590 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:41:48.689552 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:41:52.899836 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:41:52.899776 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-0c1fbb-5dcd8b5b65-qlkdw" podUID="d850f787-64ff-4a6b-8534-a93329e45929" containerName="payload-processing" containerID="cri-o://01a2320f95ffe6947a8e5d1cd2f648c812669cb213ce097c006d8ec15ca75b75" gracePeriod=25 Oct 31 13:41:52.927599 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:41:52.927523 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0c1fbb-9c8cfbcf-rthxp" podUID="e87aa817-1a1d-4267-bd3f-0d74c9a229fd" containerName="payload-pre-processing" containerID="cri-o://c2be1995485ae123473c10352ae83e6fb2869e33672901d14f29b19ed462f727" gracePeriod=25 Oct 31 13:41:55.232762 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:41:55.232719 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0c1fbb-9c8cfbcf-rthxp" podUID="e87aa817-1a1d-4267-bd3f-0d74c9a229fd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:41:55.233246 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:41:55.232764 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-0c1fbb-5dcd8b5b65-qlkdw" podUID="d850f787-64ff-4a6b-8534-a93329e45929" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:42:00.440538 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:00.440499 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk"] Oct 31 13:42:00.441370 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:00.441338 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk" podUID="942b3c70-0134-49ee-a30e-9631e318b001" containerName="maas-api" containerID="cri-o://1e0a99efad4594b50aa60785891c4ad1619afd1e689bc391f51045476d718275" gracePeriod=30 Oct 31 13:42:00.873050 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:00.873024 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk" Oct 31 13:42:00.990200 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:00.990162 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/942b3c70-0134-49ee-a30e-9631e318b001-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/942b3c70-0134-49ee-a30e-9631e318b001-openshift-service-ca\") pod \"942b3c70-0134-49ee-a30e-9631e318b001\" (UID: \"942b3c70-0134-49ee-a30e-9631e318b001\") " Oct 31 13:42:00.990386 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:00.990219 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/942b3c70-0134-49ee-a30e-9631e318b001-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/942b3c70-0134-49ee-a30e-9631e318b001-maas-api-tls\") pod \"942b3c70-0134-49ee-a30e-9631e318b001\" (UID: \"942b3c70-0134-49ee-a30e-9631e318b001\") " Oct 31 13:42:00.990386 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:00.990267 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/942b3c70-0134-49ee-a30e-9631e318b001-kube-api-access-4txgn\" (UniqueName: \"kubernetes.io/projected/942b3c70-0134-49ee-a30e-9631e318b001-kube-api-access-4txgn\") pod \"942b3c70-0134-49ee-a30e-9631e318b001\" (UID: \"942b3c70-0134-49ee-a30e-9631e318b001\") " Oct 31 13:42:00.990386 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:00.990353 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/942b3c70-0134-49ee-a30e-9631e318b001-metrics-certs\" (UniqueName: \"kubernetes.io/secret/942b3c70-0134-49ee-a30e-9631e318b001-metrics-certs\") pod \"942b3c70-0134-49ee-a30e-9631e318b001\" (UID: \"942b3c70-0134-49ee-a30e-9631e318b001\") " Oct 31 13:42:00.990629 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:00.990601 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/942b3c70-0134-49ee-a30e-9631e318b001-openshift-service-ca" pod "942b3c70-0134-49ee-a30e-9631e318b001" (UID: "942b3c70-0134-49ee-a30e-9631e318b001"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:42:00.990944 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:00.990927 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/942b3c70-0134-49ee-a30e-9631e318b001-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:00.992907 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:00.992880 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/942b3c70-0134-49ee-a30e-9631e318b001-metrics-certs" pod "942b3c70-0134-49ee-a30e-9631e318b001" (UID: "942b3c70-0134-49ee-a30e-9631e318b001"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:42:00.993069 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:00.993041 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/942b3c70-0134-49ee-a30e-9631e318b001-maas-api-tls" pod "942b3c70-0134-49ee-a30e-9631e318b001" (UID: "942b3c70-0134-49ee-a30e-9631e318b001"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:42:00.993215 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:00.993193 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/942b3c70-0134-49ee-a30e-9631e318b001-kube-api-access-4txgn" pod "942b3c70-0134-49ee-a30e-9631e318b001" (UID: "942b3c70-0134-49ee-a30e-9631e318b001"). InnerVolumeSpecName "kube-api-access-4txgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:42:01.091228 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:01.091134 5693 generic.go:356] "Generic (PLEG): container finished" podID="942b3c70-0134-49ee-a30e-9631e318b001" containerID="1e0a99efad4594b50aa60785891c4ad1619afd1e689bc391f51045476d718275" exitCode=0 Oct 31 13:42:01.091228 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:01.091188 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk" Oct 31 13:42:01.091228 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:01.091200 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk" event={"ID":"942b3c70-0134-49ee-a30e-9631e318b001","Type":"ContainerDied","Data":"1e0a99efad4594b50aa60785891c4ad1619afd1e689bc391f51045476d718275"} Oct 31 13:42:01.091582 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:01.091251 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk" event={"ID":"942b3c70-0134-49ee-a30e-9631e318b001","Type":"ContainerDied","Data":"56c725c5e8d6a654a80215c1ca6e9fed078215ae1babe8a44d51efaa70feab5a"} Oct 31 13:42:01.091582 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:01.091280 5693 scope.go:122] "RemoveContainer" containerID="1e0a99efad4594b50aa60785891c4ad1619afd1e689bc391f51045476d718275" Oct 31 13:42:01.092168 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:01.092139 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/942b3c70-0134-49ee-a30e-9631e318b001-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:01.092293 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:01.092172 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4txgn\" (UniqueName: \"kubernetes.io/projected/942b3c70-0134-49ee-a30e-9631e318b001-kube-api-access-4txgn\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:01.092293 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:01.092187 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/942b3c70-0134-49ee-a30e-9631e318b001-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:01.103060 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:01.103038 5693 scope.go:122] "RemoveContainer" containerID="1e0a99efad4594b50aa60785891c4ad1619afd1e689bc391f51045476d718275" Oct 31 13:42:01.103343 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:42:01.103323 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e0a99efad4594b50aa60785891c4ad1619afd1e689bc391f51045476d718275\": container with ID starting with 1e0a99efad4594b50aa60785891c4ad1619afd1e689bc391f51045476d718275 not found: ID does not exist" containerID="1e0a99efad4594b50aa60785891c4ad1619afd1e689bc391f51045476d718275" Oct 31 13:42:01.103405 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:01.103353 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e0a99efad4594b50aa60785891c4ad1619afd1e689bc391f51045476d718275"} err="failed to get container status \"1e0a99efad4594b50aa60785891c4ad1619afd1e689bc391f51045476d718275\": rpc error: code = NotFound desc = could not find container \"1e0a99efad4594b50aa60785891c4ad1619afd1e689bc391f51045476d718275\": container with ID starting with 1e0a99efad4594b50aa60785891c4ad1619afd1e689bc391f51045476d718275 not found: ID does not exist" Oct 31 13:42:01.116611 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:01.116581 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk"] Oct 31 13:42:01.121612 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:01.121582 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0c1fbb-585d857f65-nb7qk"] Oct 31 13:42:01.424774 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:01.424410 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="942b3c70-0134-49ee-a30e-9631e318b001" path="/var/lib/kubelet/pods/942b3c70-0134-49ee-a30e-9631e318b001/volumes" Oct 31 13:42:05.233079 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:05.233039 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-0c1fbb-5dcd8b5b65-qlkdw" podUID="d850f787-64ff-4a6b-8534-a93329e45929" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:42:05.233500 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:05.233044 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0c1fbb-9c8cfbcf-rthxp" podUID="e87aa817-1a1d-4267-bd3f-0d74c9a229fd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:42:12.198900 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:12.198859 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-6cef79-8695b89999-mg5vs"] Oct 31 13:42:12.211762 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:12.211733 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6cef79-7dd4f54867-sqjfq"] Oct 31 13:42:13.186338 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:13.186309 5693 generic.go:356] "Generic (PLEG): container finished" podID="d850f787-64ff-4a6b-8534-a93329e45929" containerID="01a2320f95ffe6947a8e5d1cd2f648c812669cb213ce097c006d8ec15ca75b75" exitCode=0 Oct 31 13:42:13.186483 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:13.186398 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0c1fbb-5dcd8b5b65-qlkdw" event={"ID":"d850f787-64ff-4a6b-8534-a93329e45929","Type":"ContainerDied","Data":"01a2320f95ffe6947a8e5d1cd2f648c812669cb213ce097c006d8ec15ca75b75"} Oct 31 13:42:13.188569 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:13.188418 5693 generic.go:356] "Generic (PLEG): container finished" podID="e87aa817-1a1d-4267-bd3f-0d74c9a229fd" containerID="c2be1995485ae123473c10352ae83e6fb2869e33672901d14f29b19ed462f727" exitCode=0 Oct 31 13:42:13.188569 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:13.188486 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0c1fbb-9c8cfbcf-rthxp" event={"ID":"e87aa817-1a1d-4267-bd3f-0d74c9a229fd","Type":"ContainerDied","Data":"c2be1995485ae123473c10352ae83e6fb2869e33672901d14f29b19ed462f727"} Oct 31 13:42:13.245929 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:13.245900 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0c1fbb-5dcd8b5b65-qlkdw" Oct 31 13:42:13.296999 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:13.296971 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d850f787-64ff-4a6b-8534-a93329e45929-kube-api-access-k6kgz\" (UniqueName: \"kubernetes.io/projected/d850f787-64ff-4a6b-8534-a93329e45929-kube-api-access-k6kgz\") pod \"d850f787-64ff-4a6b-8534-a93329e45929\" (UID: \"d850f787-64ff-4a6b-8534-a93329e45929\") " Oct 31 13:42:13.297139 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:13.297102 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d850f787-64ff-4a6b-8534-a93329e45929-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d850f787-64ff-4a6b-8534-a93329e45929-plugins-config-volume\") pod \"d850f787-64ff-4a6b-8534-a93329e45929\" (UID: \"d850f787-64ff-4a6b-8534-a93329e45929\") " Oct 31 13:42:13.297523 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:13.297489 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d850f787-64ff-4a6b-8534-a93329e45929-plugins-config-volume" pod "d850f787-64ff-4a6b-8534-a93329e45929" (UID: "d850f787-64ff-4a6b-8534-a93329e45929"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:42:13.297691 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:13.297672 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d850f787-64ff-4a6b-8534-a93329e45929-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:13.299717 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:13.299692 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d850f787-64ff-4a6b-8534-a93329e45929-kube-api-access-k6kgz" pod "d850f787-64ff-4a6b-8534-a93329e45929" (UID: "d850f787-64ff-4a6b-8534-a93329e45929"). InnerVolumeSpecName "kube-api-access-k6kgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:42:13.307188 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:13.307151 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0c1fbb-9c8cfbcf-rthxp" Oct 31 13:42:13.399074 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:13.399044 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e87aa817-1a1d-4267-bd3f-0d74c9a229fd-kube-api-access-4v6zn\" (UniqueName: \"kubernetes.io/projected/e87aa817-1a1d-4267-bd3f-0d74c9a229fd-kube-api-access-4v6zn\") pod \"e87aa817-1a1d-4267-bd3f-0d74c9a229fd\" (UID: \"e87aa817-1a1d-4267-bd3f-0d74c9a229fd\") " Oct 31 13:42:13.399261 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:13.399249 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e87aa817-1a1d-4267-bd3f-0d74c9a229fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e87aa817-1a1d-4267-bd3f-0d74c9a229fd-plugins-config-volume\") pod \"e87aa817-1a1d-4267-bd3f-0d74c9a229fd\" (UID: \"e87aa817-1a1d-4267-bd3f-0d74c9a229fd\") " Oct 31 13:42:13.399580 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:13.399568 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6kgz\" (UniqueName: \"kubernetes.io/projected/d850f787-64ff-4a6b-8534-a93329e45929-kube-api-access-k6kgz\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:13.399653 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:13.399603 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e87aa817-1a1d-4267-bd3f-0d74c9a229fd-plugins-config-volume" pod "e87aa817-1a1d-4267-bd3f-0d74c9a229fd" (UID: "e87aa817-1a1d-4267-bd3f-0d74c9a229fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:42:13.401366 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:13.401338 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e87aa817-1a1d-4267-bd3f-0d74c9a229fd-kube-api-access-4v6zn" pod "e87aa817-1a1d-4267-bd3f-0d74c9a229fd" (UID: "e87aa817-1a1d-4267-bd3f-0d74c9a229fd"). InnerVolumeSpecName "kube-api-access-4v6zn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:42:13.501467 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:13.501372 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e87aa817-1a1d-4267-bd3f-0d74c9a229fd-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:13.501467 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:13.501410 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v6zn\" (UniqueName: \"kubernetes.io/projected/e87aa817-1a1d-4267-bd3f-0d74c9a229fd-kube-api-access-4v6zn\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:14.196702 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:14.196604 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0c1fbb-5dcd8b5b65-qlkdw" Oct 31 13:42:14.196702 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:14.196628 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0c1fbb-5dcd8b5b65-qlkdw" event={"ID":"d850f787-64ff-4a6b-8534-a93329e45929","Type":"ContainerDied","Data":"362f4e46983cb41893a13ece9f090c2c8829f93096574d9bdc8b66671ec65408"} Oct 31 13:42:14.196937 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:14.196724 5693 scope.go:122] "RemoveContainer" containerID="01a2320f95ffe6947a8e5d1cd2f648c812669cb213ce097c006d8ec15ca75b75" Oct 31 13:42:14.198032 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:14.198015 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0c1fbb-9c8cfbcf-rthxp" Oct 31 13:42:14.198137 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:14.198013 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0c1fbb-9c8cfbcf-rthxp" event={"ID":"e87aa817-1a1d-4267-bd3f-0d74c9a229fd","Type":"ContainerDied","Data":"0f6430ced51562c3e48d24ab65006fadf944bf14c23fe6195acca0ccbb89cf1d"} Oct 31 13:42:14.207886 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:14.207867 5693 scope.go:122] "RemoveContainer" containerID="c2be1995485ae123473c10352ae83e6fb2869e33672901d14f29b19ed462f727" Oct 31 13:42:14.267875 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:14.267838 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0c1fbb-5dcd8b5b65-qlkdw"] Oct 31 13:42:14.282213 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:14.282175 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0c1fbb-5dcd8b5b65-qlkdw"] Oct 31 13:42:14.318158 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:14.318116 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0c1fbb-9c8cfbcf-rthxp"] Oct 31 13:42:14.323374 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:14.323346 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0c1fbb-9c8cfbcf-rthxp"] Oct 31 13:42:15.409833 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:15.409796 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d850f787-64ff-4a6b-8534-a93329e45929" path="/var/lib/kubelet/pods/d850f787-64ff-4a6b-8534-a93329e45929/volumes" Oct 31 13:42:15.410288 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:15.410270 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e87aa817-1a1d-4267-bd3f-0d74c9a229fd" path="/var/lib/kubelet/pods/e87aa817-1a1d-4267-bd3f-0d74c9a229fd/volumes" Oct 31 13:42:17.200974 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:17.200898 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6cef79-8695b89999-mg5vs" podUID="6b617953-8a53-49e1-bc81-80d029d098bd" containerName="payload-processing" containerID="cri-o://5c36c64c851e9a50c669aefdef3888b830343a27b5d2ebffb565e3e324836f76" gracePeriod=25 Oct 31 13:42:17.217061 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:17.217020 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6cef79-7dd4f54867-sqjfq" podUID="d6d2c321-1fae-4360-b6fa-c11cf95fece6" containerName="payload-pre-processing" containerID="cri-o://824060e06c4e5f30dcdfeab243410f4264962620f57cffa4a0a0b5bf274082c0" gracePeriod=25 Oct 31 13:42:18.692948 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:42:18.692911 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:42:24.042040 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:24.042002 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6cef79-7dd4f54867-sqjfq" podUID="d6d2c321-1fae-4360-b6fa-c11cf95fece6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:42:25.051994 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:25.051948 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6cef79-8695b89999-mg5vs" podUID="6b617953-8a53-49e1-bc81-80d029d098bd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:42:30.119734 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:30.119699 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-1e72f6/rate-test-model-1e72f6-kserve-8454cfd857-tbqhz"] Oct 31 13:42:30.416191 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:30.416153 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c24e1-54b97985f9-zvhx8"] Oct 31 13:42:34.042116 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:34.042075 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6cef79-7dd4f54867-sqjfq" podUID="d6d2c321-1fae-4360-b6fa-c11cf95fece6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:42:35.052388 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:35.052351 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6cef79-8695b89999-mg5vs" podUID="6b617953-8a53-49e1-bc81-80d029d098bd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:42:35.416979 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:35.416938 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c24e1-54b97985f9-zvhx8" podUID="a0257070-74ef-4b97-acd3-ad2a93fe5342" containerName="payload-pre-processing" containerID="cri-o://87db9712426c9e787e206a0af3a7433e4d691edb162369a0a5e477234d35335b" gracePeriod=25 Oct 31 13:42:37.385339 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:37.385301 5693 generic.go:356] "Generic (PLEG): container finished" podID="6b617953-8a53-49e1-bc81-80d029d098bd" containerID="5c36c64c851e9a50c669aefdef3888b830343a27b5d2ebffb565e3e324836f76" exitCode=0 Oct 31 13:42:37.385813 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:37.385366 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6cef79-8695b89999-mg5vs" event={"ID":"6b617953-8a53-49e1-bc81-80d029d098bd","Type":"ContainerDied","Data":"5c36c64c851e9a50c669aefdef3888b830343a27b5d2ebffb565e3e324836f76"} Oct 31 13:42:37.387503 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:37.387471 5693 generic.go:356] "Generic (PLEG): container finished" podID="d6d2c321-1fae-4360-b6fa-c11cf95fece6" containerID="824060e06c4e5f30dcdfeab243410f4264962620f57cffa4a0a0b5bf274082c0" exitCode=0 Oct 31 13:42:37.387663 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:37.387635 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6cef79-7dd4f54867-sqjfq" event={"ID":"d6d2c321-1fae-4360-b6fa-c11cf95fece6","Type":"ContainerDied","Data":"824060e06c4e5f30dcdfeab243410f4264962620f57cffa4a0a0b5bf274082c0"} Oct 31 13:42:37.510593 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:37.510566 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6cef79-8695b89999-mg5vs" Oct 31 13:42:37.567229 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:37.567205 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6cef79-7dd4f54867-sqjfq" Oct 31 13:42:37.647924 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:37.647886 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b617953-8a53-49e1-bc81-80d029d098bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b617953-8a53-49e1-bc81-80d029d098bd-plugins-config-volume\") pod \"6b617953-8a53-49e1-bc81-80d029d098bd\" (UID: \"6b617953-8a53-49e1-bc81-80d029d098bd\") " Oct 31 13:42:37.648147 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:37.648126 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b617953-8a53-49e1-bc81-80d029d098bd-kube-api-access-zl4f8\" (UniqueName: \"kubernetes.io/projected/6b617953-8a53-49e1-bc81-80d029d098bd-kube-api-access-zl4f8\") pod \"6b617953-8a53-49e1-bc81-80d029d098bd\" (UID: \"6b617953-8a53-49e1-bc81-80d029d098bd\") " Oct 31 13:42:37.651774 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:37.650137 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b617953-8a53-49e1-bc81-80d029d098bd-plugins-config-volume" pod "6b617953-8a53-49e1-bc81-80d029d098bd" (UID: "6b617953-8a53-49e1-bc81-80d029d098bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:42:37.659011 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:37.658965 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b617953-8a53-49e1-bc81-80d029d098bd-kube-api-access-zl4f8" pod "6b617953-8a53-49e1-bc81-80d029d098bd" (UID: "6b617953-8a53-49e1-bc81-80d029d098bd"). InnerVolumeSpecName "kube-api-access-zl4f8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:42:37.750086 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:37.750012 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6d2c321-1fae-4360-b6fa-c11cf95fece6-kube-api-access-k2rz4\" (UniqueName: \"kubernetes.io/projected/d6d2c321-1fae-4360-b6fa-c11cf95fece6-kube-api-access-k2rz4\") pod \"d6d2c321-1fae-4360-b6fa-c11cf95fece6\" (UID: \"d6d2c321-1fae-4360-b6fa-c11cf95fece6\") " Oct 31 13:42:37.750267 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:37.750219 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6d2c321-1fae-4360-b6fa-c11cf95fece6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6d2c321-1fae-4360-b6fa-c11cf95fece6-plugins-config-volume\") pod \"d6d2c321-1fae-4360-b6fa-c11cf95fece6\" (UID: \"d6d2c321-1fae-4360-b6fa-c11cf95fece6\") " Oct 31 13:42:37.750706 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:37.750676 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6d2c321-1fae-4360-b6fa-c11cf95fece6-plugins-config-volume" pod "d6d2c321-1fae-4360-b6fa-c11cf95fece6" (UID: "d6d2c321-1fae-4360-b6fa-c11cf95fece6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:42:37.750844 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:37.750801 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6d2c321-1fae-4360-b6fa-c11cf95fece6-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:37.750844 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:37.750815 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b617953-8a53-49e1-bc81-80d029d098bd-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:37.750844 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:37.750824 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl4f8\" (UniqueName: \"kubernetes.io/projected/6b617953-8a53-49e1-bc81-80d029d098bd-kube-api-access-zl4f8\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:37.752442 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:37.752413 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6d2c321-1fae-4360-b6fa-c11cf95fece6-kube-api-access-k2rz4" pod "d6d2c321-1fae-4360-b6fa-c11cf95fece6" (UID: "d6d2c321-1fae-4360-b6fa-c11cf95fece6"). InnerVolumeSpecName "kube-api-access-k2rz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:42:37.852054 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:37.852018 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2rz4\" (UniqueName: \"kubernetes.io/projected/d6d2c321-1fae-4360-b6fa-c11cf95fece6-kube-api-access-k2rz4\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:38.395005 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:38.394977 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6cef79-8695b89999-mg5vs" Oct 31 13:42:38.395493 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:38.394976 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6cef79-8695b89999-mg5vs" event={"ID":"6b617953-8a53-49e1-bc81-80d029d098bd","Type":"ContainerDied","Data":"550bbfb6110f3244dfbc62798290ee5ff2cd8a3e3984ed35095e753bc425c6cf"} Oct 31 13:42:38.395493 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:38.395133 5693 scope.go:122] "RemoveContainer" containerID="5c36c64c851e9a50c669aefdef3888b830343a27b5d2ebffb565e3e324836f76" Oct 31 13:42:38.396844 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:38.396624 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6cef79-7dd4f54867-sqjfq" Oct 31 13:42:38.396844 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:38.396799 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6cef79-7dd4f54867-sqjfq" event={"ID":"d6d2c321-1fae-4360-b6fa-c11cf95fece6","Type":"ContainerDied","Data":"bc51ffc9c6e272f706d5e3fe70a991cdd2c3671713fef6a519e4bf81f9af995a"} Oct 31 13:42:38.407552 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:38.407528 5693 scope.go:122] "RemoveContainer" containerID="824060e06c4e5f30dcdfeab243410f4264962620f57cffa4a0a0b5bf274082c0" Oct 31 13:42:38.437026 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:38.436993 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-6cef79-8695b89999-mg5vs"] Oct 31 13:42:38.442082 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:38.442055 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-6cef79-8695b89999-mg5vs"] Oct 31 13:42:38.455617 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:38.455586 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6cef79-7dd4f54867-sqjfq"] Oct 31 13:42:38.465300 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:38.465277 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6cef79-7dd4f54867-sqjfq"] Oct 31 13:42:39.411767 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:39.411733 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b617953-8a53-49e1-bc81-80d029d098bd" path="/var/lib/kubelet/pods/6b617953-8a53-49e1-bc81-80d029d098bd/volumes" Oct 31 13:42:39.412444 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:39.412421 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6d2c321-1fae-4360-b6fa-c11cf95fece6" path="/var/lib/kubelet/pods/d6d2c321-1fae-4360-b6fa-c11cf95fece6/volumes" Oct 31 13:42:41.518779 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:41.518744 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c24e1-54b97985f9-zvhx8" podUID="a0257070-74ef-4b97-acd3-ad2a93fe5342" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:42:45.149963 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:45.149921 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-1e72f6/rate-test-model-1e72f6-kserve-8454cfd857-tbqhz" podUID="3cc7b994-8cee-47e2-9a6d-b587e3ac1eef" containerName="main" containerID="cri-o://f997a307b48686a73d703cf54fe1995c13494529a4bb63c6868f269d6e39dbce" gracePeriod=45 Oct 31 13:42:45.435683 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:45.435659 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-1e72f6/rate-test-model-1e72f6-kserve-8454cfd857-tbqhz" Oct 31 13:42:45.449439 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:45.449409 5693 generic.go:356] "Generic (PLEG): container finished" podID="3cc7b994-8cee-47e2-9a6d-b587e3ac1eef" containerID="f997a307b48686a73d703cf54fe1995c13494529a4bb63c6868f269d6e39dbce" exitCode=0 Oct 31 13:42:45.449590 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:45.449453 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-1e72f6/rate-test-model-1e72f6-kserve-8454cfd857-tbqhz" Oct 31 13:42:45.449590 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:45.449499 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-1e72f6/rate-test-model-1e72f6-kserve-8454cfd857-tbqhz" event={"ID":"3cc7b994-8cee-47e2-9a6d-b587e3ac1eef","Type":"ContainerDied","Data":"f997a307b48686a73d703cf54fe1995c13494529a4bb63c6868f269d6e39dbce"} Oct 31 13:42:45.449590 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:45.449551 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-1e72f6/rate-test-model-1e72f6-kserve-8454cfd857-tbqhz" event={"ID":"3cc7b994-8cee-47e2-9a6d-b587e3ac1eef","Type":"ContainerDied","Data":"8c182f62fea885448bc80335ae18ba6949168dd1632c1d51387cb1f8e33c25b0"} Oct 31 13:42:45.449590 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:45.449572 5693 scope.go:122] "RemoveContainer" containerID="f997a307b48686a73d703cf54fe1995c13494529a4bb63c6868f269d6e39dbce" Oct 31 13:42:45.459249 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:45.459230 5693 scope.go:122] "RemoveContainer" containerID="f997a307b48686a73d703cf54fe1995c13494529a4bb63c6868f269d6e39dbce" Oct 31 13:42:45.459547 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:42:45.459526 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f997a307b48686a73d703cf54fe1995c13494529a4bb63c6868f269d6e39dbce\": container with ID starting with f997a307b48686a73d703cf54fe1995c13494529a4bb63c6868f269d6e39dbce not found: ID does not exist" containerID="f997a307b48686a73d703cf54fe1995c13494529a4bb63c6868f269d6e39dbce" Oct 31 13:42:45.459592 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:45.459555 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f997a307b48686a73d703cf54fe1995c13494529a4bb63c6868f269d6e39dbce"} err="failed to get container status \"f997a307b48686a73d703cf54fe1995c13494529a4bb63c6868f269d6e39dbce\": rpc error: code = NotFound desc = could not find container \"f997a307b48686a73d703cf54fe1995c13494529a4bb63c6868f269d6e39dbce\": container with ID starting with f997a307b48686a73d703cf54fe1995c13494529a4bb63c6868f269d6e39dbce not found: ID does not exist" Oct 31 13:42:45.525014 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:45.524976 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-kube-api-access-mpj7l\" (UniqueName: \"kubernetes.io/projected/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-kube-api-access-mpj7l\") pod \"3cc7b994-8cee-47e2-9a6d-b587e3ac1eef\" (UID: \"3cc7b994-8cee-47e2-9a6d-b587e3ac1eef\") " Oct 31 13:42:45.525211 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:45.525076 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-tmp-dir\") pod \"3cc7b994-8cee-47e2-9a6d-b587e3ac1eef\" (UID: \"3cc7b994-8cee-47e2-9a6d-b587e3ac1eef\") " Oct 31 13:42:45.525211 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:45.525129 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-home\" (UniqueName: \"kubernetes.io/empty-dir/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-home\") pod \"3cc7b994-8cee-47e2-9a6d-b587e3ac1eef\" (UID: \"3cc7b994-8cee-47e2-9a6d-b587e3ac1eef\") " Oct 31 13:42:45.525349 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:45.525215 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-dshm\") pod \"3cc7b994-8cee-47e2-9a6d-b587e3ac1eef\" (UID: \"3cc7b994-8cee-47e2-9a6d-b587e3ac1eef\") " Oct 31 13:42:45.525349 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:45.525333 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-tls-certs\" (UniqueName: \"kubernetes.io/secret/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-tls-certs\") pod \"3cc7b994-8cee-47e2-9a6d-b587e3ac1eef\" (UID: \"3cc7b994-8cee-47e2-9a6d-b587e3ac1eef\") " Oct 31 13:42:45.525452 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:45.525373 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-tmp-dir" pod "3cc7b994-8cee-47e2-9a6d-b587e3ac1eef" (UID: "3cc7b994-8cee-47e2-9a6d-b587e3ac1eef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:45.525452 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:45.525417 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-home" pod "3cc7b994-8cee-47e2-9a6d-b587e3ac1eef" (UID: "3cc7b994-8cee-47e2-9a6d-b587e3ac1eef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:45.525452 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:45.525442 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-model-cache\") pod \"3cc7b994-8cee-47e2-9a6d-b587e3ac1eef\" (UID: \"3cc7b994-8cee-47e2-9a6d-b587e3ac1eef\") " Oct 31 13:42:45.525921 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:45.525881 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-model-cache" pod "3cc7b994-8cee-47e2-9a6d-b587e3ac1eef" (UID: "3cc7b994-8cee-47e2-9a6d-b587e3ac1eef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:45.526197 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:45.526014 5693 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-home\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:45.526197 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:45.526030 5693 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-model-cache\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:45.526197 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:45.526044 5693 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-tmp-dir\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:45.527690 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:45.527664 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-dshm" pod "3cc7b994-8cee-47e2-9a6d-b587e3ac1eef" (UID: "3cc7b994-8cee-47e2-9a6d-b587e3ac1eef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:45.527804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:45.527712 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-kube-api-access-mpj7l" pod "3cc7b994-8cee-47e2-9a6d-b587e3ac1eef" (UID: "3cc7b994-8cee-47e2-9a6d-b587e3ac1eef"). InnerVolumeSpecName "kube-api-access-mpj7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:42:45.528018 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:45.527999 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-tls-certs" pod "3cc7b994-8cee-47e2-9a6d-b587e3ac1eef" (UID: "3cc7b994-8cee-47e2-9a6d-b587e3ac1eef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:42:45.627196 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:45.627160 5693 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-tls-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:45.627196 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:45.627199 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpj7l\" (UniqueName: \"kubernetes.io/projected/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-kube-api-access-mpj7l\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:45.627394 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:45.627214 5693 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef-dshm\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:45.808863 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:45.808830 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-1e72f6/rate-test-model-1e72f6-kserve-8454cfd857-tbqhz"] Oct 31 13:42:45.864284 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:45.864248 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-1e72f6/rate-test-model-1e72f6-kserve-8454cfd857-tbqhz"] Oct 31 13:42:47.420368 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:47.418141 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cc7b994-8cee-47e2-9a6d-b587e3ac1eef" path="/var/lib/kubelet/pods/3cc7b994-8cee-47e2-9a6d-b587e3ac1eef/volumes" Oct 31 13:42:48.687014 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:42:48.686978 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:42:51.519490 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:51.519449 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c24e1-54b97985f9-zvhx8" podUID="a0257070-74ef-4b97-acd3-ad2a93fe5342" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:42:52.645033 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:52.644998 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v"] Oct 31 13:42:54.744946 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:54.744908 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5dc570-5cfd7479bc-csb46"] Oct 31 13:42:55.596107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:55.595521 5693 generic.go:356] "Generic (PLEG): container finished" podID="a0257070-74ef-4b97-acd3-ad2a93fe5342" containerID="87db9712426c9e787e206a0af3a7433e4d691edb162369a0a5e477234d35335b" exitCode=0 Oct 31 13:42:55.596107 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:55.595582 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c24e1-54b97985f9-zvhx8" event={"ID":"a0257070-74ef-4b97-acd3-ad2a93fe5342","Type":"ContainerDied","Data":"87db9712426c9e787e206a0af3a7433e4d691edb162369a0a5e477234d35335b"} Oct 31 13:42:55.735489 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:55.735453 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c24e1-54b97985f9-zvhx8" Oct 31 13:42:55.862066 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:55.862033 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0257070-74ef-4b97-acd3-ad2a93fe5342-kube-api-access-6v5wj\" (UniqueName: \"kubernetes.io/projected/a0257070-74ef-4b97-acd3-ad2a93fe5342-kube-api-access-6v5wj\") pod \"a0257070-74ef-4b97-acd3-ad2a93fe5342\" (UID: \"a0257070-74ef-4b97-acd3-ad2a93fe5342\") " Oct 31 13:42:55.862486 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:55.862164 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0257070-74ef-4b97-acd3-ad2a93fe5342-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0257070-74ef-4b97-acd3-ad2a93fe5342-plugins-config-volume\") pod \"a0257070-74ef-4b97-acd3-ad2a93fe5342\" (UID: \"a0257070-74ef-4b97-acd3-ad2a93fe5342\") " Oct 31 13:42:55.862544 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:55.862517 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0257070-74ef-4b97-acd3-ad2a93fe5342-plugins-config-volume" pod "a0257070-74ef-4b97-acd3-ad2a93fe5342" (UID: "a0257070-74ef-4b97-acd3-ad2a93fe5342"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:42:55.864317 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:55.864295 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0257070-74ef-4b97-acd3-ad2a93fe5342-kube-api-access-6v5wj" pod "a0257070-74ef-4b97-acd3-ad2a93fe5342" (UID: "a0257070-74ef-4b97-acd3-ad2a93fe5342"). InnerVolumeSpecName "kube-api-access-6v5wj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:42:55.964409 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:55.964322 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v5wj\" (UniqueName: \"kubernetes.io/projected/a0257070-74ef-4b97-acd3-ad2a93fe5342-kube-api-access-6v5wj\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:55.964409 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:55.964354 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0257070-74ef-4b97-acd3-ad2a93fe5342-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:56.603609 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:56.603582 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c24e1-54b97985f9-zvhx8" Oct 31 13:42:56.603877 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:56.603609 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c24e1-54b97985f9-zvhx8" event={"ID":"a0257070-74ef-4b97-acd3-ad2a93fe5342","Type":"ContainerDied","Data":"23c42fc43d308dd861e3012cb09185ab408fa34eaee0222d3d3f8924befbb78a"} Oct 31 13:42:56.603877 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:56.603707 5693 scope.go:122] "RemoveContainer" containerID="87db9712426c9e787e206a0af3a7433e4d691edb162369a0a5e477234d35335b" Oct 31 13:42:56.628466 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:56.628441 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c24e1-54b97985f9-zvhx8"] Oct 31 13:42:56.635769 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:56.635738 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c24e1-54b97985f9-zvhx8"] Oct 31 13:42:57.412483 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:57.412450 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0257070-74ef-4b97-acd3-ad2a93fe5342" path="/var/lib/kubelet/pods/a0257070-74ef-4b97-acd3-ad2a93fe5342/volumes" Oct 31 13:42:59.746070 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:42:59.746027 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5dc570-5cfd7479bc-csb46" podUID="1ad184ee-c710-47b0-9d44-b4bbcb5487e5" containerName="payload-pre-processing" containerID="cri-o://f82d62c9bb362858d5fad6d0f1ff1e6b390ddd1614f0483aa73b95bda97ed116" gracePeriod=25 Oct 31 13:43:00.061179 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:00.061144 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5dc570-5cfd7479bc-csb46" Oct 31 13:43:00.127779 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:00.127746 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ad184ee-c710-47b0-9d44-b4bbcb5487e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ad184ee-c710-47b0-9d44-b4bbcb5487e5-plugins-config-volume\") pod \"1ad184ee-c710-47b0-9d44-b4bbcb5487e5\" (UID: \"1ad184ee-c710-47b0-9d44-b4bbcb5487e5\") " Oct 31 13:43:00.127979 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:00.127857 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ad184ee-c710-47b0-9d44-b4bbcb5487e5-kube-api-access-ks8kq\" (UniqueName: \"kubernetes.io/projected/1ad184ee-c710-47b0-9d44-b4bbcb5487e5-kube-api-access-ks8kq\") pod \"1ad184ee-c710-47b0-9d44-b4bbcb5487e5\" (UID: \"1ad184ee-c710-47b0-9d44-b4bbcb5487e5\") " Oct 31 13:43:00.128276 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:00.128248 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ad184ee-c710-47b0-9d44-b4bbcb5487e5-plugins-config-volume" pod "1ad184ee-c710-47b0-9d44-b4bbcb5487e5" (UID: "1ad184ee-c710-47b0-9d44-b4bbcb5487e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:43:00.128500 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:00.128482 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ad184ee-c710-47b0-9d44-b4bbcb5487e5-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:00.130243 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:00.130217 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ad184ee-c710-47b0-9d44-b4bbcb5487e5-kube-api-access-ks8kq" pod "1ad184ee-c710-47b0-9d44-b4bbcb5487e5" (UID: "1ad184ee-c710-47b0-9d44-b4bbcb5487e5"). InnerVolumeSpecName "kube-api-access-ks8kq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:00.229479 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:00.229447 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks8kq\" (UniqueName: \"kubernetes.io/projected/1ad184ee-c710-47b0-9d44-b4bbcb5487e5-kube-api-access-ks8kq\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:00.634079 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:00.634045 5693 generic.go:356] "Generic (PLEG): container finished" podID="1ad184ee-c710-47b0-9d44-b4bbcb5487e5" containerID="f82d62c9bb362858d5fad6d0f1ff1e6b390ddd1614f0483aa73b95bda97ed116" exitCode=0 Oct 31 13:43:00.634263 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:00.634098 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5dc570-5cfd7479bc-csb46" Oct 31 13:43:00.634263 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:00.634136 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5dc570-5cfd7479bc-csb46" event={"ID":"1ad184ee-c710-47b0-9d44-b4bbcb5487e5","Type":"ContainerDied","Data":"f82d62c9bb362858d5fad6d0f1ff1e6b390ddd1614f0483aa73b95bda97ed116"} Oct 31 13:43:00.634263 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:00.634194 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5dc570-5cfd7479bc-csb46" event={"ID":"1ad184ee-c710-47b0-9d44-b4bbcb5487e5","Type":"ContainerDied","Data":"7e9aa40fa9606a611cee6afb8907f61a89e741f826bf41e5eeace828d3252821"} Oct 31 13:43:00.634263 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:00.634223 5693 scope.go:122] "RemoveContainer" containerID="f82d62c9bb362858d5fad6d0f1ff1e6b390ddd1614f0483aa73b95bda97ed116" Oct 31 13:43:00.645113 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:00.645093 5693 scope.go:122] "RemoveContainer" containerID="f82d62c9bb362858d5fad6d0f1ff1e6b390ddd1614f0483aa73b95bda97ed116" Oct 31 13:43:00.645393 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:43:00.645375 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f82d62c9bb362858d5fad6d0f1ff1e6b390ddd1614f0483aa73b95bda97ed116\": container with ID starting with f82d62c9bb362858d5fad6d0f1ff1e6b390ddd1614f0483aa73b95bda97ed116 not found: ID does not exist" containerID="f82d62c9bb362858d5fad6d0f1ff1e6b390ddd1614f0483aa73b95bda97ed116" Oct 31 13:43:00.645443 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:00.645402 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f82d62c9bb362858d5fad6d0f1ff1e6b390ddd1614f0483aa73b95bda97ed116"} err="failed to get container status \"f82d62c9bb362858d5fad6d0f1ff1e6b390ddd1614f0483aa73b95bda97ed116\": rpc error: code = NotFound desc = could not find container \"f82d62c9bb362858d5fad6d0f1ff1e6b390ddd1614f0483aa73b95bda97ed116\": container with ID starting with f82d62c9bb362858d5fad6d0f1ff1e6b390ddd1614f0483aa73b95bda97ed116 not found: ID does not exist" Oct 31 13:43:00.666117 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:00.666089 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5dc570-5cfd7479bc-csb46"] Oct 31 13:43:00.674454 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:00.674431 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5dc570-5cfd7479bc-csb46"] Oct 31 13:43:01.409954 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:01.409924 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ad184ee-c710-47b0-9d44-b4bbcb5487e5" path="/var/lib/kubelet/pods/1ad184ee-c710-47b0-9d44-b4bbcb5487e5/volumes" Oct 31 13:43:07.669721 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:07.669632 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" podUID="17d56bfd-8cb1-42bf-bb11-05810c46d469" containerName="main" containerID="cri-o://c92189754374164d4d3e5651ed1a6eef96c07caf0f02c06c73ec6b035beeb4fa" gracePeriod=45 Oct 31 13:43:07.832301 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:07.832130 5693 generic.go:356] "Generic (PLEG): container finished" podID="17d56bfd-8cb1-42bf-bb11-05810c46d469" containerID="c92189754374164d4d3e5651ed1a6eef96c07caf0f02c06c73ec6b035beeb4fa" exitCode=0 Oct 31 13:43:07.832301 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:07.832262 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" event={"ID":"17d56bfd-8cb1-42bf-bb11-05810c46d469","Type":"ContainerDied","Data":"c92189754374164d4d3e5651ed1a6eef96c07caf0f02c06c73ec6b035beeb4fa"} Oct 31 13:43:07.988296 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:07.988271 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" Oct 31 13:43:08.089801 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:08.089762 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17d56bfd-8cb1-42bf-bb11-05810c46d469-dshm\" (UniqueName: \"kubernetes.io/empty-dir/17d56bfd-8cb1-42bf-bb11-05810c46d469-dshm\") pod \"17d56bfd-8cb1-42bf-bb11-05810c46d469\" (UID: \"17d56bfd-8cb1-42bf-bb11-05810c46d469\") " Oct 31 13:43:08.090002 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:08.089835 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17d56bfd-8cb1-42bf-bb11-05810c46d469-kube-api-access-tf8xh\" (UniqueName: \"kubernetes.io/projected/17d56bfd-8cb1-42bf-bb11-05810c46d469-kube-api-access-tf8xh\") pod \"17d56bfd-8cb1-42bf-bb11-05810c46d469\" (UID: \"17d56bfd-8cb1-42bf-bb11-05810c46d469\") " Oct 31 13:43:08.090002 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:08.089873 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17d56bfd-8cb1-42bf-bb11-05810c46d469-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17d56bfd-8cb1-42bf-bb11-05810c46d469-model-cache\") pod \"17d56bfd-8cb1-42bf-bb11-05810c46d469\" (UID: \"17d56bfd-8cb1-42bf-bb11-05810c46d469\") " Oct 31 13:43:08.090002 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:08.089959 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17d56bfd-8cb1-42bf-bb11-05810c46d469-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17d56bfd-8cb1-42bf-bb11-05810c46d469-tmp-dir\") pod \"17d56bfd-8cb1-42bf-bb11-05810c46d469\" (UID: \"17d56bfd-8cb1-42bf-bb11-05810c46d469\") " Oct 31 13:43:08.090268 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:08.090214 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17d56bfd-8cb1-42bf-bb11-05810c46d469-model-cache" pod "17d56bfd-8cb1-42bf-bb11-05810c46d469" (UID: "17d56bfd-8cb1-42bf-bb11-05810c46d469"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:08.090268 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:08.090265 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17d56bfd-8cb1-42bf-bb11-05810c46d469-tls-certs\" (UniqueName: \"kubernetes.io/secret/17d56bfd-8cb1-42bf-bb11-05810c46d469-tls-certs\") pod \"17d56bfd-8cb1-42bf-bb11-05810c46d469\" (UID: \"17d56bfd-8cb1-42bf-bb11-05810c46d469\") " Oct 31 13:43:08.090492 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:08.090269 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17d56bfd-8cb1-42bf-bb11-05810c46d469-tmp-dir" pod "17d56bfd-8cb1-42bf-bb11-05810c46d469" (UID: "17d56bfd-8cb1-42bf-bb11-05810c46d469"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:08.090492 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:08.090314 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17d56bfd-8cb1-42bf-bb11-05810c46d469-home\" (UniqueName: \"kubernetes.io/empty-dir/17d56bfd-8cb1-42bf-bb11-05810c46d469-home\") pod \"17d56bfd-8cb1-42bf-bb11-05810c46d469\" (UID: \"17d56bfd-8cb1-42bf-bb11-05810c46d469\") " Oct 31 13:43:08.090778 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:08.090627 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17d56bfd-8cb1-42bf-bb11-05810c46d469-home" pod "17d56bfd-8cb1-42bf-bb11-05810c46d469" (UID: "17d56bfd-8cb1-42bf-bb11-05810c46d469"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:08.091678 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:08.091607 5693 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17d56bfd-8cb1-42bf-bb11-05810c46d469-model-cache\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:08.091678 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:08.091667 5693 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17d56bfd-8cb1-42bf-bb11-05810c46d469-tmp-dir\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:08.091875 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:08.091685 5693 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17d56bfd-8cb1-42bf-bb11-05810c46d469-home\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:08.092584 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:08.092559 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17d56bfd-8cb1-42bf-bb11-05810c46d469-dshm" pod "17d56bfd-8cb1-42bf-bb11-05810c46d469" (UID: "17d56bfd-8cb1-42bf-bb11-05810c46d469"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:08.092584 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:08.092571 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17d56bfd-8cb1-42bf-bb11-05810c46d469-kube-api-access-tf8xh" pod "17d56bfd-8cb1-42bf-bb11-05810c46d469" (UID: "17d56bfd-8cb1-42bf-bb11-05810c46d469"). InnerVolumeSpecName "kube-api-access-tf8xh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:08.092807 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:08.092789 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17d56bfd-8cb1-42bf-bb11-05810c46d469-tls-certs" pod "17d56bfd-8cb1-42bf-bb11-05810c46d469" (UID: "17d56bfd-8cb1-42bf-bb11-05810c46d469"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:43:08.193754 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:08.193721 5693 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17d56bfd-8cb1-42bf-bb11-05810c46d469-dshm\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:08.193754 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:08.193758 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf8xh\" (UniqueName: \"kubernetes.io/projected/17d56bfd-8cb1-42bf-bb11-05810c46d469-kube-api-access-tf8xh\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:08.193947 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:08.193774 5693 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17d56bfd-8cb1-42bf-bb11-05810c46d469-tls-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:08.839996 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:08.839927 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" Oct 31 13:43:08.839996 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:08.839962 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v" event={"ID":"17d56bfd-8cb1-42bf-bb11-05810c46d469","Type":"ContainerDied","Data":"ee19eecabf6fabe7c5cf1ca0c320eb2a462208460a4464cdb3fb489cc8adfa01"} Oct 31 13:43:08.840521 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:08.840012 5693 scope.go:122] "RemoveContainer" containerID="c92189754374164d4d3e5651ed1a6eef96c07caf0f02c06c73ec6b035beeb4fa" Oct 31 13:43:08.870552 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:08.870527 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v"] Oct 31 13:43:08.873659 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:08.873613 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-622e03/simulator-w1-622e03-kserve-55b6694ffb-84l5v"] Oct 31 13:43:09.409070 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:09.409037 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17d56bfd-8cb1-42bf-bb11-05810c46d469" path="/var/lib/kubelet/pods/17d56bfd-8cb1-42bf-bb11-05810c46d469/volumes" Oct 31 13:43:11.020025 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:11.019989 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4574ec/rate-test-model-4574ec-kserve-66686cb98f-lzbn9"] Oct 31 13:43:18.688599 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:43:18.688558 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:43:22.746987 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:22.746948 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c2c98b-755946465f-m2g26"] Oct 31 13:43:24.188751 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:24.188716 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5"] Oct 31 13:43:24.189808 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:24.189481 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5" podUID="32747ebc-c185-4a7f-877a-f4903862b567" containerName="maas-api" containerID="cri-o://410c4bc509c278aea61f47a4cbe1f74f3eb6fb486ed2946dca12a281d108101c" gracePeriod=30 Oct 31 13:43:24.577427 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:24.577399 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5" Oct 31 13:43:24.602728 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:24.602562 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32747ebc-c185-4a7f-877a-f4903862b567-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32747ebc-c185-4a7f-877a-f4903862b567-maas-api-tls\") pod \"32747ebc-c185-4a7f-877a-f4903862b567\" (UID: \"32747ebc-c185-4a7f-877a-f4903862b567\") " Oct 31 13:43:24.602728 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:24.602717 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32747ebc-c185-4a7f-877a-f4903862b567-kube-api-access-vfcw4\" (UniqueName: \"kubernetes.io/projected/32747ebc-c185-4a7f-877a-f4903862b567-kube-api-access-vfcw4\") pod \"32747ebc-c185-4a7f-877a-f4903862b567\" (UID: \"32747ebc-c185-4a7f-877a-f4903862b567\") " Oct 31 13:43:24.602968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:24.602794 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32747ebc-c185-4a7f-877a-f4903862b567-metrics-certs\" (UniqueName: \"kubernetes.io/secret/32747ebc-c185-4a7f-877a-f4903862b567-metrics-certs\") pod \"32747ebc-c185-4a7f-877a-f4903862b567\" (UID: \"32747ebc-c185-4a7f-877a-f4903862b567\") " Oct 31 13:43:24.602968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:24.602929 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32747ebc-c185-4a7f-877a-f4903862b567-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32747ebc-c185-4a7f-877a-f4903862b567-openshift-service-ca\") pod \"32747ebc-c185-4a7f-877a-f4903862b567\" (UID: \"32747ebc-c185-4a7f-877a-f4903862b567\") " Oct 31 13:43:24.603460 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:24.603434 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32747ebc-c185-4a7f-877a-f4903862b567-openshift-service-ca" pod "32747ebc-c185-4a7f-877a-f4903862b567" (UID: "32747ebc-c185-4a7f-877a-f4903862b567"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:43:24.611745 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:24.606630 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32747ebc-c185-4a7f-877a-f4903862b567-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:24.611745 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:24.608825 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32747ebc-c185-4a7f-877a-f4903862b567-maas-api-tls" pod "32747ebc-c185-4a7f-877a-f4903862b567" (UID: "32747ebc-c185-4a7f-877a-f4903862b567"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:43:24.626363 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:24.626324 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32747ebc-c185-4a7f-877a-f4903862b567-kube-api-access-vfcw4" pod "32747ebc-c185-4a7f-877a-f4903862b567" (UID: "32747ebc-c185-4a7f-877a-f4903862b567"). InnerVolumeSpecName "kube-api-access-vfcw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:24.629821 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:24.629792 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32747ebc-c185-4a7f-877a-f4903862b567-metrics-certs" pod "32747ebc-c185-4a7f-877a-f4903862b567" (UID: "32747ebc-c185-4a7f-877a-f4903862b567"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:43:24.707723 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:24.707617 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32747ebc-c185-4a7f-877a-f4903862b567-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:24.707723 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:24.707696 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfcw4\" (UniqueName: \"kubernetes.io/projected/32747ebc-c185-4a7f-877a-f4903862b567-kube-api-access-vfcw4\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:24.707957 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:24.707739 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32747ebc-c185-4a7f-877a-f4903862b567-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:24.993842 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:24.993810 5693 generic.go:356] "Generic (PLEG): container finished" podID="32747ebc-c185-4a7f-877a-f4903862b567" containerID="410c4bc509c278aea61f47a4cbe1f74f3eb6fb486ed2946dca12a281d108101c" exitCode=0 Oct 31 13:43:24.994032 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:24.993873 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5" Oct 31 13:43:24.994032 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:24.993903 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5" event={"ID":"32747ebc-c185-4a7f-877a-f4903862b567","Type":"ContainerDied","Data":"410c4bc509c278aea61f47a4cbe1f74f3eb6fb486ed2946dca12a281d108101c"} Oct 31 13:43:24.994032 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:24.993993 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5" event={"ID":"32747ebc-c185-4a7f-877a-f4903862b567","Type":"ContainerDied","Data":"42ac5a60c5321a2f90a9bf8e3c4deff4a946e7310bc54159ecb20a0b2ab6be85"} Oct 31 13:43:24.994032 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:24.994021 5693 scope.go:122] "RemoveContainer" containerID="410c4bc509c278aea61f47a4cbe1f74f3eb6fb486ed2946dca12a281d108101c" Oct 31 13:43:25.035659 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:25.035299 5693 scope.go:122] "RemoveContainer" containerID="410c4bc509c278aea61f47a4cbe1f74f3eb6fb486ed2946dca12a281d108101c" Oct 31 13:43:25.036124 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:43:25.036100 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"410c4bc509c278aea61f47a4cbe1f74f3eb6fb486ed2946dca12a281d108101c\": container with ID starting with 410c4bc509c278aea61f47a4cbe1f74f3eb6fb486ed2946dca12a281d108101c not found: ID does not exist" containerID="410c4bc509c278aea61f47a4cbe1f74f3eb6fb486ed2946dca12a281d108101c" Oct 31 13:43:25.036227 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:25.036138 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"410c4bc509c278aea61f47a4cbe1f74f3eb6fb486ed2946dca12a281d108101c"} err="failed to get container status \"410c4bc509c278aea61f47a4cbe1f74f3eb6fb486ed2946dca12a281d108101c\": rpc error: code = NotFound desc = could not find container \"410c4bc509c278aea61f47a4cbe1f74f3eb6fb486ed2946dca12a281d108101c\": container with ID starting with 410c4bc509c278aea61f47a4cbe1f74f3eb6fb486ed2946dca12a281d108101c not found: ID does not exist" Oct 31 13:43:25.051189 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:25.051163 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5"] Oct 31 13:43:25.060427 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:25.060402 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c2c98b-9f789db96-5x8r5"] Oct 31 13:43:25.411152 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:25.411120 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32747ebc-c185-4a7f-877a-f4903862b567" path="/var/lib/kubelet/pods/32747ebc-c185-4a7f-877a-f4903862b567/volumes" Oct 31 13:43:26.047604 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:26.047566 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-4574ec/rate-test-model-4574ec-kserve-66686cb98f-lzbn9" podUID="0191a8b7-5eee-4437-aa2b-c228014966cd" containerName="main" containerID="cri-o://cc217b88d77ea8ab3c6694459e996d77c2813de173dc680a30469ded26cffa4e" gracePeriod=45 Oct 31 13:43:26.325811 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:26.325788 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4574ec/rate-test-model-4574ec-kserve-66686cb98f-lzbn9" Oct 31 13:43:26.445472 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:26.445437 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0191a8b7-5eee-4437-aa2b-c228014966cd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0191a8b7-5eee-4437-aa2b-c228014966cd-model-cache\") pod \"0191a8b7-5eee-4437-aa2b-c228014966cd\" (UID: \"0191a8b7-5eee-4437-aa2b-c228014966cd\") " Oct 31 13:43:26.445988 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:26.445500 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0191a8b7-5eee-4437-aa2b-c228014966cd-home\" (UniqueName: \"kubernetes.io/empty-dir/0191a8b7-5eee-4437-aa2b-c228014966cd-home\") pod \"0191a8b7-5eee-4437-aa2b-c228014966cd\" (UID: \"0191a8b7-5eee-4437-aa2b-c228014966cd\") " Oct 31 13:43:26.445988 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:26.445555 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0191a8b7-5eee-4437-aa2b-c228014966cd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0191a8b7-5eee-4437-aa2b-c228014966cd-tmp-dir\") pod \"0191a8b7-5eee-4437-aa2b-c228014966cd\" (UID: \"0191a8b7-5eee-4437-aa2b-c228014966cd\") " Oct 31 13:43:26.445988 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:26.445589 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0191a8b7-5eee-4437-aa2b-c228014966cd-kube-api-access-lqhrj\" (UniqueName: \"kubernetes.io/projected/0191a8b7-5eee-4437-aa2b-c228014966cd-kube-api-access-lqhrj\") pod \"0191a8b7-5eee-4437-aa2b-c228014966cd\" (UID: \"0191a8b7-5eee-4437-aa2b-c228014966cd\") " Oct 31 13:43:26.445988 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:26.445698 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0191a8b7-5eee-4437-aa2b-c228014966cd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0191a8b7-5eee-4437-aa2b-c228014966cd-dshm\") pod \"0191a8b7-5eee-4437-aa2b-c228014966cd\" (UID: \"0191a8b7-5eee-4437-aa2b-c228014966cd\") " Oct 31 13:43:26.445988 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:26.445731 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0191a8b7-5eee-4437-aa2b-c228014966cd-tls-certs\" (UniqueName: \"kubernetes.io/secret/0191a8b7-5eee-4437-aa2b-c228014966cd-tls-certs\") pod \"0191a8b7-5eee-4437-aa2b-c228014966cd\" (UID: \"0191a8b7-5eee-4437-aa2b-c228014966cd\") " Oct 31 13:43:26.445988 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:26.445782 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0191a8b7-5eee-4437-aa2b-c228014966cd-model-cache" pod "0191a8b7-5eee-4437-aa2b-c228014966cd" (UID: "0191a8b7-5eee-4437-aa2b-c228014966cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:26.445988 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:26.445871 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0191a8b7-5eee-4437-aa2b-c228014966cd-home" pod "0191a8b7-5eee-4437-aa2b-c228014966cd" (UID: "0191a8b7-5eee-4437-aa2b-c228014966cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:26.445988 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:26.445886 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0191a8b7-5eee-4437-aa2b-c228014966cd-tmp-dir" pod "0191a8b7-5eee-4437-aa2b-c228014966cd" (UID: "0191a8b7-5eee-4437-aa2b-c228014966cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:26.446489 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:26.446467 5693 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0191a8b7-5eee-4437-aa2b-c228014966cd-model-cache\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:26.446572 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:26.446526 5693 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0191a8b7-5eee-4437-aa2b-c228014966cd-home\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:26.446635 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:26.446584 5693 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0191a8b7-5eee-4437-aa2b-c228014966cd-tmp-dir\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:26.448186 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:26.448163 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0191a8b7-5eee-4437-aa2b-c228014966cd-kube-api-access-lqhrj" pod "0191a8b7-5eee-4437-aa2b-c228014966cd" (UID: "0191a8b7-5eee-4437-aa2b-c228014966cd"). InnerVolumeSpecName "kube-api-access-lqhrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:26.448310 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:26.448212 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0191a8b7-5eee-4437-aa2b-c228014966cd-dshm" pod "0191a8b7-5eee-4437-aa2b-c228014966cd" (UID: "0191a8b7-5eee-4437-aa2b-c228014966cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:26.448454 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:26.448440 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0191a8b7-5eee-4437-aa2b-c228014966cd-tls-certs" pod "0191a8b7-5eee-4437-aa2b-c228014966cd" (UID: "0191a8b7-5eee-4437-aa2b-c228014966cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:43:26.548438 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:26.548399 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqhrj\" (UniqueName: \"kubernetes.io/projected/0191a8b7-5eee-4437-aa2b-c228014966cd-kube-api-access-lqhrj\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:26.548438 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:26.548435 5693 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0191a8b7-5eee-4437-aa2b-c228014966cd-dshm\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:26.548664 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:26.548449 5693 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0191a8b7-5eee-4437-aa2b-c228014966cd-tls-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:27.010472 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:27.010436 5693 generic.go:356] "Generic (PLEG): container finished" podID="0191a8b7-5eee-4437-aa2b-c228014966cd" containerID="cc217b88d77ea8ab3c6694459e996d77c2813de173dc680a30469ded26cffa4e" exitCode=0 Oct 31 13:43:27.010694 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:27.010488 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4574ec/rate-test-model-4574ec-kserve-66686cb98f-lzbn9" Oct 31 13:43:27.010694 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:27.010525 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4574ec/rate-test-model-4574ec-kserve-66686cb98f-lzbn9" event={"ID":"0191a8b7-5eee-4437-aa2b-c228014966cd","Type":"ContainerDied","Data":"cc217b88d77ea8ab3c6694459e996d77c2813de173dc680a30469ded26cffa4e"} Oct 31 13:43:27.010694 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:27.010571 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4574ec/rate-test-model-4574ec-kserve-66686cb98f-lzbn9" event={"ID":"0191a8b7-5eee-4437-aa2b-c228014966cd","Type":"ContainerDied","Data":"9a8978728ba77cce6c1bc785ee86d1c802e46360f3ad0e65806e068d5a98d685"} Oct 31 13:43:27.010694 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:27.010590 5693 scope.go:122] "RemoveContainer" containerID="cc217b88d77ea8ab3c6694459e996d77c2813de173dc680a30469ded26cffa4e" Oct 31 13:43:27.022000 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:27.021984 5693 scope.go:122] "RemoveContainer" containerID="cc217b88d77ea8ab3c6694459e996d77c2813de173dc680a30469ded26cffa4e" Oct 31 13:43:27.023365 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:43:27.023345 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc217b88d77ea8ab3c6694459e996d77c2813de173dc680a30469ded26cffa4e\": container with ID starting with cc217b88d77ea8ab3c6694459e996d77c2813de173dc680a30469ded26cffa4e not found: ID does not exist" containerID="cc217b88d77ea8ab3c6694459e996d77c2813de173dc680a30469ded26cffa4e" Oct 31 13:43:27.023436 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:27.023371 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc217b88d77ea8ab3c6694459e996d77c2813de173dc680a30469ded26cffa4e"} err="failed to get container status \"cc217b88d77ea8ab3c6694459e996d77c2813de173dc680a30469ded26cffa4e\": rpc error: code = NotFound desc = could not find container \"cc217b88d77ea8ab3c6694459e996d77c2813de173dc680a30469ded26cffa4e\": container with ID starting with cc217b88d77ea8ab3c6694459e996d77c2813de173dc680a30469ded26cffa4e not found: ID does not exist" Oct 31 13:43:27.039625 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:27.039495 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4574ec/rate-test-model-4574ec-kserve-66686cb98f-lzbn9"] Oct 31 13:43:27.042267 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:27.042246 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4574ec/rate-test-model-4574ec-kserve-66686cb98f-lzbn9"] Oct 31 13:43:27.410621 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:27.410588 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0191a8b7-5eee-4437-aa2b-c228014966cd" path="/var/lib/kubelet/pods/0191a8b7-5eee-4437-aa2b-c228014966cd/volumes" Oct 31 13:43:27.747575 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:27.747480 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-c2c98b-755946465f-m2g26" podUID="08846306-817b-4ccf-8ec3-470e1bba6257" containerName="payload-processing" containerID="cri-o://f020ac917a90d181f8441089aa3cad7751032e9cb1a8d171d22dec9b16de1536" gracePeriod=25 Oct 31 13:43:32.366485 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:32.366400 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-c2c98b-755946465f-m2g26" podUID="08846306-817b-4ccf-8ec3-470e1bba6257" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:43:36.968258 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:36.968219 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw"] Oct 31 13:43:36.969143 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:36.969106 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" podUID="9fe38c78-fe1a-4633-bbc7-696b2e517ea8" containerName="istio-proxy" containerID="cri-o://948fe94efd57ef9f19e41753577a8bdcb0e5b07434eba7dcdd9a7783097fe9ae" gracePeriod=30 Oct 31 13:43:42.132675 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:42.132624 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-45bcad-77bfcf8bc9-pk9mj"] Oct 31 13:43:42.366897 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:42.366863 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-c2c98b-755946465f-m2g26" podUID="08846306-817b-4ccf-8ec3-470e1bba6257" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:43:42.707186 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:42.707163 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" Oct 31 13:43:42.739033 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:42.738985 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-workload-certs\") pod \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\" (UID: \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\") " Oct 31 13:43:42.739230 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:42.739073 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-workload-socket\") pod \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\" (UID: \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\") " Oct 31 13:43:42.739230 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:42.739121 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-istio-podinfo\") pod \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\" (UID: \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\") " Oct 31 13:43:42.739230 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:42.739151 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-credential-socket\") pod \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\" (UID: \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\") " Oct 31 13:43:42.739474 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:42.739237 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-kube-api-access-ssjmj\" (UniqueName: \"kubernetes.io/projected/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-kube-api-access-ssjmj\") pod \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\" (UID: \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\") " Oct 31 13:43:42.739474 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:42.739311 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-istiod-ca-cert\") pod \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\" (UID: \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\") " Oct 31 13:43:42.739474 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:42.739455 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-istio-data\") pod \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\" (UID: \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\") " Oct 31 13:43:42.739705 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:42.739522 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-istio-envoy\") pod \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\" (UID: \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\") " Oct 31 13:43:42.739705 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:42.739575 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-istio-token\" (UniqueName: \"kubernetes.io/projected/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-istio-token\") pod \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\" (UID: \"9fe38c78-fe1a-4633-bbc7-696b2e517ea8\") " Oct 31 13:43:42.739847 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:42.739775 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-credential-socket" pod "9fe38c78-fe1a-4633-bbc7-696b2e517ea8" (UID: "9fe38c78-fe1a-4633-bbc7-696b2e517ea8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:42.739953 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:42.739925 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-istio-data" pod "9fe38c78-fe1a-4633-bbc7-696b2e517ea8" (UID: "9fe38c78-fe1a-4633-bbc7-696b2e517ea8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:42.740289 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:42.740263 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-workload-certs" pod "9fe38c78-fe1a-4633-bbc7-696b2e517ea8" (UID: "9fe38c78-fe1a-4633-bbc7-696b2e517ea8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:42.740393 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:42.740370 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-istiod-ca-cert" pod "9fe38c78-fe1a-4633-bbc7-696b2e517ea8" (UID: "9fe38c78-fe1a-4633-bbc7-696b2e517ea8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:43:42.740460 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:42.740190 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:42.740460 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:42.740420 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:42.743176 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:42.743134 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-workload-socket" pod "9fe38c78-fe1a-4633-bbc7-696b2e517ea8" (UID: "9fe38c78-fe1a-4633-bbc7-696b2e517ea8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:42.757536 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:42.757500 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-istio-podinfo" pod "9fe38c78-fe1a-4633-bbc7-696b2e517ea8" (UID: "9fe38c78-fe1a-4633-bbc7-696b2e517ea8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:43:42.757536 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:42.757519 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-istio-token" pod "9fe38c78-fe1a-4633-bbc7-696b2e517ea8" (UID: "9fe38c78-fe1a-4633-bbc7-696b2e517ea8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:42.757803 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:42.757667 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-istio-envoy" pod "9fe38c78-fe1a-4633-bbc7-696b2e517ea8" (UID: "9fe38c78-fe1a-4633-bbc7-696b2e517ea8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:42.757803 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:42.757690 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-kube-api-access-ssjmj" pod "9fe38c78-fe1a-4633-bbc7-696b2e517ea8" (UID: "9fe38c78-fe1a-4633-bbc7-696b2e517ea8"). InnerVolumeSpecName "kube-api-access-ssjmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:42.842121 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:42.842089 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssjmj\" (UniqueName: \"kubernetes.io/projected/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-kube-api-access-ssjmj\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:42.842121 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:42.842122 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:42.842121 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:42.842138 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:42.842121 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:42.842151 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:42.842121 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:42.842164 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:42.842121 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:42.842175 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:42.842121 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:42.842184 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fe38c78-fe1a-4633-bbc7-696b2e517ea8-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:43.031479 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:43.031394 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r"] Oct 31 13:43:43.031720 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:43.031673 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r" podUID="f31ac998-2cc1-482d-8bbe-32ec51bc5dd1" containerName="maas-api" containerID="cri-o://f0d1273800fcbdd1b2fb9664ab02d6e8e2fbf74165ff406c8b638a9da587d89a" gracePeriod=30 Oct 31 13:43:43.132810 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:43.132777 5693 generic.go:356] "Generic (PLEG): container finished" podID="9fe38c78-fe1a-4633-bbc7-696b2e517ea8" containerID="948fe94efd57ef9f19e41753577a8bdcb0e5b07434eba7dcdd9a7783097fe9ae" exitCode=0 Oct 31 13:43:43.133214 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:43.132825 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" Oct 31 13:43:43.133214 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:43.132851 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" event={"ID":"9fe38c78-fe1a-4633-bbc7-696b2e517ea8","Type":"ContainerDied","Data":"948fe94efd57ef9f19e41753577a8bdcb0e5b07434eba7dcdd9a7783097fe9ae"} Oct 31 13:43:43.133214 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:43.132904 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw" event={"ID":"9fe38c78-fe1a-4633-bbc7-696b2e517ea8","Type":"ContainerDied","Data":"6a7a4f59bbed9cc1ac9d5e95eb62068d47c56ef9f5e1a0f3518426a8315db0e1"} Oct 31 13:43:43.133214 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:43.132926 5693 scope.go:122] "RemoveContainer" containerID="948fe94efd57ef9f19e41753577a8bdcb0e5b07434eba7dcdd9a7783097fe9ae" Oct 31 13:43:43.145186 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:43.145169 5693 scope.go:122] "RemoveContainer" containerID="948fe94efd57ef9f19e41753577a8bdcb0e5b07434eba7dcdd9a7783097fe9ae" Oct 31 13:43:43.145652 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:43:43.145611 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"948fe94efd57ef9f19e41753577a8bdcb0e5b07434eba7dcdd9a7783097fe9ae\": container with ID starting with 948fe94efd57ef9f19e41753577a8bdcb0e5b07434eba7dcdd9a7783097fe9ae not found: ID does not exist" containerID="948fe94efd57ef9f19e41753577a8bdcb0e5b07434eba7dcdd9a7783097fe9ae" Oct 31 13:43:43.145740 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:43.145688 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"948fe94efd57ef9f19e41753577a8bdcb0e5b07434eba7dcdd9a7783097fe9ae"} err="failed to get container status \"948fe94efd57ef9f19e41753577a8bdcb0e5b07434eba7dcdd9a7783097fe9ae\": rpc error: code = NotFound desc = could not find container \"948fe94efd57ef9f19e41753577a8bdcb0e5b07434eba7dcdd9a7783097fe9ae\": container with ID starting with 948fe94efd57ef9f19e41753577a8bdcb0e5b07434eba7dcdd9a7783097fe9ae not found: ID does not exist" Oct 31 13:43:43.165843 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:43.165813 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw"] Oct 31 13:43:43.181020 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:43.177946 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-684kw"] Oct 31 13:43:43.285450 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:43.285388 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r" Oct 31 13:43:43.348678 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:43.348620 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f31ac998-2cc1-482d-8bbe-32ec51bc5dd1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f31ac998-2cc1-482d-8bbe-32ec51bc5dd1-openshift-service-ca\") pod \"f31ac998-2cc1-482d-8bbe-32ec51bc5dd1\" (UID: \"f31ac998-2cc1-482d-8bbe-32ec51bc5dd1\") " Oct 31 13:43:43.348867 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:43.348800 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f31ac998-2cc1-482d-8bbe-32ec51bc5dd1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f31ac998-2cc1-482d-8bbe-32ec51bc5dd1-metrics-certs\") pod \"f31ac998-2cc1-482d-8bbe-32ec51bc5dd1\" (UID: \"f31ac998-2cc1-482d-8bbe-32ec51bc5dd1\") " Oct 31 13:43:43.348954 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:43.348936 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f31ac998-2cc1-482d-8bbe-32ec51bc5dd1-kube-api-access-9c2x6\" (UniqueName: \"kubernetes.io/projected/f31ac998-2cc1-482d-8bbe-32ec51bc5dd1-kube-api-access-9c2x6\") pod \"f31ac998-2cc1-482d-8bbe-32ec51bc5dd1\" (UID: \"f31ac998-2cc1-482d-8bbe-32ec51bc5dd1\") " Oct 31 13:43:43.349022 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:43.348946 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f31ac998-2cc1-482d-8bbe-32ec51bc5dd1-openshift-service-ca" pod "f31ac998-2cc1-482d-8bbe-32ec51bc5dd1" (UID: "f31ac998-2cc1-482d-8bbe-32ec51bc5dd1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:43:43.349078 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:43.349030 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f31ac998-2cc1-482d-8bbe-32ec51bc5dd1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f31ac998-2cc1-482d-8bbe-32ec51bc5dd1-maas-api-tls\") pod \"f31ac998-2cc1-482d-8bbe-32ec51bc5dd1\" (UID: \"f31ac998-2cc1-482d-8bbe-32ec51bc5dd1\") " Oct 31 13:43:43.349593 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:43.349567 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f31ac998-2cc1-482d-8bbe-32ec51bc5dd1-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:43.351362 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:43.351340 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f31ac998-2cc1-482d-8bbe-32ec51bc5dd1-metrics-certs" pod "f31ac998-2cc1-482d-8bbe-32ec51bc5dd1" (UID: "f31ac998-2cc1-482d-8bbe-32ec51bc5dd1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:43:43.351773 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:43.351750 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f31ac998-2cc1-482d-8bbe-32ec51bc5dd1-kube-api-access-9c2x6" pod "f31ac998-2cc1-482d-8bbe-32ec51bc5dd1" (UID: "f31ac998-2cc1-482d-8bbe-32ec51bc5dd1"). InnerVolumeSpecName "kube-api-access-9c2x6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:43.351900 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:43.351795 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f31ac998-2cc1-482d-8bbe-32ec51bc5dd1-maas-api-tls" pod "f31ac998-2cc1-482d-8bbe-32ec51bc5dd1" (UID: "f31ac998-2cc1-482d-8bbe-32ec51bc5dd1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:43:43.410208 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:43.410177 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fe38c78-fe1a-4633-bbc7-696b2e517ea8" path="/var/lib/kubelet/pods/9fe38c78-fe1a-4633-bbc7-696b2e517ea8/volumes" Oct 31 13:43:43.450752 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:43.450673 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f31ac998-2cc1-482d-8bbe-32ec51bc5dd1-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:43.450752 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:43.450707 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c2x6\" (UniqueName: \"kubernetes.io/projected/f31ac998-2cc1-482d-8bbe-32ec51bc5dd1-kube-api-access-9c2x6\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:43.450752 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:43.450723 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f31ac998-2cc1-482d-8bbe-32ec51bc5dd1-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:44.142048 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:44.142018 5693 generic.go:356] "Generic (PLEG): container finished" podID="f31ac998-2cc1-482d-8bbe-32ec51bc5dd1" containerID="f0d1273800fcbdd1b2fb9664ab02d6e8e2fbf74165ff406c8b638a9da587d89a" exitCode=0 Oct 31 13:43:44.142456 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:44.142068 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r" Oct 31 13:43:44.142456 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:44.142067 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r" event={"ID":"f31ac998-2cc1-482d-8bbe-32ec51bc5dd1","Type":"ContainerDied","Data":"f0d1273800fcbdd1b2fb9664ab02d6e8e2fbf74165ff406c8b638a9da587d89a"} Oct 31 13:43:44.142456 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:44.142201 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r" event={"ID":"f31ac998-2cc1-482d-8bbe-32ec51bc5dd1","Type":"ContainerDied","Data":"4b6bddd1095e38d3146bcde82679aedddbb52e5915e10e4f94a01b6ec4255ef1"} Oct 31 13:43:44.142456 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:44.142235 5693 scope.go:122] "RemoveContainer" containerID="f0d1273800fcbdd1b2fb9664ab02d6e8e2fbf74165ff406c8b638a9da587d89a" Oct 31 13:43:44.152807 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:44.152784 5693 scope.go:122] "RemoveContainer" containerID="f0d1273800fcbdd1b2fb9664ab02d6e8e2fbf74165ff406c8b638a9da587d89a" Oct 31 13:43:44.153127 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:43:44.153106 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0d1273800fcbdd1b2fb9664ab02d6e8e2fbf74165ff406c8b638a9da587d89a\": container with ID starting with f0d1273800fcbdd1b2fb9664ab02d6e8e2fbf74165ff406c8b638a9da587d89a not found: ID does not exist" containerID="f0d1273800fcbdd1b2fb9664ab02d6e8e2fbf74165ff406c8b638a9da587d89a" Oct 31 13:43:44.153184 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:44.153136 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0d1273800fcbdd1b2fb9664ab02d6e8e2fbf74165ff406c8b638a9da587d89a"} err="failed to get container status \"f0d1273800fcbdd1b2fb9664ab02d6e8e2fbf74165ff406c8b638a9da587d89a\": rpc error: code = NotFound desc = could not find container \"f0d1273800fcbdd1b2fb9664ab02d6e8e2fbf74165ff406c8b638a9da587d89a\": container with ID starting with f0d1273800fcbdd1b2fb9664ab02d6e8e2fbf74165ff406c8b638a9da587d89a not found: ID does not exist" Oct 31 13:43:44.187056 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:44.184832 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r"] Oct 31 13:43:44.192073 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:44.192042 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r"] Oct 31 13:43:45.409897 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:45.409862 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f31ac998-2cc1-482d-8bbe-32ec51bc5dd1" path="/var/lib/kubelet/pods/f31ac998-2cc1-482d-8bbe-32ec51bc5dd1/volumes" Oct 31 13:43:46.220509 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:46.220468 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-45bcad-78958844dc-njr2r" podUID="f31ac998-2cc1-482d-8bbe-32ec51bc5dd1" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.140:8443/health\": context deadline exceeded" Oct 31 13:43:47.133799 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:47.133754 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-45bcad-77bfcf8bc9-pk9mj" podUID="a1b0fe6e-146d-4f30-90c2-d44f49d6e17f" containerName="payload-processing" containerID="cri-o://799602286a71706e495ec5e66c4f20420baefa9d2087cd03f2cc44b52694b5e5" gracePeriod=25 Oct 31 13:43:48.038860 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:48.038827 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c2c98b-755946465f-m2g26" Oct 31 13:43:48.137436 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:48.137407 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08846306-817b-4ccf-8ec3-470e1bba6257-kube-api-access-6rcjt\" (UniqueName: \"kubernetes.io/projected/08846306-817b-4ccf-8ec3-470e1bba6257-kube-api-access-6rcjt\") pod \"08846306-817b-4ccf-8ec3-470e1bba6257\" (UID: \"08846306-817b-4ccf-8ec3-470e1bba6257\") " Oct 31 13:43:48.137947 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:48.137535 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08846306-817b-4ccf-8ec3-470e1bba6257-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08846306-817b-4ccf-8ec3-470e1bba6257-plugins-config-volume\") pod \"08846306-817b-4ccf-8ec3-470e1bba6257\" (UID: \"08846306-817b-4ccf-8ec3-470e1bba6257\") " Oct 31 13:43:48.138020 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:48.137976 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08846306-817b-4ccf-8ec3-470e1bba6257-plugins-config-volume" pod "08846306-817b-4ccf-8ec3-470e1bba6257" (UID: "08846306-817b-4ccf-8ec3-470e1bba6257"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:43:48.138140 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:48.138122 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08846306-817b-4ccf-8ec3-470e1bba6257-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:48.139838 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:48.139813 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08846306-817b-4ccf-8ec3-470e1bba6257-kube-api-access-6rcjt" pod "08846306-817b-4ccf-8ec3-470e1bba6257" (UID: "08846306-817b-4ccf-8ec3-470e1bba6257"). InnerVolumeSpecName "kube-api-access-6rcjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:48.168097 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:48.168071 5693 generic.go:356] "Generic (PLEG): container finished" podID="08846306-817b-4ccf-8ec3-470e1bba6257" containerID="f020ac917a90d181f8441089aa3cad7751032e9cb1a8d171d22dec9b16de1536" exitCode=0 Oct 31 13:43:48.168240 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:48.168109 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c2c98b-755946465f-m2g26" event={"ID":"08846306-817b-4ccf-8ec3-470e1bba6257","Type":"ContainerDied","Data":"f020ac917a90d181f8441089aa3cad7751032e9cb1a8d171d22dec9b16de1536"} Oct 31 13:43:48.168240 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:48.168115 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c2c98b-755946465f-m2g26" Oct 31 13:43:48.168240 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:48.168162 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c2c98b-755946465f-m2g26" event={"ID":"08846306-817b-4ccf-8ec3-470e1bba6257","Type":"ContainerDied","Data":"1701db71ff9c1eb9384c11e47406f0846ca5886ab2b84af9131456f831f2b2e1"} Oct 31 13:43:48.168240 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:48.168192 5693 scope.go:122] "RemoveContainer" containerID="f020ac917a90d181f8441089aa3cad7751032e9cb1a8d171d22dec9b16de1536" Oct 31 13:43:48.178996 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:48.178975 5693 scope.go:122] "RemoveContainer" containerID="f020ac917a90d181f8441089aa3cad7751032e9cb1a8d171d22dec9b16de1536" Oct 31 13:43:48.179270 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:43:48.179251 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f020ac917a90d181f8441089aa3cad7751032e9cb1a8d171d22dec9b16de1536\": container with ID starting with f020ac917a90d181f8441089aa3cad7751032e9cb1a8d171d22dec9b16de1536 not found: ID does not exist" containerID="f020ac917a90d181f8441089aa3cad7751032e9cb1a8d171d22dec9b16de1536" Oct 31 13:43:48.179323 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:48.179306 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f020ac917a90d181f8441089aa3cad7751032e9cb1a8d171d22dec9b16de1536"} err="failed to get container status \"f020ac917a90d181f8441089aa3cad7751032e9cb1a8d171d22dec9b16de1536\": rpc error: code = NotFound desc = could not find container \"f020ac917a90d181f8441089aa3cad7751032e9cb1a8d171d22dec9b16de1536\": container with ID starting with f020ac917a90d181f8441089aa3cad7751032e9cb1a8d171d22dec9b16de1536 not found: ID does not exist" Oct 31 13:43:48.206954 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:48.206921 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c2c98b-755946465f-m2g26"] Oct 31 13:43:48.220996 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:48.220962 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c2c98b-755946465f-m2g26"] Oct 31 13:43:48.239131 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:48.239105 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rcjt\" (UniqueName: \"kubernetes.io/projected/08846306-817b-4ccf-8ec3-470e1bba6257-kube-api-access-6rcjt\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:48.705459 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:43:48.705414 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:43:49.408111 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:49.408082 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08846306-817b-4ccf-8ec3-470e1bba6257" path="/var/lib/kubelet/pods/08846306-817b-4ccf-8ec3-470e1bba6257/volumes" Oct 31 13:43:53.421027 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:53.420987 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5"] Oct 31 13:43:53.421568 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:53.421238 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" podUID="f5d9b1d1-f122-4108-9e23-d8d6eff64bc7" containerName="istio-proxy" containerID="cri-o://c811a756f24a3eef88e1b5f3033cf051983f59e756b5f32363dc07ac949d0e2f" gracePeriod=30 Oct 31 13:43:54.937756 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:54.937710 5693 patch_prober.go:29] interesting pod/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:43:54.938145 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:54.937758 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" podUID="f5d9b1d1-f122-4108-9e23-d8d6eff64bc7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:43:56.060101 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:56.060059 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-45bcad-77bfcf8bc9-pk9mj" podUID="a1b0fe6e-146d-4f30-90c2-d44f49d6e17f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:43:58.525755 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:58.525721 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5a9e45-7b647fbff9-6xcpd"] Oct 31 13:43:58.540893 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:58.540863 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5a9e45-698ff58799-ztzw8"] Oct 31 13:43:58.745384 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:58.745354 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" Oct 31 13:43:58.899168 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:58.899112 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-istio-token\" (UniqueName: \"kubernetes.io/projected/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-istio-token\") pod \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\" (UID: \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\") " Oct 31 13:43:58.899384 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:58.899224 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-istio-data\") pod \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\" (UID: \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\") " Oct 31 13:43:58.899384 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:58.899299 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-workload-certs\") pod \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\" (UID: \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\") " Oct 31 13:43:58.899384 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:58.899340 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-kube-api-access-nfltj\" (UniqueName: \"kubernetes.io/projected/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-kube-api-access-nfltj\") pod \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\" (UID: \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\") " Oct 31 13:43:58.899384 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:58.899375 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-workload-socket\") pod \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\" (UID: \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\") " Oct 31 13:43:58.899611 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:58.899413 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-istiod-ca-cert\") pod \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\" (UID: \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\") " Oct 31 13:43:58.899611 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:58.899458 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-istio-data" pod "f5d9b1d1-f122-4108-9e23-d8d6eff64bc7" (UID: "f5d9b1d1-f122-4108-9e23-d8d6eff64bc7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:58.899611 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:58.899579 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-istio-podinfo\") pod \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\" (UID: \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\") " Oct 31 13:43:58.899847 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:58.899663 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-istio-envoy\") pod \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\" (UID: \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\") " Oct 31 13:43:58.899847 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:58.899734 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-workload-socket" pod "f5d9b1d1-f122-4108-9e23-d8d6eff64bc7" (UID: "f5d9b1d1-f122-4108-9e23-d8d6eff64bc7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:58.899847 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:58.899738 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-credential-socket\") pod \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\" (UID: \"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7\") " Oct 31 13:43:58.900023 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:58.899913 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-workload-certs" pod "f5d9b1d1-f122-4108-9e23-d8d6eff64bc7" (UID: "f5d9b1d1-f122-4108-9e23-d8d6eff64bc7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:58.900267 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:58.900248 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:58.900345 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:58.900272 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:58.900345 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:58.900287 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:58.907162 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:58.907072 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-istiod-ca-cert" pod "f5d9b1d1-f122-4108-9e23-d8d6eff64bc7" (UID: "f5d9b1d1-f122-4108-9e23-d8d6eff64bc7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:43:58.907162 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:58.907218 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-istio-token" pod "f5d9b1d1-f122-4108-9e23-d8d6eff64bc7" (UID: "f5d9b1d1-f122-4108-9e23-d8d6eff64bc7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:58.907748 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:58.907722 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-credential-socket" pod "f5d9b1d1-f122-4108-9e23-d8d6eff64bc7" (UID: "f5d9b1d1-f122-4108-9e23-d8d6eff64bc7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:58.911116 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:58.910803 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-kube-api-access-nfltj" pod "f5d9b1d1-f122-4108-9e23-d8d6eff64bc7" (UID: "f5d9b1d1-f122-4108-9e23-d8d6eff64bc7"). InnerVolumeSpecName "kube-api-access-nfltj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:58.911116 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:58.911066 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-istio-envoy" pod "f5d9b1d1-f122-4108-9e23-d8d6eff64bc7" (UID: "f5d9b1d1-f122-4108-9e23-d8d6eff64bc7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:58.911304 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:58.911135 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-istio-podinfo" pod "f5d9b1d1-f122-4108-9e23-d8d6eff64bc7" (UID: "f5d9b1d1-f122-4108-9e23-d8d6eff64bc7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:43:59.001927 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:59.001845 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:59.001927 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:59.001886 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:59.001927 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:59.001902 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:59.001927 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:59.001916 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:59.002184 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:59.001931 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfltj\" (UniqueName: \"kubernetes.io/projected/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-kube-api-access-nfltj\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:59.002184 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:59.001946 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:59.272295 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:59.272208 5693 generic.go:356] "Generic (PLEG): container finished" podID="f5d9b1d1-f122-4108-9e23-d8d6eff64bc7" containerID="c811a756f24a3eef88e1b5f3033cf051983f59e756b5f32363dc07ac949d0e2f" exitCode=0 Oct 31 13:43:59.272295 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:59.272254 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" Oct 31 13:43:59.272502 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:59.272291 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" event={"ID":"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7","Type":"ContainerDied","Data":"c811a756f24a3eef88e1b5f3033cf051983f59e756b5f32363dc07ac949d0e2f"} Oct 31 13:43:59.272502 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:59.272349 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5" event={"ID":"f5d9b1d1-f122-4108-9e23-d8d6eff64bc7","Type":"ContainerDied","Data":"9f31eaba8964dc82463aec90a27c58e72f32bc23576380a25b0351f403a91e54"} Oct 31 13:43:59.272502 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:59.272375 5693 scope.go:122] "RemoveContainer" containerID="c811a756f24a3eef88e1b5f3033cf051983f59e756b5f32363dc07ac949d0e2f" Oct 31 13:43:59.282843 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:59.282824 5693 scope.go:122] "RemoveContainer" containerID="c811a756f24a3eef88e1b5f3033cf051983f59e756b5f32363dc07ac949d0e2f" Oct 31 13:43:59.283112 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:43:59.283096 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c811a756f24a3eef88e1b5f3033cf051983f59e756b5f32363dc07ac949d0e2f\": container with ID starting with c811a756f24a3eef88e1b5f3033cf051983f59e756b5f32363dc07ac949d0e2f not found: ID does not exist" containerID="c811a756f24a3eef88e1b5f3033cf051983f59e756b5f32363dc07ac949d0e2f" Oct 31 13:43:59.283153 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:59.283120 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c811a756f24a3eef88e1b5f3033cf051983f59e756b5f32363dc07ac949d0e2f"} err="failed to get container status \"c811a756f24a3eef88e1b5f3033cf051983f59e756b5f32363dc07ac949d0e2f\": rpc error: code = NotFound desc = could not find container \"c811a756f24a3eef88e1b5f3033cf051983f59e756b5f32363dc07ac949d0e2f\": container with ID starting with c811a756f24a3eef88e1b5f3033cf051983f59e756b5f32363dc07ac949d0e2f not found: ID does not exist" Oct 31 13:43:59.311484 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:59.311450 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5"] Oct 31 13:43:59.316907 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:59.316883 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-2njx5"] Oct 31 13:43:59.408567 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:43:59.408533 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5d9b1d1-f122-4108-9e23-d8d6eff64bc7" path="/var/lib/kubelet/pods/f5d9b1d1-f122-4108-9e23-d8d6eff64bc7/volumes" Oct 31 13:44:01.446720 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:01.446679 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-569075-6b667f965f-cfjvr"] Oct 31 13:44:01.462479 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:01.462432 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-569075-67d86d8547-blfvq"] Oct 31 13:44:03.526447 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:03.526398 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-5a9e45-7b647fbff9-6xcpd" podUID="b21df5f3-b4bb-4e47-9287-b42befdf81f1" containerName="payload-processing" containerID="cri-o://543a985c382264c3ed555b9e0ebd58c69e1182de2dfc79493a48f4ce118692dd" gracePeriod=25 Oct 31 13:44:03.541562 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:03.541531 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5a9e45-698ff58799-ztzw8" podUID="2b31a91e-a876-450d-b35c-d5134eb09c1c" containerName="payload-pre-processing" containerID="cri-o://f9dce66d13c776d6a60bd7e3902907f8778805af66be47d432db359df8ec81ac" gracePeriod=25 Oct 31 13:44:03.829065 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:03.829038 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-5a9e45-7b647fbff9-6xcpd" Oct 31 13:44:03.873467 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:03.873435 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b21df5f3-b4bb-4e47-9287-b42befdf81f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b21df5f3-b4bb-4e47-9287-b42befdf81f1-plugins-config-volume\") pod \"b21df5f3-b4bb-4e47-9287-b42befdf81f1\" (UID: \"b21df5f3-b4bb-4e47-9287-b42befdf81f1\") " Oct 31 13:44:03.873702 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:03.873687 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b21df5f3-b4bb-4e47-9287-b42befdf81f1-kube-api-access-v97r8\" (UniqueName: \"kubernetes.io/projected/b21df5f3-b4bb-4e47-9287-b42befdf81f1-kube-api-access-v97r8\") pod \"b21df5f3-b4bb-4e47-9287-b42befdf81f1\" (UID: \"b21df5f3-b4bb-4e47-9287-b42befdf81f1\") " Oct 31 13:44:03.873829 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:03.873808 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b21df5f3-b4bb-4e47-9287-b42befdf81f1-plugins-config-volume" pod "b21df5f3-b4bb-4e47-9287-b42befdf81f1" (UID: "b21df5f3-b4bb-4e47-9287-b42befdf81f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:44:03.874086 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:03.874072 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b21df5f3-b4bb-4e47-9287-b42befdf81f1-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:03.876052 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:03.876029 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b21df5f3-b4bb-4e47-9287-b42befdf81f1-kube-api-access-v97r8" pod "b21df5f3-b4bb-4e47-9287-b42befdf81f1" (UID: "b21df5f3-b4bb-4e47-9287-b42befdf81f1"). InnerVolumeSpecName "kube-api-access-v97r8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:03.975482 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:03.975451 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v97r8\" (UniqueName: \"kubernetes.io/projected/b21df5f3-b4bb-4e47-9287-b42befdf81f1-kube-api-access-v97r8\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:04.303845 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:04.303808 5693 generic.go:356] "Generic (PLEG): container finished" podID="b21df5f3-b4bb-4e47-9287-b42befdf81f1" containerID="543a985c382264c3ed555b9e0ebd58c69e1182de2dfc79493a48f4ce118692dd" exitCode=0 Oct 31 13:44:04.304039 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:04.303863 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-5a9e45-7b647fbff9-6xcpd" Oct 31 13:44:04.304039 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:04.303895 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-5a9e45-7b647fbff9-6xcpd" event={"ID":"b21df5f3-b4bb-4e47-9287-b42befdf81f1","Type":"ContainerDied","Data":"543a985c382264c3ed555b9e0ebd58c69e1182de2dfc79493a48f4ce118692dd"} Oct 31 13:44:04.304039 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:04.303949 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-5a9e45-7b647fbff9-6xcpd" event={"ID":"b21df5f3-b4bb-4e47-9287-b42befdf81f1","Type":"ContainerDied","Data":"c7f1637d0887ceccbc665c43aeab0308806943ee75953d3edc8894aeb44668c1"} Oct 31 13:44:04.304039 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:04.303975 5693 scope.go:122] "RemoveContainer" containerID="543a985c382264c3ed555b9e0ebd58c69e1182de2dfc79493a48f4ce118692dd" Oct 31 13:44:04.316099 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:04.316076 5693 scope.go:122] "RemoveContainer" containerID="543a985c382264c3ed555b9e0ebd58c69e1182de2dfc79493a48f4ce118692dd" Oct 31 13:44:04.316370 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:44:04.316346 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"543a985c382264c3ed555b9e0ebd58c69e1182de2dfc79493a48f4ce118692dd\": container with ID starting with 543a985c382264c3ed555b9e0ebd58c69e1182de2dfc79493a48f4ce118692dd not found: ID does not exist" containerID="543a985c382264c3ed555b9e0ebd58c69e1182de2dfc79493a48f4ce118692dd" Oct 31 13:44:04.316440 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:04.316378 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"543a985c382264c3ed555b9e0ebd58c69e1182de2dfc79493a48f4ce118692dd"} err="failed to get container status \"543a985c382264c3ed555b9e0ebd58c69e1182de2dfc79493a48f4ce118692dd\": rpc error: code = NotFound desc = could not find container \"543a985c382264c3ed555b9e0ebd58c69e1182de2dfc79493a48f4ce118692dd\": container with ID starting with 543a985c382264c3ed555b9e0ebd58c69e1182de2dfc79493a48f4ce118692dd not found: ID does not exist" Oct 31 13:44:04.329626 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:04.329573 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5a9e45-7b647fbff9-6xcpd"] Oct 31 13:44:04.332195 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:04.332169 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5a9e45-7b647fbff9-6xcpd"] Oct 31 13:44:05.408545 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:05.408510 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b21df5f3-b4bb-4e47-9287-b42befdf81f1" path="/var/lib/kubelet/pods/b21df5f3-b4bb-4e47-9287-b42befdf81f1/volumes" Oct 31 13:44:06.044519 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:06.044482 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7"] Oct 31 13:44:06.044766 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:06.044737 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7" podUID="421c561e-1dcd-46b7-85bf-1416e1e007cd" containerName="maas-api" containerID="cri-o://5025613ea840a66fc294cd9ecd26b5a042ebfe8aa1425cc8663af19da43dc910" gracePeriod=30 Oct 31 13:44:06.060376 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:06.060335 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-45bcad-77bfcf8bc9-pk9mj" podUID="a1b0fe6e-146d-4f30-90c2-d44f49d6e17f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:44:06.341147 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:06.341003 5693 generic.go:356] "Generic (PLEG): container finished" podID="421c561e-1dcd-46b7-85bf-1416e1e007cd" containerID="5025613ea840a66fc294cd9ecd26b5a042ebfe8aa1425cc8663af19da43dc910" exitCode=0 Oct 31 13:44:06.341531 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:06.341057 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7" event={"ID":"421c561e-1dcd-46b7-85bf-1416e1e007cd","Type":"ContainerDied","Data":"5025613ea840a66fc294cd9ecd26b5a042ebfe8aa1425cc8663af19da43dc910"} Oct 31 13:44:06.416594 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:06.415124 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7" Oct 31 13:44:06.462982 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:06.462929 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-569075-67d86d8547-blfvq" podUID="29da7f75-0ee1-4981-8ee3-b23f46a67876" containerName="payload-pre-processing" containerID="cri-o://32beede0c6559c47a12501f657ffc1c0f0d8d23ad08599d8018827e8a7332a0a" gracePeriod=25 Oct 31 13:44:06.517796 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:06.517769 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/421c561e-1dcd-46b7-85bf-1416e1e007cd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/421c561e-1dcd-46b7-85bf-1416e1e007cd-openshift-service-ca\") pod \"421c561e-1dcd-46b7-85bf-1416e1e007cd\" (UID: \"421c561e-1dcd-46b7-85bf-1416e1e007cd\") " Oct 31 13:44:06.517936 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:06.517898 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/421c561e-1dcd-46b7-85bf-1416e1e007cd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/421c561e-1dcd-46b7-85bf-1416e1e007cd-metrics-certs\") pod \"421c561e-1dcd-46b7-85bf-1416e1e007cd\" (UID: \"421c561e-1dcd-46b7-85bf-1416e1e007cd\") " Oct 31 13:44:06.517991 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:06.517955 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/421c561e-1dcd-46b7-85bf-1416e1e007cd-kube-api-access-4z9lk\" (UniqueName: \"kubernetes.io/projected/421c561e-1dcd-46b7-85bf-1416e1e007cd-kube-api-access-4z9lk\") pod \"421c561e-1dcd-46b7-85bf-1416e1e007cd\" (UID: \"421c561e-1dcd-46b7-85bf-1416e1e007cd\") " Oct 31 13:44:06.517991 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:06.517985 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/421c561e-1dcd-46b7-85bf-1416e1e007cd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/421c561e-1dcd-46b7-85bf-1416e1e007cd-maas-api-tls\") pod \"421c561e-1dcd-46b7-85bf-1416e1e007cd\" (UID: \"421c561e-1dcd-46b7-85bf-1416e1e007cd\") " Oct 31 13:44:06.518141 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:06.518117 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/421c561e-1dcd-46b7-85bf-1416e1e007cd-openshift-service-ca" pod "421c561e-1dcd-46b7-85bf-1416e1e007cd" (UID: "421c561e-1dcd-46b7-85bf-1416e1e007cd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:44:06.518900 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:06.518579 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/421c561e-1dcd-46b7-85bf-1416e1e007cd-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:06.520226 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:06.520203 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/421c561e-1dcd-46b7-85bf-1416e1e007cd-metrics-certs" pod "421c561e-1dcd-46b7-85bf-1416e1e007cd" (UID: "421c561e-1dcd-46b7-85bf-1416e1e007cd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:44:06.520326 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:06.520279 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/421c561e-1dcd-46b7-85bf-1416e1e007cd-maas-api-tls" pod "421c561e-1dcd-46b7-85bf-1416e1e007cd" (UID: "421c561e-1dcd-46b7-85bf-1416e1e007cd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:44:06.520451 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:06.520430 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/421c561e-1dcd-46b7-85bf-1416e1e007cd-kube-api-access-4z9lk" pod "421c561e-1dcd-46b7-85bf-1416e1e007cd" (UID: "421c561e-1dcd-46b7-85bf-1416e1e007cd"). InnerVolumeSpecName "kube-api-access-4z9lk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:06.619486 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:06.619399 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/421c561e-1dcd-46b7-85bf-1416e1e007cd-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:06.619486 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:06.619434 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/421c561e-1dcd-46b7-85bf-1416e1e007cd-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:06.619486 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:06.619448 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z9lk\" (UniqueName: \"kubernetes.io/projected/421c561e-1dcd-46b7-85bf-1416e1e007cd-kube-api-access-4z9lk\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:07.291323 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:07.291266 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-569075-6b667f965f-cfjvr" podUID="3dc3c2a0-104c-4a6e-a708-9b212dc74267" containerName="payload-processing" containerID="cri-o://2b3cd993a5642f729e432c3b420830b5cc94ea6d3c4a96da5a893fb04bf79b50" gracePeriod=25 Oct 31 13:44:07.350844 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:07.350814 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7" Oct 31 13:44:07.350844 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:07.350831 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7" event={"ID":"421c561e-1dcd-46b7-85bf-1416e1e007cd","Type":"ContainerDied","Data":"d876c47b7f936f11616b91270a3a621dbf8f6d5d011c7a0b6aaf49748537ca00"} Oct 31 13:44:07.351101 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:07.350895 5693 scope.go:122] "RemoveContainer" containerID="5025613ea840a66fc294cd9ecd26b5a042ebfe8aa1425cc8663af19da43dc910" Oct 31 13:44:07.353604 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:07.353571 5693 generic.go:356] "Generic (PLEG): container finished" podID="a1b0fe6e-146d-4f30-90c2-d44f49d6e17f" containerID="799602286a71706e495ec5e66c4f20420baefa9d2087cd03f2cc44b52694b5e5" exitCode=0 Oct 31 13:44:07.353760 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:07.353629 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-45bcad-77bfcf8bc9-pk9mj" event={"ID":"a1b0fe6e-146d-4f30-90c2-d44f49d6e17f","Type":"ContainerDied","Data":"799602286a71706e495ec5e66c4f20420baefa9d2087cd03f2cc44b52694b5e5"} Oct 31 13:44:07.381498 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:07.381455 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7"] Oct 31 13:44:07.386471 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:07.386445 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-569075-666c967965-q6nj7"] Oct 31 13:44:07.409125 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:07.409101 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="421c561e-1dcd-46b7-85bf-1416e1e007cd" path="/var/lib/kubelet/pods/421c561e-1dcd-46b7-85bf-1416e1e007cd/volumes" Oct 31 13:44:07.425237 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:07.425218 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-45bcad-77bfcf8bc9-pk9mj" Oct 31 13:44:07.530029 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:07.530001 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1b0fe6e-146d-4f30-90c2-d44f49d6e17f-kube-api-access-84wt5\" (UniqueName: \"kubernetes.io/projected/a1b0fe6e-146d-4f30-90c2-d44f49d6e17f-kube-api-access-84wt5\") pod \"a1b0fe6e-146d-4f30-90c2-d44f49d6e17f\" (UID: \"a1b0fe6e-146d-4f30-90c2-d44f49d6e17f\") " Oct 31 13:44:07.530226 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:07.530066 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1b0fe6e-146d-4f30-90c2-d44f49d6e17f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1b0fe6e-146d-4f30-90c2-d44f49d6e17f-plugins-config-volume\") pod \"a1b0fe6e-146d-4f30-90c2-d44f49d6e17f\" (UID: \"a1b0fe6e-146d-4f30-90c2-d44f49d6e17f\") " Oct 31 13:44:07.530495 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:07.530464 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1b0fe6e-146d-4f30-90c2-d44f49d6e17f-plugins-config-volume" pod "a1b0fe6e-146d-4f30-90c2-d44f49d6e17f" (UID: "a1b0fe6e-146d-4f30-90c2-d44f49d6e17f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:44:07.532262 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:07.532242 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1b0fe6e-146d-4f30-90c2-d44f49d6e17f-kube-api-access-84wt5" pod "a1b0fe6e-146d-4f30-90c2-d44f49d6e17f" (UID: "a1b0fe6e-146d-4f30-90c2-d44f49d6e17f"). InnerVolumeSpecName "kube-api-access-84wt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:07.632261 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:07.632221 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84wt5\" (UniqueName: \"kubernetes.io/projected/a1b0fe6e-146d-4f30-90c2-d44f49d6e17f-kube-api-access-84wt5\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:07.632261 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:07.632257 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1b0fe6e-146d-4f30-90c2-d44f49d6e17f-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:08.225500 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:08.225462 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf"] Oct 31 13:44:08.225763 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:08.225703 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" podUID="ddfd9d12-e8bb-42ee-a375-aba470b89eeb" containerName="istio-proxy" containerID="cri-o://323676112f62d08aa0da0625ee1bcf7a47d234088e905b86f1ce22e5d68bdd23" gracePeriod=30 Oct 31 13:44:08.228241 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:08.228217 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6"] Oct 31 13:44:08.228419 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:08.228398 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" podUID="bedfdf67-dd1e-4902-9ca8-4f404d3a0841" containerName="istio-proxy" containerID="cri-o://ecfb8eaafa0fca016f14c25f3264ff12834189fcd99bb34db8642a0b516d0679" gracePeriod=30 Oct 31 13:44:08.439509 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:08.439471 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-45bcad-77bfcf8bc9-pk9mj" event={"ID":"a1b0fe6e-146d-4f30-90c2-d44f49d6e17f","Type":"ContainerDied","Data":"477ae209df062042d03a2a8a2578ea39f4dc81581d8a8181cc9c69e4fbe283e1"} Oct 31 13:44:08.440116 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:08.440095 5693 scope.go:122] "RemoveContainer" containerID="799602286a71706e495ec5e66c4f20420baefa9d2087cd03f2cc44b52694b5e5" Oct 31 13:44:08.440369 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:08.440353 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-45bcad-77bfcf8bc9-pk9mj" Oct 31 13:44:08.485306 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:08.485222 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-45bcad-77bfcf8bc9-pk9mj"] Oct 31 13:44:08.507960 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:08.507931 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-45bcad-77bfcf8bc9-pk9mj"] Oct 31 13:44:09.409564 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:09.409467 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1b0fe6e-146d-4f30-90c2-d44f49d6e17f" path="/var/lib/kubelet/pods/a1b0fe6e-146d-4f30-90c2-d44f49d6e17f/volumes" Oct 31 13:44:11.404087 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:11.404043 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5a9e45-698ff58799-ztzw8" podUID="2b31a91e-a876-450d-b35c-d5134eb09c1c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:44:13.475752 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.475718 5693 generic.go:356] "Generic (PLEG): container finished" podID="ddfd9d12-e8bb-42ee-a375-aba470b89eeb" containerID="323676112f62d08aa0da0625ee1bcf7a47d234088e905b86f1ce22e5d68bdd23" exitCode=0 Oct 31 13:44:13.476247 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.475799 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" event={"ID":"ddfd9d12-e8bb-42ee-a375-aba470b89eeb","Type":"ContainerDied","Data":"323676112f62d08aa0da0625ee1bcf7a47d234088e905b86f1ce22e5d68bdd23"} Oct 31 13:44:13.477738 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.477694 5693 generic.go:356] "Generic (PLEG): container finished" podID="bedfdf67-dd1e-4902-9ca8-4f404d3a0841" containerID="ecfb8eaafa0fca016f14c25f3264ff12834189fcd99bb34db8642a0b516d0679" exitCode=0 Oct 31 13:44:13.477928 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.477752 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" event={"ID":"bedfdf67-dd1e-4902-9ca8-4f404d3a0841","Type":"ContainerDied","Data":"ecfb8eaafa0fca016f14c25f3264ff12834189fcd99bb34db8642a0b516d0679"} Oct 31 13:44:13.508621 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.508550 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" Oct 31 13:44:13.528552 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.528525 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" Oct 31 13:44:13.608872 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.608844 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-istio-token\" (UniqueName: \"kubernetes.io/projected/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-istio-token\") pod \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\" (UID: \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\") " Oct 31 13:44:13.609394 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.608879 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-istiod-ca-cert\") pod \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\" (UID: \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\") " Oct 31 13:44:13.609394 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.608936 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-istio-podinfo\") pod \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\" (UID: \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\") " Oct 31 13:44:13.609394 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.608966 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-workload-socket\") pod \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\" (UID: \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\") " Oct 31 13:44:13.609394 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.609044 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-istio-data\") pod \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\" (UID: \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\") " Oct 31 13:44:13.609394 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.609087 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-credential-socket\") pod \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\" (UID: \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\") " Oct 31 13:44:13.609394 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.609131 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-workload-certs\") pod \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\" (UID: \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\") " Oct 31 13:44:13.609394 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.609165 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-istio-podinfo\") pod \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\" (UID: \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\") " Oct 31 13:44:13.609394 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.609222 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-kube-api-access-jcx22\" (UniqueName: \"kubernetes.io/projected/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-kube-api-access-jcx22\") pod \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\" (UID: \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\") " Oct 31 13:44:13.609394 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.609289 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-istio-data\") pod \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\" (UID: \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\") " Oct 31 13:44:13.609394 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.609320 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-workload-certs\") pod \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\" (UID: \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\") " Oct 31 13:44:13.609394 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.609351 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-workload-socket\") pod \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\" (UID: \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\") " Oct 31 13:44:13.609394 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.609351 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-istio-data" pod "bedfdf67-dd1e-4902-9ca8-4f404d3a0841" (UID: "bedfdf67-dd1e-4902-9ca8-4f404d3a0841"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:13.609394 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.609376 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-istiod-ca-cert" pod "bedfdf67-dd1e-4902-9ca8-4f404d3a0841" (UID: "bedfdf67-dd1e-4902-9ca8-4f404d3a0841"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:44:13.609394 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.609389 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-credential-socket\") pod \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\" (UID: \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\") " Oct 31 13:44:13.610287 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.609420 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-istio-envoy\") pod \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\" (UID: \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\") " Oct 31 13:44:13.610287 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.609448 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-istio-token\" (UniqueName: \"kubernetes.io/projected/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-istio-token\") pod \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\" (UID: \"bedfdf67-dd1e-4902-9ca8-4f404d3a0841\") " Oct 31 13:44:13.610287 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.609498 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-istio-envoy\") pod \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\" (UID: \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\") " Oct 31 13:44:13.610287 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.609526 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-workload-socket" pod "ddfd9d12-e8bb-42ee-a375-aba470b89eeb" (UID: "ddfd9d12-e8bb-42ee-a375-aba470b89eeb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:13.610287 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.609536 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-istiod-ca-cert\") pod \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\" (UID: \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\") " Oct 31 13:44:13.610287 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.609544 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-credential-socket" pod "bedfdf67-dd1e-4902-9ca8-4f404d3a0841" (UID: "bedfdf67-dd1e-4902-9ca8-4f404d3a0841"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:13.610287 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.609593 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-istio-data" pod "ddfd9d12-e8bb-42ee-a375-aba470b89eeb" (UID: "ddfd9d12-e8bb-42ee-a375-aba470b89eeb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:13.610287 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.609606 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-kube-api-access-cv5dj\" (UniqueName: \"kubernetes.io/projected/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-kube-api-access-cv5dj\") pod \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\" (UID: \"ddfd9d12-e8bb-42ee-a375-aba470b89eeb\") " Oct 31 13:44:13.610287 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.609781 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-workload-certs" pod "bedfdf67-dd1e-4902-9ca8-4f404d3a0841" (UID: "bedfdf67-dd1e-4902-9ca8-4f404d3a0841"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:13.610287 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.609827 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-credential-socket" pod "ddfd9d12-e8bb-42ee-a375-aba470b89eeb" (UID: "ddfd9d12-e8bb-42ee-a375-aba470b89eeb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:13.610287 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.609877 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-istiod-ca-cert" pod "ddfd9d12-e8bb-42ee-a375-aba470b89eeb" (UID: "ddfd9d12-e8bb-42ee-a375-aba470b89eeb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:44:13.610287 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.610005 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-workload-socket" pod "bedfdf67-dd1e-4902-9ca8-4f404d3a0841" (UID: "bedfdf67-dd1e-4902-9ca8-4f404d3a0841"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:13.610287 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.610021 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-workload-certs" pod "ddfd9d12-e8bb-42ee-a375-aba470b89eeb" (UID: "ddfd9d12-e8bb-42ee-a375-aba470b89eeb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:13.611295 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.610300 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:13.611295 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.610321 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:13.611295 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.610335 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:13.611295 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.610349 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:13.611295 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.610362 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:13.611295 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.610375 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:13.611295 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.610387 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:13.611295 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.610400 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:13.611295 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.610413 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:13.611295 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.610426 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:13.612857 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.612451 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-istio-token" pod "ddfd9d12-e8bb-42ee-a375-aba470b89eeb" (UID: "ddfd9d12-e8bb-42ee-a375-aba470b89eeb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:13.612857 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.612594 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-istio-envoy" pod "bedfdf67-dd1e-4902-9ca8-4f404d3a0841" (UID: "bedfdf67-dd1e-4902-9ca8-4f404d3a0841"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:13.613022 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.612870 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-istio-podinfo" pod "bedfdf67-dd1e-4902-9ca8-4f404d3a0841" (UID: "bedfdf67-dd1e-4902-9ca8-4f404d3a0841"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:44:13.613022 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.612937 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-kube-api-access-cv5dj" pod "ddfd9d12-e8bb-42ee-a375-aba470b89eeb" (UID: "ddfd9d12-e8bb-42ee-a375-aba470b89eeb"). InnerVolumeSpecName "kube-api-access-cv5dj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:13.613117 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.613032 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-kube-api-access-jcx22" pod "bedfdf67-dd1e-4902-9ca8-4f404d3a0841" (UID: "bedfdf67-dd1e-4902-9ca8-4f404d3a0841"). InnerVolumeSpecName "kube-api-access-jcx22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:13.613176 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.613126 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-istio-token" pod "bedfdf67-dd1e-4902-9ca8-4f404d3a0841" (UID: "bedfdf67-dd1e-4902-9ca8-4f404d3a0841"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:13.613231 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.613193 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-istio-envoy" pod "ddfd9d12-e8bb-42ee-a375-aba470b89eeb" (UID: "ddfd9d12-e8bb-42ee-a375-aba470b89eeb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:13.614021 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.614007 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-istio-podinfo" pod "ddfd9d12-e8bb-42ee-a375-aba470b89eeb" (UID: "ddfd9d12-e8bb-42ee-a375-aba470b89eeb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:44:13.712184 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.712140 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:13.712184 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.712181 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:13.712184 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.712191 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:13.712184 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.712202 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcx22\" (UniqueName: \"kubernetes.io/projected/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-kube-api-access-jcx22\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:13.712485 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.712211 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:13.712485 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.712222 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bedfdf67-dd1e-4902-9ca8-4f404d3a0841-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:13.712485 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.712233 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:13.712485 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:13.712246 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv5dj\" (UniqueName: \"kubernetes.io/projected/ddfd9d12-e8bb-42ee-a375-aba470b89eeb-kube-api-access-cv5dj\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:14.484816 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:14.484785 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" Oct 31 13:44:14.485312 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:14.484815 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf" event={"ID":"ddfd9d12-e8bb-42ee-a375-aba470b89eeb","Type":"ContainerDied","Data":"09aee59fb47c0ab5e6b5d0d7bd78dd7a4085ccbcfb779c5b1045f0ed6764737b"} Oct 31 13:44:14.485312 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:14.484883 5693 scope.go:122] "RemoveContainer" containerID="323676112f62d08aa0da0625ee1bcf7a47d234088e905b86f1ce22e5d68bdd23" Oct 31 13:44:14.486290 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:14.486273 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" Oct 31 13:44:14.486361 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:14.486295 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6" event={"ID":"bedfdf67-dd1e-4902-9ca8-4f404d3a0841","Type":"ContainerDied","Data":"71b188e89648cd743bb9ccac0f81c9668a13bf9746d10bf52a2e5e45c19141c0"} Oct 31 13:44:14.496483 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:14.496419 5693 scope.go:122] "RemoveContainer" containerID="ecfb8eaafa0fca016f14c25f3264ff12834189fcd99bb34db8642a0b516d0679" Oct 31 13:44:14.527768 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:14.527730 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf"] Oct 31 13:44:14.531270 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:14.531235 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-h4czf"] Oct 31 13:44:14.545762 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:14.545730 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6"] Oct 31 13:44:14.548499 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:14.548473 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6b8s6"] Oct 31 13:44:15.411632 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:15.411589 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bedfdf67-dd1e-4902-9ca8-4f404d3a0841" path="/var/lib/kubelet/pods/bedfdf67-dd1e-4902-9ca8-4f404d3a0841/volumes" Oct 31 13:44:15.413675 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:15.412320 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddfd9d12-e8bb-42ee-a375-aba470b89eeb" path="/var/lib/kubelet/pods/ddfd9d12-e8bb-42ee-a375-aba470b89eeb/volumes" Oct 31 13:44:15.618882 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:15.618844 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-569075-6b667f965f-cfjvr" podUID="3dc3c2a0-104c-4a6e-a708-9b212dc74267" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:44:15.619331 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:15.618891 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-569075-67d86d8547-blfvq" podUID="29da7f75-0ee1-4981-8ee3-b23f46a67876" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:44:15.725439 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:15.722516 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc"] Oct 31 13:44:15.725439 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:15.722838 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" podUID="fb054e25-5c0a-4525-a337-1a29d0c7fe0a" containerName="istio-proxy" containerID="cri-o://cc2e3b6f760cafba62d9f44c96c764de06c0ee2e24e15495df493952d98aa2de" gracePeriod=30 Oct 31 13:44:17.398006 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:17.397971 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt"] Oct 31 13:44:17.411696 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:17.411664 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9"] Oct 31 13:44:18.687899 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:44:18.687862 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:44:20.992695 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:20.992663 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" Oct 31 13:44:21.035675 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:21.035150 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-workload-certs\") pod \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\" (UID: \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\") " Oct 31 13:44:21.035675 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:21.035203 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-istio-envoy\") pod \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\" (UID: \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\") " Oct 31 13:44:21.035675 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:21.035251 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-credential-socket\") pod \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\" (UID: \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\") " Oct 31 13:44:21.035675 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:21.035577 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-workload-certs" pod "fb054e25-5c0a-4525-a337-1a29d0c7fe0a" (UID: "fb054e25-5c0a-4525-a337-1a29d0c7fe0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:21.035675 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:21.035568 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-credential-socket" pod "fb054e25-5c0a-4525-a337-1a29d0c7fe0a" (UID: "fb054e25-5c0a-4525-a337-1a29d0c7fe0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:21.036348 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:21.036320 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-istio-podinfo\") pod \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\" (UID: \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\") " Oct 31 13:44:21.036497 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:21.036388 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-istiod-ca-cert\") pod \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\" (UID: \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\") " Oct 31 13:44:21.036497 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:21.036482 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-istio-token\" (UniqueName: \"kubernetes.io/projected/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-istio-token\") pod \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\" (UID: \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\") " Oct 31 13:44:21.036724 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:21.036687 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-workload-socket\") pod \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\" (UID: \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\") " Oct 31 13:44:21.037031 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:21.036909 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-workload-socket" pod "fb054e25-5c0a-4525-a337-1a29d0c7fe0a" (UID: "fb054e25-5c0a-4525-a337-1a29d0c7fe0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:21.037123 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:21.037070 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-kube-api-access-q9w2l\" (UniqueName: \"kubernetes.io/projected/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-kube-api-access-q9w2l\") pod \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\" (UID: \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\") " Oct 31 13:44:21.037187 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:21.037117 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-istiod-ca-cert" pod "fb054e25-5c0a-4525-a337-1a29d0c7fe0a" (UID: "fb054e25-5c0a-4525-a337-1a29d0c7fe0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:44:21.037580 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:21.037549 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-istio-data\") pod \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\" (UID: \"fb054e25-5c0a-4525-a337-1a29d0c7fe0a\") " Oct 31 13:44:21.038084 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:21.037997 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-istio-data" pod "fb054e25-5c0a-4525-a337-1a29d0c7fe0a" (UID: "fb054e25-5c0a-4525-a337-1a29d0c7fe0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:21.038260 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:21.038233 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-istio-podinfo" pod "fb054e25-5c0a-4525-a337-1a29d0c7fe0a" (UID: "fb054e25-5c0a-4525-a337-1a29d0c7fe0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:44:21.038681 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:21.038631 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:21.038769 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:21.038680 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:21.038769 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:21.038695 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:21.038769 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:21.038709 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:21.038769 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:21.038722 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:21.038769 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:21.038735 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:21.039266 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:21.039229 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-istio-token" pod "fb054e25-5c0a-4525-a337-1a29d0c7fe0a" (UID: "fb054e25-5c0a-4525-a337-1a29d0c7fe0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:21.040088 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:21.040063 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-istio-envoy" pod "fb054e25-5c0a-4525-a337-1a29d0c7fe0a" (UID: "fb054e25-5c0a-4525-a337-1a29d0c7fe0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:21.041037 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:21.041015 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-kube-api-access-q9w2l" pod "fb054e25-5c0a-4525-a337-1a29d0c7fe0a" (UID: "fb054e25-5c0a-4525-a337-1a29d0c7fe0a"). InnerVolumeSpecName "kube-api-access-q9w2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:21.139459 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:21.139422 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:21.139459 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:21.139451 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:21.139459 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:21.139461 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9w2l\" (UniqueName: \"kubernetes.io/projected/fb054e25-5c0a-4525-a337-1a29d0c7fe0a-kube-api-access-q9w2l\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:21.403860 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:21.403818 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5a9e45-698ff58799-ztzw8" podUID="2b31a91e-a876-450d-b35c-d5134eb09c1c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:44:21.535037 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:21.535005 5693 generic.go:356] "Generic (PLEG): container finished" podID="fb054e25-5c0a-4525-a337-1a29d0c7fe0a" containerID="cc2e3b6f760cafba62d9f44c96c764de06c0ee2e24e15495df493952d98aa2de" exitCode=0 Oct 31 13:44:21.535223 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:21.535051 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" Oct 31 13:44:21.535223 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:21.535086 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" event={"ID":"fb054e25-5c0a-4525-a337-1a29d0c7fe0a","Type":"ContainerDied","Data":"cc2e3b6f760cafba62d9f44c96c764de06c0ee2e24e15495df493952d98aa2de"} Oct 31 13:44:21.535223 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:21.535145 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc" event={"ID":"fb054e25-5c0a-4525-a337-1a29d0c7fe0a","Type":"ContainerDied","Data":"68363ae29167356f88c9a86cdf7b4e38a9aacf54be26a3aefb275bd30693f6d9"} Oct 31 13:44:21.535223 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:21.535174 5693 scope.go:122] "RemoveContainer" containerID="cc2e3b6f760cafba62d9f44c96c764de06c0ee2e24e15495df493952d98aa2de" Oct 31 13:44:21.545511 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:21.545480 5693 scope.go:122] "RemoveContainer" containerID="cc2e3b6f760cafba62d9f44c96c764de06c0ee2e24e15495df493952d98aa2de" Oct 31 13:44:21.545810 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:44:21.545792 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc2e3b6f760cafba62d9f44c96c764de06c0ee2e24e15495df493952d98aa2de\": container with ID starting with cc2e3b6f760cafba62d9f44c96c764de06c0ee2e24e15495df493952d98aa2de not found: ID does not exist" containerID="cc2e3b6f760cafba62d9f44c96c764de06c0ee2e24e15495df493952d98aa2de" Oct 31 13:44:21.545875 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:21.545817 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc2e3b6f760cafba62d9f44c96c764de06c0ee2e24e15495df493952d98aa2de"} err="failed to get container status \"cc2e3b6f760cafba62d9f44c96c764de06c0ee2e24e15495df493952d98aa2de\": rpc error: code = NotFound desc = could not find container \"cc2e3b6f760cafba62d9f44c96c764de06c0ee2e24e15495df493952d98aa2de\": container with ID starting with cc2e3b6f760cafba62d9f44c96c764de06c0ee2e24e15495df493952d98aa2de not found: ID does not exist" Oct 31 13:44:21.554931 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:21.554907 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc"] Oct 31 13:44:21.568190 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:21.568169 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-df2bc"] Oct 31 13:44:23.409100 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:23.409067 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb054e25-5c0a-4525-a337-1a29d0c7fe0a" path="/var/lib/kubelet/pods/fb054e25-5c0a-4525-a337-1a29d0c7fe0a/volumes" Oct 31 13:44:23.807377 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:23.807352 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5a9e45-698ff58799-ztzw8" Oct 31 13:44:23.836849 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:23.836819 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-4574ec-76b8ddbc7f-7jbsx"] Oct 31 13:44:23.875463 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:23.875432 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b31a91e-a876-450d-b35c-d5134eb09c1c-kube-api-access-gf74f\" (UniqueName: \"kubernetes.io/projected/2b31a91e-a876-450d-b35c-d5134eb09c1c-kube-api-access-gf74f\") pod \"2b31a91e-a876-450d-b35c-d5134eb09c1c\" (UID: \"2b31a91e-a876-450d-b35c-d5134eb09c1c\") " Oct 31 13:44:23.875663 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:23.875471 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b31a91e-a876-450d-b35c-d5134eb09c1c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b31a91e-a876-450d-b35c-d5134eb09c1c-plugins-config-volume\") pod \"2b31a91e-a876-450d-b35c-d5134eb09c1c\" (UID: \"2b31a91e-a876-450d-b35c-d5134eb09c1c\") " Oct 31 13:44:23.876006 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:23.875947 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b31a91e-a876-450d-b35c-d5134eb09c1c-plugins-config-volume" pod "2b31a91e-a876-450d-b35c-d5134eb09c1c" (UID: "2b31a91e-a876-450d-b35c-d5134eb09c1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:44:23.876155 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:23.876105 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b31a91e-a876-450d-b35c-d5134eb09c1c-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:23.878847 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:23.878823 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b31a91e-a876-450d-b35c-d5134eb09c1c-kube-api-access-gf74f" pod "2b31a91e-a876-450d-b35c-d5134eb09c1c" (UID: "2b31a91e-a876-450d-b35c-d5134eb09c1c"). InnerVolumeSpecName "kube-api-access-gf74f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:23.881090 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:23.881065 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4574ec-bdd99c999-65drk"] Oct 31 13:44:23.977303 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:23.977226 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf74f\" (UniqueName: \"kubernetes.io/projected/2b31a91e-a876-450d-b35c-d5134eb09c1c-kube-api-access-gf74f\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:24.557417 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:24.557382 5693 generic.go:356] "Generic (PLEG): container finished" podID="2b31a91e-a876-450d-b35c-d5134eb09c1c" containerID="f9dce66d13c776d6a60bd7e3902907f8778805af66be47d432db359df8ec81ac" exitCode=0 Oct 31 13:44:24.557951 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:24.557463 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5a9e45-698ff58799-ztzw8" Oct 31 13:44:24.557951 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:24.557469 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5a9e45-698ff58799-ztzw8" event={"ID":"2b31a91e-a876-450d-b35c-d5134eb09c1c","Type":"ContainerDied","Data":"f9dce66d13c776d6a60bd7e3902907f8778805af66be47d432db359df8ec81ac"} Oct 31 13:44:24.557951 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:24.557572 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5a9e45-698ff58799-ztzw8" event={"ID":"2b31a91e-a876-450d-b35c-d5134eb09c1c","Type":"ContainerDied","Data":"81e731b4d35173ec0f746e51121de8e747cc1b0b25c67f7dfac4fba4d789fe3a"} Oct 31 13:44:24.557951 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:24.557601 5693 scope.go:122] "RemoveContainer" containerID="f9dce66d13c776d6a60bd7e3902907f8778805af66be47d432db359df8ec81ac" Oct 31 13:44:24.574384 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:24.574362 5693 scope.go:122] "RemoveContainer" containerID="f9dce66d13c776d6a60bd7e3902907f8778805af66be47d432db359df8ec81ac" Oct 31 13:44:24.574714 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:44:24.574691 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9dce66d13c776d6a60bd7e3902907f8778805af66be47d432db359df8ec81ac\": container with ID starting with f9dce66d13c776d6a60bd7e3902907f8778805af66be47d432db359df8ec81ac not found: ID does not exist" containerID="f9dce66d13c776d6a60bd7e3902907f8778805af66be47d432db359df8ec81ac" Oct 31 13:44:24.574797 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:24.574728 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9dce66d13c776d6a60bd7e3902907f8778805af66be47d432db359df8ec81ac"} err="failed to get container status \"f9dce66d13c776d6a60bd7e3902907f8778805af66be47d432db359df8ec81ac\": rpc error: code = NotFound desc = could not find container \"f9dce66d13c776d6a60bd7e3902907f8778805af66be47d432db359df8ec81ac\": container with ID starting with f9dce66d13c776d6a60bd7e3902907f8778805af66be47d432db359df8ec81ac not found: ID does not exist" Oct 31 13:44:24.589218 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:24.589164 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5a9e45-698ff58799-ztzw8"] Oct 31 13:44:24.597900 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:24.597867 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5a9e45-698ff58799-ztzw8"] Oct 31 13:44:25.408823 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:25.408791 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b31a91e-a876-450d-b35c-d5134eb09c1c" path="/var/lib/kubelet/pods/2b31a91e-a876-450d-b35c-d5134eb09c1c/volumes" Oct 31 13:44:25.619579 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:25.619539 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-569075-6b667f965f-cfjvr" podUID="3dc3c2a0-104c-4a6e-a708-9b212dc74267" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:44:25.619999 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:25.619539 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-569075-67d86d8547-blfvq" podUID="29da7f75-0ee1-4981-8ee3-b23f46a67876" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:44:26.723337 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:26.723269 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-569075-67d86d8547-blfvq" Oct 31 13:44:26.818461 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:26.818427 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29da7f75-0ee1-4981-8ee3-b23f46a67876-kube-api-access-7cl24\" (UniqueName: \"kubernetes.io/projected/29da7f75-0ee1-4981-8ee3-b23f46a67876-kube-api-access-7cl24\") pod \"29da7f75-0ee1-4981-8ee3-b23f46a67876\" (UID: \"29da7f75-0ee1-4981-8ee3-b23f46a67876\") " Oct 31 13:44:26.818671 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:26.818493 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29da7f75-0ee1-4981-8ee3-b23f46a67876-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29da7f75-0ee1-4981-8ee3-b23f46a67876-plugins-config-volume\") pod \"29da7f75-0ee1-4981-8ee3-b23f46a67876\" (UID: \"29da7f75-0ee1-4981-8ee3-b23f46a67876\") " Oct 31 13:44:26.818956 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:26.818925 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29da7f75-0ee1-4981-8ee3-b23f46a67876-plugins-config-volume" pod "29da7f75-0ee1-4981-8ee3-b23f46a67876" (UID: "29da7f75-0ee1-4981-8ee3-b23f46a67876"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:44:26.819163 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:26.819146 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29da7f75-0ee1-4981-8ee3-b23f46a67876-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:26.820900 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:26.820875 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29da7f75-0ee1-4981-8ee3-b23f46a67876-kube-api-access-7cl24" pod "29da7f75-0ee1-4981-8ee3-b23f46a67876" (UID: "29da7f75-0ee1-4981-8ee3-b23f46a67876"). InnerVolumeSpecName "kube-api-access-7cl24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:26.920723 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:26.920686 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cl24\" (UniqueName: \"kubernetes.io/projected/29da7f75-0ee1-4981-8ee3-b23f46a67876-kube-api-access-7cl24\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:27.578874 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:27.578842 5693 generic.go:356] "Generic (PLEG): container finished" podID="29da7f75-0ee1-4981-8ee3-b23f46a67876" containerID="32beede0c6559c47a12501f657ffc1c0f0d8d23ad08599d8018827e8a7332a0a" exitCode=0 Oct 31 13:44:27.579013 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:27.578888 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-569075-67d86d8547-blfvq" Oct 31 13:44:27.579013 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:27.578933 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-569075-67d86d8547-blfvq" event={"ID":"29da7f75-0ee1-4981-8ee3-b23f46a67876","Type":"ContainerDied","Data":"32beede0c6559c47a12501f657ffc1c0f0d8d23ad08599d8018827e8a7332a0a"} Oct 31 13:44:27.579013 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:27.578991 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-569075-67d86d8547-blfvq" event={"ID":"29da7f75-0ee1-4981-8ee3-b23f46a67876","Type":"ContainerDied","Data":"15f659dee4ad6512617499d6f189cbd455fa72a9a381eff5306663fb80a632e8"} Oct 31 13:44:27.579165 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:27.579022 5693 scope.go:122] "RemoveContainer" containerID="32beede0c6559c47a12501f657ffc1c0f0d8d23ad08599d8018827e8a7332a0a" Oct 31 13:44:27.580523 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:27.580501 5693 generic.go:356] "Generic (PLEG): container finished" podID="3dc3c2a0-104c-4a6e-a708-9b212dc74267" containerID="2b3cd993a5642f729e432c3b420830b5cc94ea6d3c4a96da5a893fb04bf79b50" exitCode=0 Oct 31 13:44:27.580600 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:27.580561 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-569075-6b667f965f-cfjvr" event={"ID":"3dc3c2a0-104c-4a6e-a708-9b212dc74267","Type":"ContainerDied","Data":"2b3cd993a5642f729e432c3b420830b5cc94ea6d3c4a96da5a893fb04bf79b50"} Oct 31 13:44:27.580900 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:27.580776 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-569075-6b667f965f-cfjvr" event={"ID":"3dc3c2a0-104c-4a6e-a708-9b212dc74267","Type":"ContainerDied","Data":"5199e4ebcee2531c01a080295d592658f418b0df930e90de45981f2ea83cc8fc"} Oct 31 13:44:27.580900 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:27.580804 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5199e4ebcee2531c01a080295d592658f418b0df930e90de45981f2ea83cc8fc" Oct 31 13:44:27.591607 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:27.591584 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-569075-6b667f965f-cfjvr" Oct 31 13:44:27.592069 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:27.591837 5693 scope.go:122] "RemoveContainer" containerID="32beede0c6559c47a12501f657ffc1c0f0d8d23ad08599d8018827e8a7332a0a" Oct 31 13:44:27.592127 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:44:27.592099 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32beede0c6559c47a12501f657ffc1c0f0d8d23ad08599d8018827e8a7332a0a\": container with ID starting with 32beede0c6559c47a12501f657ffc1c0f0d8d23ad08599d8018827e8a7332a0a not found: ID does not exist" containerID="32beede0c6559c47a12501f657ffc1c0f0d8d23ad08599d8018827e8a7332a0a" Oct 31 13:44:27.592169 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:27.592130 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32beede0c6559c47a12501f657ffc1c0f0d8d23ad08599d8018827e8a7332a0a"} err="failed to get container status \"32beede0c6559c47a12501f657ffc1c0f0d8d23ad08599d8018827e8a7332a0a\": rpc error: code = NotFound desc = could not find container \"32beede0c6559c47a12501f657ffc1c0f0d8d23ad08599d8018827e8a7332a0a\": container with ID starting with 32beede0c6559c47a12501f657ffc1c0f0d8d23ad08599d8018827e8a7332a0a not found: ID does not exist" Oct 31 13:44:27.604023 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:27.603989 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-569075-67d86d8547-blfvq"] Oct 31 13:44:27.606217 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:27.606193 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-569075-67d86d8547-blfvq"] Oct 31 13:44:27.731166 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:27.731137 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dc3c2a0-104c-4a6e-a708-9b212dc74267-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dc3c2a0-104c-4a6e-a708-9b212dc74267-plugins-config-volume\") pod \"3dc3c2a0-104c-4a6e-a708-9b212dc74267\" (UID: \"3dc3c2a0-104c-4a6e-a708-9b212dc74267\") " Oct 31 13:44:27.731556 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:27.731233 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dc3c2a0-104c-4a6e-a708-9b212dc74267-kube-api-access-4bnvw\" (UniqueName: \"kubernetes.io/projected/3dc3c2a0-104c-4a6e-a708-9b212dc74267-kube-api-access-4bnvw\") pod \"3dc3c2a0-104c-4a6e-a708-9b212dc74267\" (UID: \"3dc3c2a0-104c-4a6e-a708-9b212dc74267\") " Oct 31 13:44:27.731556 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:27.731492 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dc3c2a0-104c-4a6e-a708-9b212dc74267-plugins-config-volume" pod "3dc3c2a0-104c-4a6e-a708-9b212dc74267" (UID: "3dc3c2a0-104c-4a6e-a708-9b212dc74267"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:44:27.731776 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:27.731761 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dc3c2a0-104c-4a6e-a708-9b212dc74267-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:27.733582 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:27.733561 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dc3c2a0-104c-4a6e-a708-9b212dc74267-kube-api-access-4bnvw" pod "3dc3c2a0-104c-4a6e-a708-9b212dc74267" (UID: "3dc3c2a0-104c-4a6e-a708-9b212dc74267"). InnerVolumeSpecName "kube-api-access-4bnvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:27.832696 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:27.832604 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bnvw\" (UniqueName: \"kubernetes.io/projected/3dc3c2a0-104c-4a6e-a708-9b212dc74267-kube-api-access-4bnvw\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:28.588793 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:28.588763 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-569075-6b667f965f-cfjvr" Oct 31 13:44:28.613991 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:28.613956 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-569075-6b667f965f-cfjvr"] Oct 31 13:44:28.625908 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:28.625878 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-569075-6b667f965f-cfjvr"] Oct 31 13:44:28.837920 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:28.837878 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4574ec-76b8ddbc7f-7jbsx" podUID="eccfdc26-fb5e-4d07-8f67-43723a93450b" containerName="payload-processing" containerID="cri-o://ce153237605e2a96e3b560565ea5d20b532e109bb30efa56acb7487771f1afd0" gracePeriod=25 Oct 31 13:44:28.881893 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:28.881789 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4574ec-bdd99c999-65drk" podUID="8fbf694c-e0a8-4a8a-a214-a06c58f7642b" containerName="payload-pre-processing" containerID="cri-o://14a881eda934b8b064d3794928889c4c4473eaa892f0a40f2572e33c56ae0640" gracePeriod=25 Oct 31 13:44:29.410138 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:29.410107 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29da7f75-0ee1-4981-8ee3-b23f46a67876" path="/var/lib/kubelet/pods/29da7f75-0ee1-4981-8ee3-b23f46a67876/volumes" Oct 31 13:44:29.410518 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:29.410505 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dc3c2a0-104c-4a6e-a708-9b212dc74267" path="/var/lib/kubelet/pods/3dc3c2a0-104c-4a6e-a708-9b212dc74267/volumes" Oct 31 13:44:30.202261 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:30.202227 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4574ec-76b8ddbc7f-7jbsx" podUID="eccfdc26-fb5e-4d07-8f67-43723a93450b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:44:30.202633 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:30.202258 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4574ec-bdd99c999-65drk" podUID="8fbf694c-e0a8-4a8a-a214-a06c58f7642b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:44:32.016231 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:32.016196 5693 scope.go:122] "RemoveContainer" containerID="2b3cd993a5642f729e432c3b420830b5cc94ea6d3c4a96da5a893fb04bf79b50" Oct 31 13:44:32.421030 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:32.420995 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" podUID="ff9a9909-6fc3-475b-a9fe-562ec150c76a" containerName="main" containerID="cri-o://f1cc3f00414a45bb1a71fb9d0d22a83d2534cdc0a46a25a58ff116ddfbd9988c" gracePeriod=45 Oct 31 13:44:32.434195 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:32.434155 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" podUID="710a52fa-2bc4-4bb9-96be-4b08dc60de67" containerName="main" containerID="cri-o://08a64c99723396b4e181f3e1e86de7013dbd2fe18bc6f6bc7193511246fd5463" gracePeriod=45 Oct 31 13:44:32.521986 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:32.521936 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" podUID="ff9a9909-6fc3-475b-a9fe-562ec150c76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": dial tcp 10.133.0.75:8000: connect: connection refused" Oct 31 13:44:32.621607 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:32.616206 5693 generic.go:356] "Generic (PLEG): container finished" podID="ff9a9909-6fc3-475b-a9fe-562ec150c76a" containerID="f1cc3f00414a45bb1a71fb9d0d22a83d2534cdc0a46a25a58ff116ddfbd9988c" exitCode=0 Oct 31 13:44:32.621607 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:32.616438 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" event={"ID":"ff9a9909-6fc3-475b-a9fe-562ec150c76a","Type":"ContainerDied","Data":"f1cc3f00414a45bb1a71fb9d0d22a83d2534cdc0a46a25a58ff116ddfbd9988c"} Oct 31 13:44:32.621607 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:32.618746 5693 generic.go:356] "Generic (PLEG): container finished" podID="710a52fa-2bc4-4bb9-96be-4b08dc60de67" containerID="08a64c99723396b4e181f3e1e86de7013dbd2fe18bc6f6bc7193511246fd5463" exitCode=0 Oct 31 13:44:32.621607 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:32.618843 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" event={"ID":"710a52fa-2bc4-4bb9-96be-4b08dc60de67","Type":"ContainerDied","Data":"08a64c99723396b4e181f3e1e86de7013dbd2fe18bc6f6bc7193511246fd5463"} Oct 31 13:44:32.753159 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:32.747365 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" Oct 31 13:44:32.753159 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:32.751484 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" Oct 31 13:44:32.898581 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:32.898548 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff9a9909-6fc3-475b-a9fe-562ec150c76a-tls-certs\" (UniqueName: \"kubernetes.io/secret/ff9a9909-6fc3-475b-a9fe-562ec150c76a-tls-certs\") pod \"ff9a9909-6fc3-475b-a9fe-562ec150c76a\" (UID: \"ff9a9909-6fc3-475b-a9fe-562ec150c76a\") " Oct 31 13:44:32.898783 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:32.898612 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/710a52fa-2bc4-4bb9-96be-4b08dc60de67-dshm\" (UniqueName: \"kubernetes.io/empty-dir/710a52fa-2bc4-4bb9-96be-4b08dc60de67-dshm\") pod \"710a52fa-2bc4-4bb9-96be-4b08dc60de67\" (UID: \"710a52fa-2bc4-4bb9-96be-4b08dc60de67\") " Oct 31 13:44:32.898783 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:32.898678 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/710a52fa-2bc4-4bb9-96be-4b08dc60de67-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/710a52fa-2bc4-4bb9-96be-4b08dc60de67-model-cache\") pod \"710a52fa-2bc4-4bb9-96be-4b08dc60de67\" (UID: \"710a52fa-2bc4-4bb9-96be-4b08dc60de67\") " Oct 31 13:44:32.898910 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:32.898783 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/710a52fa-2bc4-4bb9-96be-4b08dc60de67-home\" (UniqueName: \"kubernetes.io/empty-dir/710a52fa-2bc4-4bb9-96be-4b08dc60de67-home\") pod \"710a52fa-2bc4-4bb9-96be-4b08dc60de67\" (UID: \"710a52fa-2bc4-4bb9-96be-4b08dc60de67\") " Oct 31 13:44:32.898910 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:32.898861 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff9a9909-6fc3-475b-a9fe-562ec150c76a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff9a9909-6fc3-475b-a9fe-562ec150c76a-dshm\") pod \"ff9a9909-6fc3-475b-a9fe-562ec150c76a\" (UID: \"ff9a9909-6fc3-475b-a9fe-562ec150c76a\") " Oct 31 13:44:32.899011 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:32.898929 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/710a52fa-2bc4-4bb9-96be-4b08dc60de67-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/710a52fa-2bc4-4bb9-96be-4b08dc60de67-tmp-dir\") pod \"710a52fa-2bc4-4bb9-96be-4b08dc60de67\" (UID: \"710a52fa-2bc4-4bb9-96be-4b08dc60de67\") " Oct 31 13:44:32.899011 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:32.898935 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/710a52fa-2bc4-4bb9-96be-4b08dc60de67-model-cache" pod "710a52fa-2bc4-4bb9-96be-4b08dc60de67" (UID: "710a52fa-2bc4-4bb9-96be-4b08dc60de67"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:32.899011 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:32.898985 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/710a52fa-2bc4-4bb9-96be-4b08dc60de67-kube-api-access-4nmdx\" (UniqueName: \"kubernetes.io/projected/710a52fa-2bc4-4bb9-96be-4b08dc60de67-kube-api-access-4nmdx\") pod \"710a52fa-2bc4-4bb9-96be-4b08dc60de67\" (UID: \"710a52fa-2bc4-4bb9-96be-4b08dc60de67\") " Oct 31 13:44:32.899181 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:32.899018 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/710a52fa-2bc4-4bb9-96be-4b08dc60de67-home" pod "710a52fa-2bc4-4bb9-96be-4b08dc60de67" (UID: "710a52fa-2bc4-4bb9-96be-4b08dc60de67"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:32.899181 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:32.899057 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff9a9909-6fc3-475b-a9fe-562ec150c76a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff9a9909-6fc3-475b-a9fe-562ec150c76a-tmp-dir\") pod \"ff9a9909-6fc3-475b-a9fe-562ec150c76a\" (UID: \"ff9a9909-6fc3-475b-a9fe-562ec150c76a\") " Oct 31 13:44:32.899181 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:32.899101 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/710a52fa-2bc4-4bb9-96be-4b08dc60de67-tls-certs\" (UniqueName: \"kubernetes.io/secret/710a52fa-2bc4-4bb9-96be-4b08dc60de67-tls-certs\") pod \"710a52fa-2bc4-4bb9-96be-4b08dc60de67\" (UID: \"710a52fa-2bc4-4bb9-96be-4b08dc60de67\") " Oct 31 13:44:32.899181 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:32.899146 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/710a52fa-2bc4-4bb9-96be-4b08dc60de67-tmp-dir" pod "710a52fa-2bc4-4bb9-96be-4b08dc60de67" (UID: "710a52fa-2bc4-4bb9-96be-4b08dc60de67"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:32.899181 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:32.899161 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff9a9909-6fc3-475b-a9fe-562ec150c76a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff9a9909-6fc3-475b-a9fe-562ec150c76a-model-cache\") pod \"ff9a9909-6fc3-475b-a9fe-562ec150c76a\" (UID: \"ff9a9909-6fc3-475b-a9fe-562ec150c76a\") " Oct 31 13:44:32.899433 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:32.899192 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff9a9909-6fc3-475b-a9fe-562ec150c76a-kube-api-access-wptc9\" (UniqueName: \"kubernetes.io/projected/ff9a9909-6fc3-475b-a9fe-562ec150c76a-kube-api-access-wptc9\") pod \"ff9a9909-6fc3-475b-a9fe-562ec150c76a\" (UID: \"ff9a9909-6fc3-475b-a9fe-562ec150c76a\") " Oct 31 13:44:32.899433 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:32.899263 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff9a9909-6fc3-475b-a9fe-562ec150c76a-home\" (UniqueName: \"kubernetes.io/empty-dir/ff9a9909-6fc3-475b-a9fe-562ec150c76a-home\") pod \"ff9a9909-6fc3-475b-a9fe-562ec150c76a\" (UID: \"ff9a9909-6fc3-475b-a9fe-562ec150c76a\") " Oct 31 13:44:32.899529 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:32.899508 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff9a9909-6fc3-475b-a9fe-562ec150c76a-model-cache" pod "ff9a9909-6fc3-475b-a9fe-562ec150c76a" (UID: "ff9a9909-6fc3-475b-a9fe-562ec150c76a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:32.899798 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:32.899764 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff9a9909-6fc3-475b-a9fe-562ec150c76a-home" pod "ff9a9909-6fc3-475b-a9fe-562ec150c76a" (UID: "ff9a9909-6fc3-475b-a9fe-562ec150c76a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:32.900005 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:32.899933 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff9a9909-6fc3-475b-a9fe-562ec150c76a-tmp-dir" pod "ff9a9909-6fc3-475b-a9fe-562ec150c76a" (UID: "ff9a9909-6fc3-475b-a9fe-562ec150c76a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:32.900378 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:32.900269 5693 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff9a9909-6fc3-475b-a9fe-562ec150c76a-home\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:32.900378 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:32.900297 5693 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/710a52fa-2bc4-4bb9-96be-4b08dc60de67-model-cache\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:32.900378 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:32.900313 5693 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/710a52fa-2bc4-4bb9-96be-4b08dc60de67-home\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:32.900378 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:32.900327 5693 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/710a52fa-2bc4-4bb9-96be-4b08dc60de67-tmp-dir\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:32.900378 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:32.900340 5693 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff9a9909-6fc3-475b-a9fe-562ec150c76a-tmp-dir\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:32.900378 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:32.900353 5693 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff9a9909-6fc3-475b-a9fe-562ec150c76a-model-cache\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:32.901789 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:32.901719 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/710a52fa-2bc4-4bb9-96be-4b08dc60de67-dshm" pod "710a52fa-2bc4-4bb9-96be-4b08dc60de67" (UID: "710a52fa-2bc4-4bb9-96be-4b08dc60de67"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:32.901789 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:32.901736 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/710a52fa-2bc4-4bb9-96be-4b08dc60de67-tls-certs" pod "710a52fa-2bc4-4bb9-96be-4b08dc60de67" (UID: "710a52fa-2bc4-4bb9-96be-4b08dc60de67"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:44:32.901789 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:32.901749 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/710a52fa-2bc4-4bb9-96be-4b08dc60de67-kube-api-access-4nmdx" pod "710a52fa-2bc4-4bb9-96be-4b08dc60de67" (UID: "710a52fa-2bc4-4bb9-96be-4b08dc60de67"). InnerVolumeSpecName "kube-api-access-4nmdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:32.901789 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:32.901771 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff9a9909-6fc3-475b-a9fe-562ec150c76a-tls-certs" pod "ff9a9909-6fc3-475b-a9fe-562ec150c76a" (UID: "ff9a9909-6fc3-475b-a9fe-562ec150c76a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:44:32.902423 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:32.902400 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff9a9909-6fc3-475b-a9fe-562ec150c76a-dshm" pod "ff9a9909-6fc3-475b-a9fe-562ec150c76a" (UID: "ff9a9909-6fc3-475b-a9fe-562ec150c76a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:32.902496 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:32.902480 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff9a9909-6fc3-475b-a9fe-562ec150c76a-kube-api-access-wptc9" pod "ff9a9909-6fc3-475b-a9fe-562ec150c76a" (UID: "ff9a9909-6fc3-475b-a9fe-562ec150c76a"). InnerVolumeSpecName "kube-api-access-wptc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:33.002884 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:33.001955 5693 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff9a9909-6fc3-475b-a9fe-562ec150c76a-dshm\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:33.002884 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:33.001990 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nmdx\" (UniqueName: \"kubernetes.io/projected/710a52fa-2bc4-4bb9-96be-4b08dc60de67-kube-api-access-4nmdx\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:33.002884 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:33.002006 5693 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/710a52fa-2bc4-4bb9-96be-4b08dc60de67-tls-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:33.002884 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:33.002022 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wptc9\" (UniqueName: \"kubernetes.io/projected/ff9a9909-6fc3-475b-a9fe-562ec150c76a-kube-api-access-wptc9\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:33.002884 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:33.002037 5693 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff9a9909-6fc3-475b-a9fe-562ec150c76a-tls-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:33.002884 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:33.002050 5693 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/710a52fa-2bc4-4bb9-96be-4b08dc60de67-dshm\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:33.627711 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:33.627680 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" Oct 31 13:44:33.627711 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:33.627679 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt" event={"ID":"ff9a9909-6fc3-475b-a9fe-562ec150c76a","Type":"ContainerDied","Data":"e0350c6e7ea27c82c582d71851551ce39d4d62adcb3162615022e80ec2aff666"} Oct 31 13:44:33.628302 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:33.627735 5693 scope.go:122] "RemoveContainer" containerID="f1cc3f00414a45bb1a71fb9d0d22a83d2534cdc0a46a25a58ff116ddfbd9988c" Oct 31 13:44:33.629485 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:33.629444 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" event={"ID":"710a52fa-2bc4-4bb9-96be-4b08dc60de67","Type":"ContainerDied","Data":"ade6940a3d9bc4b56f7d64e574bef8ae721f91497e277bea78d5fb0274572f32"} Oct 31 13:44:33.629485 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:33.629483 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9" Oct 31 13:44:33.641430 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:33.641405 5693 scope.go:122] "RemoveContainer" containerID="08a64c99723396b4e181f3e1e86de7013dbd2fe18bc6f6bc7193511246fd5463" Oct 31 13:44:33.650300 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:33.650270 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt"] Oct 31 13:44:33.653633 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:33.653611 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-3dd1e5/distinct-2-w2-3dd1e5-kserve-7d4d5d9bdb-h72zt"] Oct 31 13:44:33.673402 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:33.673320 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9"] Oct 31 13:44:33.682341 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:33.682313 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-3dd1e5/distinct-w2-3dd1e5-kserve-6c9bdf6bfc-psxb9"] Oct 31 13:44:35.407790 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:35.407751 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="710a52fa-2bc4-4bb9-96be-4b08dc60de67" path="/var/lib/kubelet/pods/710a52fa-2bc4-4bb9-96be-4b08dc60de67/volumes" Oct 31 13:44:35.408217 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:35.408202 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff9a9909-6fc3-475b-a9fe-562ec150c76a" path="/var/lib/kubelet/pods/ff9a9909-6fc3-475b-a9fe-562ec150c76a/volumes" Oct 31 13:44:40.202120 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:40.202027 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4574ec-bdd99c999-65drk" podUID="8fbf694c-e0a8-4a8a-a214-a06c58f7642b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:44:40.202588 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:40.202027 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4574ec-76b8ddbc7f-7jbsx" podUID="eccfdc26-fb5e-4d07-8f67-43723a93450b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:44:42.258901 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:42.258865 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5452fc-768b7c485f-z66t4"] Oct 31 13:44:47.260208 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:47.260143 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5452fc-768b7c485f-z66t4" podUID="55afb55f-1e19-420a-a30e-7c14e5082e97" containerName="payload-pre-processing" containerID="cri-o://3c487137982fc2a6d565cfcbf2d149751f9089137b1407903adc5f114104d02e" gracePeriod=25 Oct 31 13:44:48.690790 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:44:48.690754 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:44:49.141159 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:49.141127 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4574ec-76b8ddbc7f-7jbsx" Oct 31 13:44:49.186684 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:49.186654 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4574ec-bdd99c999-65drk" Oct 31 13:44:49.227546 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:49.227518 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eccfdc26-fb5e-4d07-8f67-43723a93450b-kube-api-access-ll2gv\" (UniqueName: \"kubernetes.io/projected/eccfdc26-fb5e-4d07-8f67-43723a93450b-kube-api-access-ll2gv\") pod \"eccfdc26-fb5e-4d07-8f67-43723a93450b\" (UID: \"eccfdc26-fb5e-4d07-8f67-43723a93450b\") " Oct 31 13:44:49.227748 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:49.227626 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fbf694c-e0a8-4a8a-a214-a06c58f7642b-kube-api-access-5lvxs\" (UniqueName: \"kubernetes.io/projected/8fbf694c-e0a8-4a8a-a214-a06c58f7642b-kube-api-access-5lvxs\") pod \"8fbf694c-e0a8-4a8a-a214-a06c58f7642b\" (UID: \"8fbf694c-e0a8-4a8a-a214-a06c58f7642b\") " Oct 31 13:44:49.227748 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:49.227698 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fbf694c-e0a8-4a8a-a214-a06c58f7642b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fbf694c-e0a8-4a8a-a214-a06c58f7642b-plugins-config-volume\") pod \"8fbf694c-e0a8-4a8a-a214-a06c58f7642b\" (UID: \"8fbf694c-e0a8-4a8a-a214-a06c58f7642b\") " Oct 31 13:44:49.227906 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:49.227776 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eccfdc26-fb5e-4d07-8f67-43723a93450b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eccfdc26-fb5e-4d07-8f67-43723a93450b-plugins-config-volume\") pod \"eccfdc26-fb5e-4d07-8f67-43723a93450b\" (UID: \"eccfdc26-fb5e-4d07-8f67-43723a93450b\") " Oct 31 13:44:49.228120 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:49.228091 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fbf694c-e0a8-4a8a-a214-a06c58f7642b-plugins-config-volume" pod "8fbf694c-e0a8-4a8a-a214-a06c58f7642b" (UID: "8fbf694c-e0a8-4a8a-a214-a06c58f7642b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:44:49.228237 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:49.228196 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eccfdc26-fb5e-4d07-8f67-43723a93450b-plugins-config-volume" pod "eccfdc26-fb5e-4d07-8f67-43723a93450b" (UID: "eccfdc26-fb5e-4d07-8f67-43723a93450b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:44:49.229951 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:49.229931 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eccfdc26-fb5e-4d07-8f67-43723a93450b-kube-api-access-ll2gv" pod "eccfdc26-fb5e-4d07-8f67-43723a93450b" (UID: "eccfdc26-fb5e-4d07-8f67-43723a93450b"). InnerVolumeSpecName "kube-api-access-ll2gv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:49.230053 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:49.229947 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fbf694c-e0a8-4a8a-a214-a06c58f7642b-kube-api-access-5lvxs" pod "8fbf694c-e0a8-4a8a-a214-a06c58f7642b" (UID: "8fbf694c-e0a8-4a8a-a214-a06c58f7642b"). InnerVolumeSpecName "kube-api-access-5lvxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:49.329393 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:49.329304 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lvxs\" (UniqueName: \"kubernetes.io/projected/8fbf694c-e0a8-4a8a-a214-a06c58f7642b-kube-api-access-5lvxs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:49.329393 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:49.329331 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fbf694c-e0a8-4a8a-a214-a06c58f7642b-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:49.329393 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:49.329341 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eccfdc26-fb5e-4d07-8f67-43723a93450b-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:49.329393 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:49.329350 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll2gv\" (UniqueName: \"kubernetes.io/projected/eccfdc26-fb5e-4d07-8f67-43723a93450b-kube-api-access-ll2gv\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:49.752416 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:49.752373 5693 generic.go:356] "Generic (PLEG): container finished" podID="eccfdc26-fb5e-4d07-8f67-43723a93450b" containerID="ce153237605e2a96e3b560565ea5d20b532e109bb30efa56acb7487771f1afd0" exitCode=0 Oct 31 13:44:49.752416 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:49.752421 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4574ec-76b8ddbc7f-7jbsx" Oct 31 13:44:49.752964 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:49.752449 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4574ec-76b8ddbc7f-7jbsx" event={"ID":"eccfdc26-fb5e-4d07-8f67-43723a93450b","Type":"ContainerDied","Data":"ce153237605e2a96e3b560565ea5d20b532e109bb30efa56acb7487771f1afd0"} Oct 31 13:44:49.752964 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:49.752519 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4574ec-76b8ddbc7f-7jbsx" event={"ID":"eccfdc26-fb5e-4d07-8f67-43723a93450b","Type":"ContainerDied","Data":"c2ef6d7c5bc7690f66a664027080d639de3614f22a8341f917b56774dff9205d"} Oct 31 13:44:49.752964 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:49.752548 5693 scope.go:122] "RemoveContainer" containerID="ce153237605e2a96e3b560565ea5d20b532e109bb30efa56acb7487771f1afd0" Oct 31 13:44:49.754029 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:49.754009 5693 generic.go:356] "Generic (PLEG): container finished" podID="8fbf694c-e0a8-4a8a-a214-a06c58f7642b" containerID="14a881eda934b8b064d3794928889c4c4473eaa892f0a40f2572e33c56ae0640" exitCode=0 Oct 31 13:44:49.754156 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:49.754059 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4574ec-bdd99c999-65drk" Oct 31 13:44:49.754156 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:49.754070 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4574ec-bdd99c999-65drk" event={"ID":"8fbf694c-e0a8-4a8a-a214-a06c58f7642b","Type":"ContainerDied","Data":"14a881eda934b8b064d3794928889c4c4473eaa892f0a40f2572e33c56ae0640"} Oct 31 13:44:49.754156 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:49.754104 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4574ec-bdd99c999-65drk" event={"ID":"8fbf694c-e0a8-4a8a-a214-a06c58f7642b","Type":"ContainerDied","Data":"5ec1f014603c0fd1629bb6c3c247a01e0823b422e0daa122fe9acd0a9259a1ff"} Oct 31 13:44:49.765155 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:49.765137 5693 scope.go:122] "RemoveContainer" containerID="ce153237605e2a96e3b560565ea5d20b532e109bb30efa56acb7487771f1afd0" Oct 31 13:44:49.765488 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:44:49.765467 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce153237605e2a96e3b560565ea5d20b532e109bb30efa56acb7487771f1afd0\": container with ID starting with ce153237605e2a96e3b560565ea5d20b532e109bb30efa56acb7487771f1afd0 not found: ID does not exist" containerID="ce153237605e2a96e3b560565ea5d20b532e109bb30efa56acb7487771f1afd0" Oct 31 13:44:49.765558 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:49.765504 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce153237605e2a96e3b560565ea5d20b532e109bb30efa56acb7487771f1afd0"} err="failed to get container status \"ce153237605e2a96e3b560565ea5d20b532e109bb30efa56acb7487771f1afd0\": rpc error: code = NotFound desc = could not find container \"ce153237605e2a96e3b560565ea5d20b532e109bb30efa56acb7487771f1afd0\": container with ID starting with ce153237605e2a96e3b560565ea5d20b532e109bb30efa56acb7487771f1afd0 not found: ID does not exist" Oct 31 13:44:49.765558 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:49.765520 5693 scope.go:122] "RemoveContainer" containerID="14a881eda934b8b064d3794928889c4c4473eaa892f0a40f2572e33c56ae0640" Oct 31 13:44:49.775890 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:49.775854 5693 scope.go:122] "RemoveContainer" containerID="14a881eda934b8b064d3794928889c4c4473eaa892f0a40f2572e33c56ae0640" Oct 31 13:44:49.779950 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:44:49.779927 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14a881eda934b8b064d3794928889c4c4473eaa892f0a40f2572e33c56ae0640\": container with ID starting with 14a881eda934b8b064d3794928889c4c4473eaa892f0a40f2572e33c56ae0640 not found: ID does not exist" containerID="14a881eda934b8b064d3794928889c4c4473eaa892f0a40f2572e33c56ae0640" Oct 31 13:44:49.780054 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:49.779961 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14a881eda934b8b064d3794928889c4c4473eaa892f0a40f2572e33c56ae0640"} err="failed to get container status \"14a881eda934b8b064d3794928889c4c4473eaa892f0a40f2572e33c56ae0640\": rpc error: code = NotFound desc = could not find container \"14a881eda934b8b064d3794928889c4c4473eaa892f0a40f2572e33c56ae0640\": container with ID starting with 14a881eda934b8b064d3794928889c4c4473eaa892f0a40f2572e33c56ae0640 not found: ID does not exist" Oct 31 13:44:49.780680 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:49.780652 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-4574ec-76b8ddbc7f-7jbsx"] Oct 31 13:44:49.798179 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:49.798142 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-4574ec-76b8ddbc7f-7jbsx"] Oct 31 13:44:49.840146 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:49.840110 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4574ec-bdd99c999-65drk"] Oct 31 13:44:49.844766 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:49.844739 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4574ec-bdd99c999-65drk"] Oct 31 13:44:51.407950 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:51.407920 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fbf694c-e0a8-4a8a-a214-a06c58f7642b" path="/var/lib/kubelet/pods/8fbf694c-e0a8-4a8a-a214-a06c58f7642b/volumes" Oct 31 13:44:51.408340 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:51.408303 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eccfdc26-fb5e-4d07-8f67-43723a93450b" path="/var/lib/kubelet/pods/eccfdc26-fb5e-4d07-8f67-43723a93450b/volumes" Oct 31 13:44:51.863080 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:51.862984 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5452fc-768b7c485f-z66t4" podUID="55afb55f-1e19-420a-a30e-7c14e5082e97" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:44:55.241447 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:55.241414 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp"] Oct 31 13:44:55.242035 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:55.241651 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" podUID="dd8946ab-9867-4565-98e0-9d2c62eaff29" containerName="istio-proxy" containerID="cri-o://89cdb613060a8d3b421a700ead246bffd084515f0a6092a1d91fd0adf90d8659" gracePeriod=30 Oct 31 13:44:58.368410 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:58.368375 5693 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:44:58.369095 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:44:58.368438 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" podUID="dd8946ab-9867-4565-98e0-9d2c62eaff29" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:45:00.498725 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:00.498700 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" Oct 31 13:45:00.506356 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:00.506296 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd8946ab-9867-4565-98e0-9d2c62eaff29-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd8946ab-9867-4565-98e0-9d2c62eaff29-istio-envoy\") pod \"dd8946ab-9867-4565-98e0-9d2c62eaff29\" (UID: \"dd8946ab-9867-4565-98e0-9d2c62eaff29\") " Oct 31 13:45:00.506356 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:00.506343 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd8946ab-9867-4565-98e0-9d2c62eaff29-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd8946ab-9867-4565-98e0-9d2c62eaff29-workload-certs\") pod \"dd8946ab-9867-4565-98e0-9d2c62eaff29\" (UID: \"dd8946ab-9867-4565-98e0-9d2c62eaff29\") " Oct 31 13:45:00.506532 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:00.506404 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd8946ab-9867-4565-98e0-9d2c62eaff29-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd8946ab-9867-4565-98e0-9d2c62eaff29-istio-podinfo\") pod \"dd8946ab-9867-4565-98e0-9d2c62eaff29\" (UID: \"dd8946ab-9867-4565-98e0-9d2c62eaff29\") " Oct 31 13:45:00.506532 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:00.506422 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd8946ab-9867-4565-98e0-9d2c62eaff29-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd8946ab-9867-4565-98e0-9d2c62eaff29-credential-socket\") pod \"dd8946ab-9867-4565-98e0-9d2c62eaff29\" (UID: \"dd8946ab-9867-4565-98e0-9d2c62eaff29\") " Oct 31 13:45:00.506532 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:00.506504 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd8946ab-9867-4565-98e0-9d2c62eaff29-istio-token\" (UniqueName: \"kubernetes.io/projected/dd8946ab-9867-4565-98e0-9d2c62eaff29-istio-token\") pod \"dd8946ab-9867-4565-98e0-9d2c62eaff29\" (UID: \"dd8946ab-9867-4565-98e0-9d2c62eaff29\") " Oct 31 13:45:00.506732 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:00.506536 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd8946ab-9867-4565-98e0-9d2c62eaff29-kube-api-access-d2rl6\" (UniqueName: \"kubernetes.io/projected/dd8946ab-9867-4565-98e0-9d2c62eaff29-kube-api-access-d2rl6\") pod \"dd8946ab-9867-4565-98e0-9d2c62eaff29\" (UID: \"dd8946ab-9867-4565-98e0-9d2c62eaff29\") " Oct 31 13:45:00.506732 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:00.506579 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd8946ab-9867-4565-98e0-9d2c62eaff29-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd8946ab-9867-4565-98e0-9d2c62eaff29-istiod-ca-cert\") pod \"dd8946ab-9867-4565-98e0-9d2c62eaff29\" (UID: \"dd8946ab-9867-4565-98e0-9d2c62eaff29\") " Oct 31 13:45:00.506732 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:00.506616 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd8946ab-9867-4565-98e0-9d2c62eaff29-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd8946ab-9867-4565-98e0-9d2c62eaff29-istio-data\") pod \"dd8946ab-9867-4565-98e0-9d2c62eaff29\" (UID: \"dd8946ab-9867-4565-98e0-9d2c62eaff29\") " Oct 31 13:45:00.506732 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:00.506678 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd8946ab-9867-4565-98e0-9d2c62eaff29-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd8946ab-9867-4565-98e0-9d2c62eaff29-workload-socket\") pod \"dd8946ab-9867-4565-98e0-9d2c62eaff29\" (UID: \"dd8946ab-9867-4565-98e0-9d2c62eaff29\") " Oct 31 13:45:00.506732 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:00.506693 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd8946ab-9867-4565-98e0-9d2c62eaff29-workload-certs" pod "dd8946ab-9867-4565-98e0-9d2c62eaff29" (UID: "dd8946ab-9867-4565-98e0-9d2c62eaff29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:00.507003 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:00.506822 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd8946ab-9867-4565-98e0-9d2c62eaff29-credential-socket" pod "dd8946ab-9867-4565-98e0-9d2c62eaff29" (UID: "dd8946ab-9867-4565-98e0-9d2c62eaff29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:00.507063 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:00.507030 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd8946ab-9867-4565-98e0-9d2c62eaff29-workload-socket" pod "dd8946ab-9867-4565-98e0-9d2c62eaff29" (UID: "dd8946ab-9867-4565-98e0-9d2c62eaff29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:00.507255 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:00.507235 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd8946ab-9867-4565-98e0-9d2c62eaff29-istio-data" pod "dd8946ab-9867-4565-98e0-9d2c62eaff29" (UID: "dd8946ab-9867-4565-98e0-9d2c62eaff29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:00.507363 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:00.507290 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd8946ab-9867-4565-98e0-9d2c62eaff29-istiod-ca-cert" pod "dd8946ab-9867-4565-98e0-9d2c62eaff29" (UID: "dd8946ab-9867-4565-98e0-9d2c62eaff29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:00.507363 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:00.507281 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd8946ab-9867-4565-98e0-9d2c62eaff29-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:00.507452 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:00.507397 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd8946ab-9867-4565-98e0-9d2c62eaff29-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:00.507452 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:00.507413 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd8946ab-9867-4565-98e0-9d2c62eaff29-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:00.508892 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:00.508869 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd8946ab-9867-4565-98e0-9d2c62eaff29-istio-podinfo" pod "dd8946ab-9867-4565-98e0-9d2c62eaff29" (UID: "dd8946ab-9867-4565-98e0-9d2c62eaff29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:45:00.509229 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:00.509209 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd8946ab-9867-4565-98e0-9d2c62eaff29-istio-envoy" pod "dd8946ab-9867-4565-98e0-9d2c62eaff29" (UID: "dd8946ab-9867-4565-98e0-9d2c62eaff29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:00.509514 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:00.509491 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd8946ab-9867-4565-98e0-9d2c62eaff29-kube-api-access-d2rl6" pod "dd8946ab-9867-4565-98e0-9d2c62eaff29" (UID: "dd8946ab-9867-4565-98e0-9d2c62eaff29"). InnerVolumeSpecName "kube-api-access-d2rl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:00.509629 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:00.509547 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd8946ab-9867-4565-98e0-9d2c62eaff29-istio-token" pod "dd8946ab-9867-4565-98e0-9d2c62eaff29" (UID: "dd8946ab-9867-4565-98e0-9d2c62eaff29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:00.608940 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:00.608899 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd8946ab-9867-4565-98e0-9d2c62eaff29-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:00.608940 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:00.608937 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd8946ab-9867-4565-98e0-9d2c62eaff29-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:00.609132 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:00.608954 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2rl6\" (UniqueName: \"kubernetes.io/projected/dd8946ab-9867-4565-98e0-9d2c62eaff29-kube-api-access-d2rl6\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:00.609132 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:00.608969 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd8946ab-9867-4565-98e0-9d2c62eaff29-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:00.609132 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:00.608984 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd8946ab-9867-4565-98e0-9d2c62eaff29-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:00.609132 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:00.608997 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd8946ab-9867-4565-98e0-9d2c62eaff29-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:00.822631 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:00.822538 5693 generic.go:356] "Generic (PLEG): container finished" podID="dd8946ab-9867-4565-98e0-9d2c62eaff29" containerID="89cdb613060a8d3b421a700ead246bffd084515f0a6092a1d91fd0adf90d8659" exitCode=0 Oct 31 13:45:00.822631 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:00.822596 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" Oct 31 13:45:00.822631 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:00.822620 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" event={"ID":"dd8946ab-9867-4565-98e0-9d2c62eaff29","Type":"ContainerDied","Data":"89cdb613060a8d3b421a700ead246bffd084515f0a6092a1d91fd0adf90d8659"} Oct 31 13:45:00.822966 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:00.822708 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp" event={"ID":"dd8946ab-9867-4565-98e0-9d2c62eaff29","Type":"ContainerDied","Data":"fe40682a5c254ae98de9435427da2bcd4c215bbf88564067eabab19f260b2fa4"} Oct 31 13:45:00.822966 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:00.822731 5693 scope.go:122] "RemoveContainer" containerID="89cdb613060a8d3b421a700ead246bffd084515f0a6092a1d91fd0adf90d8659" Oct 31 13:45:00.833488 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:00.833467 5693 scope.go:122] "RemoveContainer" containerID="89cdb613060a8d3b421a700ead246bffd084515f0a6092a1d91fd0adf90d8659" Oct 31 13:45:00.833795 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:45:00.833775 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89cdb613060a8d3b421a700ead246bffd084515f0a6092a1d91fd0adf90d8659\": container with ID starting with 89cdb613060a8d3b421a700ead246bffd084515f0a6092a1d91fd0adf90d8659 not found: ID does not exist" containerID="89cdb613060a8d3b421a700ead246bffd084515f0a6092a1d91fd0adf90d8659" Oct 31 13:45:00.833876 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:00.833819 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89cdb613060a8d3b421a700ead246bffd084515f0a6092a1d91fd0adf90d8659"} err="failed to get container status \"89cdb613060a8d3b421a700ead246bffd084515f0a6092a1d91fd0adf90d8659\": rpc error: code = NotFound desc = could not find container \"89cdb613060a8d3b421a700ead246bffd084515f0a6092a1d91fd0adf90d8659\": container with ID starting with 89cdb613060a8d3b421a700ead246bffd084515f0a6092a1d91fd0adf90d8659 not found: ID does not exist" Oct 31 13:45:00.895892 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:00.895853 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp"] Oct 31 13:45:00.907439 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:00.907405 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-qcknp"] Oct 31 13:45:01.407291 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:01.407258 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd8946ab-9867-4565-98e0-9d2c62eaff29" path="/var/lib/kubelet/pods/dd8946ab-9867-4565-98e0-9d2c62eaff29/volumes" Oct 31 13:45:01.862383 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:01.862304 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5452fc-768b7c485f-z66t4" podUID="55afb55f-1e19-420a-a30e-7c14e5082e97" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:45:07.005526 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:07.005486 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-47bcd9-66cc57988c-9hldq"] Oct 31 13:45:07.025915 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:07.025875 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-47bcd9-5c675b8fdc-6gsn9"] Oct 31 13:45:07.609498 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:07.609473 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5452fc-768b7c485f-z66t4" Oct 31 13:45:07.732982 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:07.732950 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55afb55f-1e19-420a-a30e-7c14e5082e97-kube-api-access-tzx6l\" (UniqueName: \"kubernetes.io/projected/55afb55f-1e19-420a-a30e-7c14e5082e97-kube-api-access-tzx6l\") pod \"55afb55f-1e19-420a-a30e-7c14e5082e97\" (UID: \"55afb55f-1e19-420a-a30e-7c14e5082e97\") " Oct 31 13:45:07.733195 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:07.733047 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55afb55f-1e19-420a-a30e-7c14e5082e97-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55afb55f-1e19-420a-a30e-7c14e5082e97-plugins-config-volume\") pod \"55afb55f-1e19-420a-a30e-7c14e5082e97\" (UID: \"55afb55f-1e19-420a-a30e-7c14e5082e97\") " Oct 31 13:45:07.733459 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:07.733432 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55afb55f-1e19-420a-a30e-7c14e5082e97-plugins-config-volume" pod "55afb55f-1e19-420a-a30e-7c14e5082e97" (UID: "55afb55f-1e19-420a-a30e-7c14e5082e97"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:07.735519 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:07.735495 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55afb55f-1e19-420a-a30e-7c14e5082e97-kube-api-access-tzx6l" pod "55afb55f-1e19-420a-a30e-7c14e5082e97" (UID: "55afb55f-1e19-420a-a30e-7c14e5082e97"). InnerVolumeSpecName "kube-api-access-tzx6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:07.835050 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:07.834959 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzx6l\" (UniqueName: \"kubernetes.io/projected/55afb55f-1e19-420a-a30e-7c14e5082e97-kube-api-access-tzx6l\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:07.835050 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:07.834992 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55afb55f-1e19-420a-a30e-7c14e5082e97-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:07.863765 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:07.863730 5693 generic.go:356] "Generic (PLEG): container finished" podID="55afb55f-1e19-420a-a30e-7c14e5082e97" containerID="3c487137982fc2a6d565cfcbf2d149751f9089137b1407903adc5f114104d02e" exitCode=0 Oct 31 13:45:07.863937 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:07.863774 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5452fc-768b7c485f-z66t4" Oct 31 13:45:07.863937 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:07.863826 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5452fc-768b7c485f-z66t4" event={"ID":"55afb55f-1e19-420a-a30e-7c14e5082e97","Type":"ContainerDied","Data":"3c487137982fc2a6d565cfcbf2d149751f9089137b1407903adc5f114104d02e"} Oct 31 13:45:07.863937 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:07.863877 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5452fc-768b7c485f-z66t4" event={"ID":"55afb55f-1e19-420a-a30e-7c14e5082e97","Type":"ContainerDied","Data":"369384b6952962f1f80d03073eef7bbdd7bb20c205bbfc115738ee2cff148b74"} Oct 31 13:45:07.863937 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:07.863898 5693 scope.go:122] "RemoveContainer" containerID="3c487137982fc2a6d565cfcbf2d149751f9089137b1407903adc5f114104d02e" Oct 31 13:45:07.873900 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:07.873862 5693 scope.go:122] "RemoveContainer" containerID="3c487137982fc2a6d565cfcbf2d149751f9089137b1407903adc5f114104d02e" Oct 31 13:45:07.874158 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:45:07.874136 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c487137982fc2a6d565cfcbf2d149751f9089137b1407903adc5f114104d02e\": container with ID starting with 3c487137982fc2a6d565cfcbf2d149751f9089137b1407903adc5f114104d02e not found: ID does not exist" containerID="3c487137982fc2a6d565cfcbf2d149751f9089137b1407903adc5f114104d02e" Oct 31 13:45:07.874225 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:07.874165 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c487137982fc2a6d565cfcbf2d149751f9089137b1407903adc5f114104d02e"} err="failed to get container status \"3c487137982fc2a6d565cfcbf2d149751f9089137b1407903adc5f114104d02e\": rpc error: code = NotFound desc = could not find container \"3c487137982fc2a6d565cfcbf2d149751f9089137b1407903adc5f114104d02e\": container with ID starting with 3c487137982fc2a6d565cfcbf2d149751f9089137b1407903adc5f114104d02e not found: ID does not exist" Oct 31 13:45:07.924426 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:07.924383 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5452fc-768b7c485f-z66t4"] Oct 31 13:45:07.945952 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:07.945922 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5452fc-768b7c485f-z66t4"] Oct 31 13:45:08.201548 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.201504 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4f3bc9f-5tf48"] Oct 31 13:45:08.202042 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202027 5693 cpu_manager.go:414] "containerMap: removing container" podUID="0191a8b7-5eee-4437-aa2b-c228014966cd" containerName="main" Oct 31 13:45:08.202092 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202044 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0191a8b7-5eee-4437-aa2b-c228014966cd" containerName="main" Oct 31 13:45:08.202092 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202055 5693 cpu_manager.go:414] "containerMap: removing container" podUID="bf6026f7-85c0-4659-8888-e4532b55f73a" containerName="payload-pre-processing" Oct 31 13:45:08.202092 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202060 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf6026f7-85c0-4659-8888-e4532b55f73a" containerName="payload-pre-processing" Oct 31 13:45:08.202092 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202074 5693 cpu_manager.go:414] "containerMap: removing container" podUID="f31ac998-2cc1-482d-8bbe-32ec51bc5dd1" containerName="maas-api" Oct 31 13:45:08.202092 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202078 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f31ac998-2cc1-482d-8bbe-32ec51bc5dd1" containerName="maas-api" Oct 31 13:45:08.202092 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202085 5693 cpu_manager.go:414] "containerMap: removing container" podUID="942b3c70-0134-49ee-a30e-9631e318b001" containerName="maas-api" Oct 31 13:45:08.202092 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202089 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="942b3c70-0134-49ee-a30e-9631e318b001" containerName="maas-api" Oct 31 13:45:08.202322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202095 5693 cpu_manager.go:414] "containerMap: removing container" podUID="2b31a91e-a876-450d-b35c-d5134eb09c1c" containerName="payload-pre-processing" Oct 31 13:45:08.202322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202100 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b31a91e-a876-450d-b35c-d5134eb09c1c" containerName="payload-pre-processing" Oct 31 13:45:08.202322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202110 5693 cpu_manager.go:414] "containerMap: removing container" podUID="a0257070-74ef-4b97-acd3-ad2a93fe5342" containerName="payload-pre-processing" Oct 31 13:45:08.202322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202114 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0257070-74ef-4b97-acd3-ad2a93fe5342" containerName="payload-pre-processing" Oct 31 13:45:08.202322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202121 5693 cpu_manager.go:414] "containerMap: removing container" podUID="bedfdf67-dd1e-4902-9ca8-4f404d3a0841" containerName="istio-proxy" Oct 31 13:45:08.202322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202125 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bedfdf67-dd1e-4902-9ca8-4f404d3a0841" containerName="istio-proxy" Oct 31 13:45:08.202322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202131 5693 cpu_manager.go:414] "containerMap: removing container" podUID="d6d2c321-1fae-4360-b6fa-c11cf95fece6" containerName="payload-pre-processing" Oct 31 13:45:08.202322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202136 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6d2c321-1fae-4360-b6fa-c11cf95fece6" containerName="payload-pre-processing" Oct 31 13:45:08.202322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202147 5693 cpu_manager.go:414] "containerMap: removing container" podUID="17d56bfd-8cb1-42bf-bb11-05810c46d469" containerName="main" Oct 31 13:45:08.202322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202152 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17d56bfd-8cb1-42bf-bb11-05810c46d469" containerName="main" Oct 31 13:45:08.202322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202159 5693 cpu_manager.go:414] "containerMap: removing container" podUID="8fbf694c-e0a8-4a8a-a214-a06c58f7642b" containerName="payload-pre-processing" Oct 31 13:45:08.202322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202164 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fbf694c-e0a8-4a8a-a214-a06c58f7642b" containerName="payload-pre-processing" Oct 31 13:45:08.202322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202171 5693 cpu_manager.go:414] "containerMap: removing container" podUID="710a52fa-2bc4-4bb9-96be-4b08dc60de67" containerName="main" Oct 31 13:45:08.202322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202176 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="710a52fa-2bc4-4bb9-96be-4b08dc60de67" containerName="main" Oct 31 13:45:08.202322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202182 5693 cpu_manager.go:414] "containerMap: removing container" podUID="430e7981-8421-4898-8bef-fba2efcb1676" containerName="payload-processing" Oct 31 13:45:08.202322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202186 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="430e7981-8421-4898-8bef-fba2efcb1676" containerName="payload-processing" Oct 31 13:45:08.202322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202192 5693 cpu_manager.go:414] "containerMap: removing container" podUID="421c561e-1dcd-46b7-85bf-1416e1e007cd" containerName="maas-api" Oct 31 13:45:08.202322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202196 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="421c561e-1dcd-46b7-85bf-1416e1e007cd" containerName="maas-api" Oct 31 13:45:08.202322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202204 5693 cpu_manager.go:414] "containerMap: removing container" podUID="03b3ba33-5c21-487d-bee4-abc978d8cf06" containerName="payload-processing" Oct 31 13:45:08.202322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202208 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03b3ba33-5c21-487d-bee4-abc978d8cf06" containerName="payload-processing" Oct 31 13:45:08.202322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202215 5693 cpu_manager.go:414] "containerMap: removing container" podUID="29da7f75-0ee1-4981-8ee3-b23f46a67876" containerName="payload-pre-processing" Oct 31 13:45:08.202322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202219 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29da7f75-0ee1-4981-8ee3-b23f46a67876" containerName="payload-pre-processing" Oct 31 13:45:08.202322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202226 5693 cpu_manager.go:414] "containerMap: removing container" podUID="fb054e25-5c0a-4525-a337-1a29d0c7fe0a" containerName="istio-proxy" Oct 31 13:45:08.202322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202231 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb054e25-5c0a-4525-a337-1a29d0c7fe0a" containerName="istio-proxy" Oct 31 13:45:08.202322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202236 5693 cpu_manager.go:414] "containerMap: removing container" podUID="f5d9b1d1-f122-4108-9e23-d8d6eff64bc7" containerName="istio-proxy" Oct 31 13:45:08.202322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202241 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5d9b1d1-f122-4108-9e23-d8d6eff64bc7" containerName="istio-proxy" Oct 31 13:45:08.202322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202276 5693 cpu_manager.go:414] "containerMap: removing container" podUID="6b617953-8a53-49e1-bc81-80d029d098bd" containerName="payload-processing" Oct 31 13:45:08.202322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202281 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b617953-8a53-49e1-bc81-80d029d098bd" containerName="payload-processing" Oct 31 13:45:08.202322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202286 5693 cpu_manager.go:414] "containerMap: removing container" podUID="c2ca0d7c-7fb8-438d-9c27-09c1602339fa" containerName="payload-pre-processing" Oct 31 13:45:08.202322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202291 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2ca0d7c-7fb8-438d-9c27-09c1602339fa" containerName="payload-pre-processing" Oct 31 13:45:08.202322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202296 5693 cpu_manager.go:414] "containerMap: removing container" podUID="9fe38c78-fe1a-4633-bbc7-696b2e517ea8" containerName="istio-proxy" Oct 31 13:45:08.202322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202301 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fe38c78-fe1a-4633-bbc7-696b2e517ea8" containerName="istio-proxy" Oct 31 13:45:08.202322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202306 5693 cpu_manager.go:414] "containerMap: removing container" podUID="1a2abd3f-fb4f-4990-9a32-5f945a588068" containerName="main" Oct 31 13:45:08.202322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202310 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a2abd3f-fb4f-4990-9a32-5f945a588068" containerName="main" Oct 31 13:45:08.202322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202315 5693 cpu_manager.go:414] "containerMap: removing container" podUID="eccfdc26-fb5e-4d07-8f67-43723a93450b" containerName="payload-processing" Oct 31 13:45:08.202322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202319 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eccfdc26-fb5e-4d07-8f67-43723a93450b" containerName="payload-processing" Oct 31 13:45:08.202322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202327 5693 cpu_manager.go:414] "containerMap: removing container" podUID="55afb55f-1e19-420a-a30e-7c14e5082e97" containerName="payload-pre-processing" Oct 31 13:45:08.202322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202332 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55afb55f-1e19-420a-a30e-7c14e5082e97" containerName="payload-pre-processing" Oct 31 13:45:08.202322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202339 5693 cpu_manager.go:414] "containerMap: removing container" podUID="32747ebc-c185-4a7f-877a-f4903862b567" containerName="maas-api" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202344 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32747ebc-c185-4a7f-877a-f4903862b567" containerName="maas-api" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202355 5693 cpu_manager.go:414] "containerMap: removing container" podUID="3cc7b994-8cee-47e2-9a6d-b587e3ac1eef" containerName="main" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202360 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cc7b994-8cee-47e2-9a6d-b587e3ac1eef" containerName="main" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202367 5693 cpu_manager.go:414] "containerMap: removing container" podUID="5455d8fe-86dd-413e-a001-80b5e91c7414" containerName="istio-proxy" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202371 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5455d8fe-86dd-413e-a001-80b5e91c7414" containerName="istio-proxy" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202378 5693 cpu_manager.go:414] "containerMap: removing container" podUID="08846306-817b-4ccf-8ec3-470e1bba6257" containerName="payload-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202382 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08846306-817b-4ccf-8ec3-470e1bba6257" containerName="payload-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202389 5693 cpu_manager.go:414] "containerMap: removing container" podUID="ff9a9909-6fc3-475b-a9fe-562ec150c76a" containerName="main" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202393 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff9a9909-6fc3-475b-a9fe-562ec150c76a" containerName="main" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202399 5693 cpu_manager.go:414] "containerMap: removing container" podUID="1ad184ee-c710-47b0-9d44-b4bbcb5487e5" containerName="payload-pre-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202403 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ad184ee-c710-47b0-9d44-b4bbcb5487e5" containerName="payload-pre-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202409 5693 cpu_manager.go:414] "containerMap: removing container" podUID="5458e7f2-d785-4a4e-9b45-e5a9736b6758" containerName="payload-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202413 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5458e7f2-d785-4a4e-9b45-e5a9736b6758" containerName="payload-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202418 5693 cpu_manager.go:414] "containerMap: removing container" podUID="b21df5f3-b4bb-4e47-9287-b42befdf81f1" containerName="payload-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202422 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b21df5f3-b4bb-4e47-9287-b42befdf81f1" containerName="payload-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202429 5693 cpu_manager.go:414] "containerMap: removing container" podUID="e18dd291-4694-41ad-922c-8852d7c12960" containerName="payload-pre-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202433 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e18dd291-4694-41ad-922c-8852d7c12960" containerName="payload-pre-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202438 5693 cpu_manager.go:414] "containerMap: removing container" podUID="d850f787-64ff-4a6b-8534-a93329e45929" containerName="payload-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202442 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d850f787-64ff-4a6b-8534-a93329e45929" containerName="payload-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202450 5693 cpu_manager.go:414] "containerMap: removing container" podUID="4de1aea2-a2ee-4257-8255-e63c353af739" containerName="payload-pre-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202455 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4de1aea2-a2ee-4257-8255-e63c353af739" containerName="payload-pre-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202465 5693 cpu_manager.go:414] "containerMap: removing container" podUID="dd8946ab-9867-4565-98e0-9d2c62eaff29" containerName="istio-proxy" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202469 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd8946ab-9867-4565-98e0-9d2c62eaff29" containerName="istio-proxy" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202475 5693 cpu_manager.go:414] "containerMap: removing container" podUID="a1b0fe6e-146d-4f30-90c2-d44f49d6e17f" containerName="payload-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202479 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1b0fe6e-146d-4f30-90c2-d44f49d6e17f" containerName="payload-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202491 5693 cpu_manager.go:414] "containerMap: removing container" podUID="ddfd9d12-e8bb-42ee-a375-aba470b89eeb" containerName="istio-proxy" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202495 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddfd9d12-e8bb-42ee-a375-aba470b89eeb" containerName="istio-proxy" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202502 5693 cpu_manager.go:414] "containerMap: removing container" podUID="3dc3c2a0-104c-4a6e-a708-9b212dc74267" containerName="payload-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202506 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dc3c2a0-104c-4a6e-a708-9b212dc74267" containerName="payload-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202512 5693 cpu_manager.go:414] "containerMap: removing container" podUID="e87aa817-1a1d-4267-bd3f-0d74c9a229fd" containerName="payload-pre-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202516 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e87aa817-1a1d-4267-bd3f-0d74c9a229fd" containerName="payload-pre-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202586 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a2abd3f-fb4f-4990-9a32-5f945a588068" containerName="main" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202594 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="eccfdc26-fb5e-4d07-8f67-43723a93450b" containerName="payload-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202599 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="55afb55f-1e19-420a-a30e-7c14e5082e97" containerName="payload-pre-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202604 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="32747ebc-c185-4a7f-877a-f4903862b567" containerName="maas-api" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202612 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cc7b994-8cee-47e2-9a6d-b587e3ac1eef" containerName="main" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202617 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="5455d8fe-86dd-413e-a001-80b5e91c7414" containerName="istio-proxy" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202622 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="08846306-817b-4ccf-8ec3-470e1bba6257" containerName="payload-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202628 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff9a9909-6fc3-475b-a9fe-562ec150c76a" containerName="main" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202635 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ad184ee-c710-47b0-9d44-b4bbcb5487e5" containerName="payload-pre-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202655 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="5458e7f2-d785-4a4e-9b45-e5a9736b6758" containerName="payload-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202661 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="b21df5f3-b4bb-4e47-9287-b42befdf81f1" containerName="payload-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202667 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="e18dd291-4694-41ad-922c-8852d7c12960" containerName="payload-pre-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202671 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="d850f787-64ff-4a6b-8534-a93329e45929" containerName="payload-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202677 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="4de1aea2-a2ee-4257-8255-e63c353af739" containerName="payload-pre-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202685 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd8946ab-9867-4565-98e0-9d2c62eaff29" containerName="istio-proxy" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202690 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1b0fe6e-146d-4f30-90c2-d44f49d6e17f" containerName="payload-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202697 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddfd9d12-e8bb-42ee-a375-aba470b89eeb" containerName="istio-proxy" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202702 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dc3c2a0-104c-4a6e-a708-9b212dc74267" containerName="payload-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202707 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="e87aa817-1a1d-4267-bd3f-0d74c9a229fd" containerName="payload-pre-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202714 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="0191a8b7-5eee-4437-aa2b-c228014966cd" containerName="main" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202719 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf6026f7-85c0-4659-8888-e4532b55f73a" containerName="payload-pre-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202724 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="f31ac998-2cc1-482d-8bbe-32ec51bc5dd1" containerName="maas-api" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202729 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="942b3c70-0134-49ee-a30e-9631e318b001" containerName="maas-api" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202735 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b31a91e-a876-450d-b35c-d5134eb09c1c" containerName="payload-pre-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202740 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0257070-74ef-4b97-acd3-ad2a93fe5342" containerName="payload-pre-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202745 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="bedfdf67-dd1e-4902-9ca8-4f404d3a0841" containerName="istio-proxy" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202749 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="17d56bfd-8cb1-42bf-bb11-05810c46d469" containerName="main" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202755 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6d2c321-1fae-4360-b6fa-c11cf95fece6" containerName="payload-pre-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202760 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="710a52fa-2bc4-4bb9-96be-4b08dc60de67" containerName="main" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202766 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fbf694c-e0a8-4a8a-a214-a06c58f7642b" containerName="payload-pre-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202771 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="430e7981-8421-4898-8bef-fba2efcb1676" containerName="payload-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202777 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="421c561e-1dcd-46b7-85bf-1416e1e007cd" containerName="maas-api" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202782 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="03b3ba33-5c21-487d-bee4-abc978d8cf06" containerName="payload-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202796 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="29da7f75-0ee1-4981-8ee3-b23f46a67876" containerName="payload-pre-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202802 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb054e25-5c0a-4525-a337-1a29d0c7fe0a" containerName="istio-proxy" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202807 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5d9b1d1-f122-4108-9e23-d8d6eff64bc7" containerName="istio-proxy" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202814 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b617953-8a53-49e1-bc81-80d029d098bd" containerName="payload-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202820 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2ca0d7c-7fb8-438d-9c27-09c1602339fa" containerName="payload-pre-processing" Oct 31 13:45:08.203563 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.202825 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fe38c78-fe1a-4633-bbc7-696b2e517ea8" containerName="istio-proxy" Oct 31 13:45:08.207043 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.207025 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f3bc9f-5tf48" Oct 31 13:45:08.211572 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.211550 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:45:08.211710 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.211578 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:45:08.229042 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.229011 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4f3bc9f-5tf48"] Oct 31 13:45:08.341034 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.341001 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27ece4ce-6746-4405-b30b-65656e868f4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4f3bc9f-5tf48\" (UID: \"27ece4ce-6746-4405-b30b-65656e868f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f3bc9f-5tf48" Oct 31 13:45:08.341034 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.341037 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5848\" (UniqueName: \"kubernetes.io/projected/27ece4ce-6746-4405-b30b-65656e868f4b-kube-api-access-l5848\") pod \"maas-api-revoke-sub-d4f3bc9f-5tf48\" (UID: \"27ece4ce-6746-4405-b30b-65656e868f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f3bc9f-5tf48" Oct 31 13:45:08.341268 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.341240 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27ece4ce-6746-4405-b30b-65656e868f4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4f3bc9f-5tf48\" (UID: \"27ece4ce-6746-4405-b30b-65656e868f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f3bc9f-5tf48" Oct 31 13:45:08.442244 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.442210 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27ece4ce-6746-4405-b30b-65656e868f4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4f3bc9f-5tf48\" (UID: \"27ece4ce-6746-4405-b30b-65656e868f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f3bc9f-5tf48" Oct 31 13:45:08.442244 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.442247 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5848\" (UniqueName: \"kubernetes.io/projected/27ece4ce-6746-4405-b30b-65656e868f4b-kube-api-access-l5848\") pod \"maas-api-revoke-sub-d4f3bc9f-5tf48\" (UID: \"27ece4ce-6746-4405-b30b-65656e868f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f3bc9f-5tf48" Oct 31 13:45:08.442520 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.442359 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27ece4ce-6746-4405-b30b-65656e868f4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4f3bc9f-5tf48\" (UID: \"27ece4ce-6746-4405-b30b-65656e868f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f3bc9f-5tf48" Oct 31 13:45:08.443052 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.443030 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27ece4ce-6746-4405-b30b-65656e868f4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4f3bc9f-5tf48\" (UID: \"27ece4ce-6746-4405-b30b-65656e868f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f3bc9f-5tf48" Oct 31 13:45:08.444860 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.444841 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27ece4ce-6746-4405-b30b-65656e868f4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4f3bc9f-5tf48\" (UID: \"27ece4ce-6746-4405-b30b-65656e868f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f3bc9f-5tf48" Oct 31 13:45:08.461839 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.461752 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5848\" (UniqueName: \"kubernetes.io/projected/27ece4ce-6746-4405-b30b-65656e868f4b-kube-api-access-l5848\") pod \"maas-api-revoke-sub-d4f3bc9f-5tf48\" (UID: \"27ece4ce-6746-4405-b30b-65656e868f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f3bc9f-5tf48" Oct 31 13:45:08.520523 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.520490 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f3bc9f-5tf48" Oct 31 13:45:08.808501 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.808470 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4f3bc9f-5tf48"] Oct 31 13:45:08.810300 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:45:08.810275 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ece4ce_6746_4405_b30b_65656e868f4b.slice/crio-dc36e0b5ae939299bc807660397297d81330f42db11cd7aa4edfcf7d10b65d57 WatchSource:0}: Error finding container dc36e0b5ae939299bc807660397297d81330f42db11cd7aa4edfcf7d10b65d57: Status 404 returned error can't find the container with id dc36e0b5ae939299bc807660397297d81330f42db11cd7aa4edfcf7d10b65d57 Oct 31 13:45:08.872721 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:08.872688 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f3bc9f-5tf48" event={"ID":"27ece4ce-6746-4405-b30b-65656e868f4b","Type":"ContainerStarted","Data":"dc36e0b5ae939299bc807660397297d81330f42db11cd7aa4edfcf7d10b65d57"} Oct 31 13:45:09.409176 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:09.409143 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55afb55f-1e19-420a-a30e-7c14e5082e97" path="/var/lib/kubelet/pods/55afb55f-1e19-420a-a30e-7c14e5082e97/volumes" Oct 31 13:45:09.880204 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:09.880174 5693 generic.go:356] "Generic (PLEG): container finished" podID="27ece4ce-6746-4405-b30b-65656e868f4b" containerID="242c61ad9c4983e7a1310fd6ddded6eacd9bd828c34628c261a30d2cfff86e3a" exitCode=0 Oct 31 13:45:09.880396 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:09.880212 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f3bc9f-5tf48" event={"ID":"27ece4ce-6746-4405-b30b-65656e868f4b","Type":"ContainerDied","Data":"242c61ad9c4983e7a1310fd6ddded6eacd9bd828c34628c261a30d2cfff86e3a"} Oct 31 13:45:11.051489 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:11.051456 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f3bc9f-5tf48" Oct 31 13:45:11.175101 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:11.175065 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27ece4ce-6746-4405-b30b-65656e868f4b-kube-api-access-l5848\" (UniqueName: \"kubernetes.io/projected/27ece4ce-6746-4405-b30b-65656e868f4b-kube-api-access-l5848\") pod \"27ece4ce-6746-4405-b30b-65656e868f4b\" (UID: \"27ece4ce-6746-4405-b30b-65656e868f4b\") " Oct 31 13:45:11.175289 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:11.175217 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27ece4ce-6746-4405-b30b-65656e868f4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27ece4ce-6746-4405-b30b-65656e868f4b-maas-api-xks-ca\") pod \"27ece4ce-6746-4405-b30b-65656e868f4b\" (UID: \"27ece4ce-6746-4405-b30b-65656e868f4b\") " Oct 31 13:45:11.175289 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:11.175244 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27ece4ce-6746-4405-b30b-65656e868f4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27ece4ce-6746-4405-b30b-65656e868f4b-maas-api-openshift-ca\") pod \"27ece4ce-6746-4405-b30b-65656e868f4b\" (UID: \"27ece4ce-6746-4405-b30b-65656e868f4b\") " Oct 31 13:45:11.175700 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:11.175671 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27ece4ce-6746-4405-b30b-65656e868f4b-maas-api-openshift-ca" pod "27ece4ce-6746-4405-b30b-65656e868f4b" (UID: "27ece4ce-6746-4405-b30b-65656e868f4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:11.177372 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:11.177348 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ece4ce-6746-4405-b30b-65656e868f4b-kube-api-access-l5848" pod "27ece4ce-6746-4405-b30b-65656e868f4b" (UID: "27ece4ce-6746-4405-b30b-65656e868f4b"). InnerVolumeSpecName "kube-api-access-l5848". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:11.177445 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:11.177367 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27ece4ce-6746-4405-b30b-65656e868f4b-maas-api-xks-ca" pod "27ece4ce-6746-4405-b30b-65656e868f4b" (UID: "27ece4ce-6746-4405-b30b-65656e868f4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:45:11.276576 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:11.276544 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5848\" (UniqueName: \"kubernetes.io/projected/27ece4ce-6746-4405-b30b-65656e868f4b-kube-api-access-l5848\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:11.276576 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:11.276573 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27ece4ce-6746-4405-b30b-65656e868f4b-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:11.276576 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:11.276583 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27ece4ce-6746-4405-b30b-65656e868f4b-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:11.893051 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:11.893022 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f3bc9f-5tf48" Oct 31 13:45:11.893241 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:11.893058 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f3bc9f-5tf48" event={"ID":"27ece4ce-6746-4405-b30b-65656e868f4b","Type":"ContainerDied","Data":"dc36e0b5ae939299bc807660397297d81330f42db11cd7aa4edfcf7d10b65d57"} Oct 31 13:45:11.893241 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:11.893101 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc36e0b5ae939299bc807660397297d81330f42db11cd7aa4edfcf7d10b65d57" Oct 31 13:45:12.007019 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:12.006973 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-47bcd9-66cc57988c-9hldq" podUID="f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0" containerName="payload-processing" containerID="cri-o://d712642f3c9c668fe14e4457f173f326aa0d84e509810c1def44ce9230e7c7ea" gracePeriod=25 Oct 31 13:45:12.028482 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:12.028440 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-47bcd9-5c675b8fdc-6gsn9" podUID="c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5" containerName="payload-pre-processing" containerID="cri-o://2b47cff06188d7e60fa1c9502cf828c068352e0567da15475faff2bedbd1f67b" gracePeriod=25 Oct 31 13:45:18.687403 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:45:18.687361 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:45:21.195510 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:21.195476 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8"] Oct 31 13:45:21.196014 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:21.195744 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8" podUID="f6421f4c-ffa6-4da2-91b6-81f2cb8ab421" containerName="maas-api" containerID="cri-o://93bb269d0359be794bae4b5171616da4ef00e70baab9a40ec8abaf5c4047e283" gracePeriod=30 Oct 31 13:45:21.332309 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:21.332273 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-47bcd9-66cc57988c-9hldq" podUID="f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:45:21.332309 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:21.332302 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-47bcd9-5c675b8fdc-6gsn9" podUID="c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:45:21.493847 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:21.493823 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8" Oct 31 13:45:21.529336 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:21.529301 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6421f4c-ffa6-4da2-91b6-81f2cb8ab421-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6421f4c-ffa6-4da2-91b6-81f2cb8ab421-metrics-certs\") pod \"f6421f4c-ffa6-4da2-91b6-81f2cb8ab421\" (UID: \"f6421f4c-ffa6-4da2-91b6-81f2cb8ab421\") " Oct 31 13:45:21.529507 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:21.529361 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6421f4c-ffa6-4da2-91b6-81f2cb8ab421-kube-api-access-fvq9j\" (UniqueName: \"kubernetes.io/projected/f6421f4c-ffa6-4da2-91b6-81f2cb8ab421-kube-api-access-fvq9j\") pod \"f6421f4c-ffa6-4da2-91b6-81f2cb8ab421\" (UID: \"f6421f4c-ffa6-4da2-91b6-81f2cb8ab421\") " Oct 31 13:45:21.529507 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:21.529431 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6421f4c-ffa6-4da2-91b6-81f2cb8ab421-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6421f4c-ffa6-4da2-91b6-81f2cb8ab421-openshift-service-ca\") pod \"f6421f4c-ffa6-4da2-91b6-81f2cb8ab421\" (UID: \"f6421f4c-ffa6-4da2-91b6-81f2cb8ab421\") " Oct 31 13:45:21.529658 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:21.529542 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6421f4c-ffa6-4da2-91b6-81f2cb8ab421-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6421f4c-ffa6-4da2-91b6-81f2cb8ab421-maas-api-tls\") pod \"f6421f4c-ffa6-4da2-91b6-81f2cb8ab421\" (UID: \"f6421f4c-ffa6-4da2-91b6-81f2cb8ab421\") " Oct 31 13:45:21.529921 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:21.529877 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6421f4c-ffa6-4da2-91b6-81f2cb8ab421-openshift-service-ca" pod "f6421f4c-ffa6-4da2-91b6-81f2cb8ab421" (UID: "f6421f4c-ffa6-4da2-91b6-81f2cb8ab421"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:21.530089 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:21.530059 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6421f4c-ffa6-4da2-91b6-81f2cb8ab421-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:21.531741 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:21.531668 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6421f4c-ffa6-4da2-91b6-81f2cb8ab421-metrics-certs" pod "f6421f4c-ffa6-4da2-91b6-81f2cb8ab421" (UID: "f6421f4c-ffa6-4da2-91b6-81f2cb8ab421"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:45:21.531914 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:21.531892 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6421f4c-ffa6-4da2-91b6-81f2cb8ab421-kube-api-access-fvq9j" pod "f6421f4c-ffa6-4da2-91b6-81f2cb8ab421" (UID: "f6421f4c-ffa6-4da2-91b6-81f2cb8ab421"). InnerVolumeSpecName "kube-api-access-fvq9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:21.532138 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:21.532117 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6421f4c-ffa6-4da2-91b6-81f2cb8ab421-maas-api-tls" pod "f6421f4c-ffa6-4da2-91b6-81f2cb8ab421" (UID: "f6421f4c-ffa6-4da2-91b6-81f2cb8ab421"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:45:21.631444 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:21.631414 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6421f4c-ffa6-4da2-91b6-81f2cb8ab421-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:21.631444 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:21.631445 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6421f4c-ffa6-4da2-91b6-81f2cb8ab421-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:21.631684 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:21.631456 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvq9j\" (UniqueName: \"kubernetes.io/projected/f6421f4c-ffa6-4da2-91b6-81f2cb8ab421-kube-api-access-fvq9j\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:21.954886 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:21.954855 5693 generic.go:356] "Generic (PLEG): container finished" podID="f6421f4c-ffa6-4da2-91b6-81f2cb8ab421" containerID="93bb269d0359be794bae4b5171616da4ef00e70baab9a40ec8abaf5c4047e283" exitCode=0 Oct 31 13:45:21.955078 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:21.954909 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8" Oct 31 13:45:21.955078 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:21.954936 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8" event={"ID":"f6421f4c-ffa6-4da2-91b6-81f2cb8ab421","Type":"ContainerDied","Data":"93bb269d0359be794bae4b5171616da4ef00e70baab9a40ec8abaf5c4047e283"} Oct 31 13:45:21.955078 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:21.955019 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8" event={"ID":"f6421f4c-ffa6-4da2-91b6-81f2cb8ab421","Type":"ContainerDied","Data":"b8ff809d970073e6def9eb6ded883d30b255edbd2a2aad5cda9209d6b931f555"} Oct 31 13:45:21.955212 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:21.955118 5693 scope.go:122] "RemoveContainer" containerID="93bb269d0359be794bae4b5171616da4ef00e70baab9a40ec8abaf5c4047e283" Oct 31 13:45:21.966093 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:21.966074 5693 scope.go:122] "RemoveContainer" containerID="93bb269d0359be794bae4b5171616da4ef00e70baab9a40ec8abaf5c4047e283" Oct 31 13:45:21.968406 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:45:21.968337 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93bb269d0359be794bae4b5171616da4ef00e70baab9a40ec8abaf5c4047e283\": container with ID starting with 93bb269d0359be794bae4b5171616da4ef00e70baab9a40ec8abaf5c4047e283 not found: ID does not exist" containerID="93bb269d0359be794bae4b5171616da4ef00e70baab9a40ec8abaf5c4047e283" Oct 31 13:45:21.968406 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:21.968376 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93bb269d0359be794bae4b5171616da4ef00e70baab9a40ec8abaf5c4047e283"} err="failed to get container status \"93bb269d0359be794bae4b5171616da4ef00e70baab9a40ec8abaf5c4047e283\": rpc error: code = NotFound desc = could not find container \"93bb269d0359be794bae4b5171616da4ef00e70baab9a40ec8abaf5c4047e283\": container with ID starting with 93bb269d0359be794bae4b5171616da4ef00e70baab9a40ec8abaf5c4047e283 not found: ID does not exist" Oct 31 13:45:21.978417 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:21.978394 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8"] Oct 31 13:45:21.984495 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:21.984472 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-47bcd9-59f85b98ff-kghb8"] Oct 31 13:45:23.407198 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:23.407163 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6421f4c-ffa6-4da2-91b6-81f2cb8ab421" path="/var/lib/kubelet/pods/f6421f4c-ffa6-4da2-91b6-81f2cb8ab421/volumes" Oct 31 13:45:31.331905 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:31.331866 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-47bcd9-5c675b8fdc-6gsn9" podUID="c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:45:31.332459 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:31.331931 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-47bcd9-66cc57988c-9hldq" podUID="f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:45:32.322006 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:32.321982 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-47bcd9-66cc57988c-9hldq" Oct 31 13:45:32.339961 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:32.339936 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-47bcd9-5c675b8fdc-6gsn9" Oct 31 13:45:32.448928 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:32.448896 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0-plugins-config-volume\") pod \"f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0\" (UID: \"f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0\") " Oct 31 13:45:32.449114 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:32.449020 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0-kube-api-access-fzxs2\" (UniqueName: \"kubernetes.io/projected/f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0-kube-api-access-fzxs2\") pod \"f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0\" (UID: \"f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0\") " Oct 31 13:45:32.449158 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:32.449125 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5-kube-api-access-nht6n\" (UniqueName: \"kubernetes.io/projected/c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5-kube-api-access-nht6n\") pod \"c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5\" (UID: \"c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5\") " Oct 31 13:45:32.449205 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:32.449181 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5-plugins-config-volume\") pod \"c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5\" (UID: \"c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5\") " Oct 31 13:45:32.449283 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:32.449255 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0-plugins-config-volume" pod "f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0" (UID: "f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:32.449590 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:32.449560 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:32.449590 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:32.449569 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5-plugins-config-volume" pod "c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5" (UID: "c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:32.451332 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:32.451309 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5-kube-api-access-nht6n" pod "c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5" (UID: "c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5"). InnerVolumeSpecName "kube-api-access-nht6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:32.451454 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:32.451409 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0-kube-api-access-fzxs2" pod "f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0" (UID: "f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0"). InnerVolumeSpecName "kube-api-access-fzxs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:32.550804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:32.550725 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nht6n\" (UniqueName: \"kubernetes.io/projected/c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5-kube-api-access-nht6n\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:32.550804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:32.550751 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:32.550804 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:32.550760 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzxs2\" (UniqueName: \"kubernetes.io/projected/f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0-kube-api-access-fzxs2\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:33.029519 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:33.029481 5693 generic.go:356] "Generic (PLEG): container finished" podID="c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5" containerID="2b47cff06188d7e60fa1c9502cf828c068352e0567da15475faff2bedbd1f67b" exitCode=0 Oct 31 13:45:33.029703 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:33.029533 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-47bcd9-5c675b8fdc-6gsn9" Oct 31 13:45:33.029703 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:33.029567 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-47bcd9-5c675b8fdc-6gsn9" event={"ID":"c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5","Type":"ContainerDied","Data":"2b47cff06188d7e60fa1c9502cf828c068352e0567da15475faff2bedbd1f67b"} Oct 31 13:45:33.029703 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:33.029626 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-47bcd9-5c675b8fdc-6gsn9" event={"ID":"c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5","Type":"ContainerDied","Data":"e786140e74e77031f8bc4765b98039709dabc64ce737b8f0ab576265b63e0e85"} Oct 31 13:45:33.029703 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:33.029675 5693 scope.go:122] "RemoveContainer" containerID="2b47cff06188d7e60fa1c9502cf828c068352e0567da15475faff2bedbd1f67b" Oct 31 13:45:33.031075 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:33.031051 5693 generic.go:356] "Generic (PLEG): container finished" podID="f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0" containerID="d712642f3c9c668fe14e4457f173f326aa0d84e509810c1def44ce9230e7c7ea" exitCode=0 Oct 31 13:45:33.031248 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:33.031103 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-47bcd9-66cc57988c-9hldq" Oct 31 13:45:33.031248 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:33.031128 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-47bcd9-66cc57988c-9hldq" event={"ID":"f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0","Type":"ContainerDied","Data":"d712642f3c9c668fe14e4457f173f326aa0d84e509810c1def44ce9230e7c7ea"} Oct 31 13:45:33.031376 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:33.031266 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-47bcd9-66cc57988c-9hldq" event={"ID":"f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0","Type":"ContainerDied","Data":"2c49f3b87fc84423f317da073fbe43ee521a9feadb86dfa34d730efe358378f5"} Oct 31 13:45:33.041102 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:33.040142 5693 scope.go:122] "RemoveContainer" containerID="2b47cff06188d7e60fa1c9502cf828c068352e0567da15475faff2bedbd1f67b" Oct 31 13:45:33.041102 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:45:33.040516 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b47cff06188d7e60fa1c9502cf828c068352e0567da15475faff2bedbd1f67b\": container with ID starting with 2b47cff06188d7e60fa1c9502cf828c068352e0567da15475faff2bedbd1f67b not found: ID does not exist" containerID="2b47cff06188d7e60fa1c9502cf828c068352e0567da15475faff2bedbd1f67b" Oct 31 13:45:33.041102 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:33.040545 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b47cff06188d7e60fa1c9502cf828c068352e0567da15475faff2bedbd1f67b"} err="failed to get container status \"2b47cff06188d7e60fa1c9502cf828c068352e0567da15475faff2bedbd1f67b\": rpc error: code = NotFound desc = could not find container \"2b47cff06188d7e60fa1c9502cf828c068352e0567da15475faff2bedbd1f67b\": container with ID starting with 2b47cff06188d7e60fa1c9502cf828c068352e0567da15475faff2bedbd1f67b not found: ID does not exist" Oct 31 13:45:33.041102 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:33.040568 5693 scope.go:122] "RemoveContainer" containerID="d712642f3c9c668fe14e4457f173f326aa0d84e509810c1def44ce9230e7c7ea" Oct 31 13:45:33.050616 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:33.050599 5693 scope.go:122] "RemoveContainer" containerID="d712642f3c9c668fe14e4457f173f326aa0d84e509810c1def44ce9230e7c7ea" Oct 31 13:45:33.050915 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:45:33.050896 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d712642f3c9c668fe14e4457f173f326aa0d84e509810c1def44ce9230e7c7ea\": container with ID starting with d712642f3c9c668fe14e4457f173f326aa0d84e509810c1def44ce9230e7c7ea not found: ID does not exist" containerID="d712642f3c9c668fe14e4457f173f326aa0d84e509810c1def44ce9230e7c7ea" Oct 31 13:45:33.050968 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:33.050922 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d712642f3c9c668fe14e4457f173f326aa0d84e509810c1def44ce9230e7c7ea"} err="failed to get container status \"d712642f3c9c668fe14e4457f173f326aa0d84e509810c1def44ce9230e7c7ea\": rpc error: code = NotFound desc = could not find container \"d712642f3c9c668fe14e4457f173f326aa0d84e509810c1def44ce9230e7c7ea\": container with ID starting with d712642f3c9c668fe14e4457f173f326aa0d84e509810c1def44ce9230e7c7ea not found: ID does not exist" Oct 31 13:45:33.059669 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:33.059618 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-47bcd9-5c675b8fdc-6gsn9"] Oct 31 13:45:33.062794 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:33.062776 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-47bcd9-5c675b8fdc-6gsn9"] Oct 31 13:45:33.080280 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:33.080257 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-47bcd9-66cc57988c-9hldq"] Oct 31 13:45:33.082545 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:33.082523 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-47bcd9-66cc57988c-9hldq"] Oct 31 13:45:33.406949 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:33.406921 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5" path="/var/lib/kubelet/pods/c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5/volumes" Oct 31 13:45:33.407345 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:33.407312 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0" path="/var/lib/kubelet/pods/f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0/volumes" Oct 31 13:45:37.988353 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:37.988309 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-842ac9-5d74f7f789-w2rqn"] Oct 31 13:45:38.517509 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.517473 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5"] Oct 31 13:45:38.518196 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.518177 5693 cpu_manager.go:414] "containerMap: removing container" podUID="f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0" containerName="payload-processing" Oct 31 13:45:38.518273 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.518199 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0" containerName="payload-processing" Oct 31 13:45:38.518273 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.518212 5693 cpu_manager.go:414] "containerMap: removing container" podUID="f6421f4c-ffa6-4da2-91b6-81f2cb8ab421" containerName="maas-api" Oct 31 13:45:38.518273 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.518217 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6421f4c-ffa6-4da2-91b6-81f2cb8ab421" containerName="maas-api" Oct 31 13:45:38.518273 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.518239 5693 cpu_manager.go:414] "containerMap: removing container" podUID="c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5" containerName="payload-pre-processing" Oct 31 13:45:38.518273 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.518244 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5" containerName="payload-pre-processing" Oct 31 13:45:38.518273 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.518251 5693 cpu_manager.go:414] "containerMap: removing container" podUID="27ece4ce-6746-4405-b30b-65656e868f4b" containerName="revoke-keys" Oct 31 13:45:38.518273 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.518256 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27ece4ce-6746-4405-b30b-65656e868f4b" containerName="revoke-keys" Oct 31 13:45:38.518501 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.518325 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="c92a5443-7c18-4eb7-b2b5-cfdcfd1e25a5" containerName="payload-pre-processing" Oct 31 13:45:38.518501 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.518358 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="27ece4ce-6746-4405-b30b-65656e868f4b" containerName="revoke-keys" Oct 31 13:45:38.518501 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.518371 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8e5aafa-21a8-4daa-a44d-90a3cc1a74c0" containerName="payload-processing" Oct 31 13:45:38.518501 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.518378 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6421f4c-ffa6-4da2-91b6-81f2cb8ab421" containerName="maas-api" Oct 31 13:45:38.521735 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.521715 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" Oct 31 13:45:38.544082 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.544057 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-1e72f6-openshift-default-dockercfg-b6b7d\"" Oct 31 13:45:38.574172 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.574143 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5"] Oct 31 13:45:38.629160 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.629129 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwkpq\" (UniqueName: \"kubernetes.io/projected/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-kube-api-access-cwkpq\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5\" (UID: \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" Oct 31 13:45:38.629348 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.629174 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-credential-socket\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5\" (UID: \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" Oct 31 13:45:38.629348 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.629265 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-istio-podinfo\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5\" (UID: \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" Oct 31 13:45:38.629348 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.629312 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-workload-certs\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5\" (UID: \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" Oct 31 13:45:38.629463 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.629393 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-istio-data\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5\" (UID: \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" Oct 31 13:45:38.629463 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.629439 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-istio-envoy\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5\" (UID: \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" Oct 31 13:45:38.629463 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.629454 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-istiod-ca-cert\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5\" (UID: \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" Oct 31 13:45:38.629576 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.629478 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-istio-token\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5\" (UID: \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" Oct 31 13:45:38.629576 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.629560 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-workload-socket\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5\" (UID: \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" Oct 31 13:45:38.730890 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.730858 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwkpq\" (UniqueName: \"kubernetes.io/projected/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-kube-api-access-cwkpq\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5\" (UID: \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" Oct 31 13:45:38.731126 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.730901 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-credential-socket\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5\" (UID: \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" Oct 31 13:45:38.731126 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.730943 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-istio-podinfo\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5\" (UID: \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" Oct 31 13:45:38.731126 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.730980 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-workload-certs\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5\" (UID: \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" Oct 31 13:45:38.731126 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.731035 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-istio-data\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5\" (UID: \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" Oct 31 13:45:38.731126 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.731094 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-istio-envoy\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5\" (UID: \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" Oct 31 13:45:38.731126 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.731117 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-istiod-ca-cert\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5\" (UID: \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" Oct 31 13:45:38.731477 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.731156 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-istio-token\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5\" (UID: \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" Oct 31 13:45:38.731477 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.731212 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-workload-socket\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5\" (UID: \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" Oct 31 13:45:38.731477 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.731396 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-istio-data\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5\" (UID: \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" Oct 31 13:45:38.731668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.731532 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-workload-socket\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5\" (UID: \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" Oct 31 13:45:38.731821 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.731793 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-workload-certs\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5\" (UID: \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" Oct 31 13:45:38.732276 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.732251 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-credential-socket\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5\" (UID: \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" Oct 31 13:45:38.732471 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.732271 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-istiod-ca-cert\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5\" (UID: \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" Oct 31 13:45:38.734300 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.734254 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-istio-envoy\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5\" (UID: \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" Oct 31 13:45:38.734966 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.734887 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-istio-podinfo\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5\" (UID: \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" Oct 31 13:45:38.767504 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.767472 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwkpq\" (UniqueName: \"kubernetes.io/projected/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-kube-api-access-cwkpq\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5\" (UID: \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" Oct 31 13:45:38.768926 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.768866 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-istio-token\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5\" (UID: \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" Oct 31 13:45:38.808795 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.808765 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm"] Oct 31 13:45:38.808982 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.808960 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" podUID="a3797b53-75e3-4584-8211-a06b15e03526" containerName="istio-proxy" containerID="cri-o://5a5c583549e48c36d054da710bdec34969d42f08049fde50dfe54c6363a55ea4" gracePeriod=30 Oct 31 13:45:38.834822 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:38.834797 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" Oct 31 13:45:39.141876 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:39.141843 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5"] Oct 31 13:45:39.147345 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:39.147311 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:45:39.147445 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:39.147381 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:45:39.147445 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:39.147416 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:45:40.110709 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:40.110662 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" event={"ID":"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e","Type":"ContainerStarted","Data":"c094d479298ab544e8871572783108ab2c089a8a4d87607557cce4fd6f65d1a4"} Oct 31 13:45:40.110912 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:40.110745 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" event={"ID":"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e","Type":"ContainerStarted","Data":"59ba8a36fb6286e3e1e88eae971c813646275d1dbba21fc40a1961da154969e9"} Oct 31 13:45:40.138300 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:40.138250 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" podStartSLOduration=2.138236517 podStartE2EDuration="2.138236517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:40.136399052 +0000 UTC m=+4390.843985346" watchObservedRunningTime="2026-10-31 13:45:40.138236517 +0000 UTC m=+4390.845822811" Oct 31 13:45:40.835213 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:40.835173 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" Oct 31 13:45:40.837096 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:40.837068 5693 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body= Oct 31 13:45:40.837226 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:40.837136 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" podUID="5ee4d8a4-7509-4ae6-8da5-8015a0712a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" Oct 31 13:45:41.835370 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:41.835331 5693 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body= Oct 31 13:45:41.835803 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:41.835388 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" podUID="5ee4d8a4-7509-4ae6-8da5-8015a0712a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" Oct 31 13:45:42.835361 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:42.835326 5693 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body= Oct 31 13:45:42.835547 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:42.835399 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" podUID="5ee4d8a4-7509-4ae6-8da5-8015a0712a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" Oct 31 13:45:42.994668 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:42.988705 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-842ac9-5d74f7f789-w2rqn" podUID="efeec4f0-9411-4add-8d49-586201474004" containerName="payload-processing" containerID="cri-o://1214bde53139fc4ca67020a820b5c6c27ad78a6dfd75ea99ca6dde757018424b" gracePeriod=25 Oct 31 13:45:43.835966 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:43.835932 5693 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body= Oct 31 13:45:43.836318 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:43.836001 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" podUID="5ee4d8a4-7509-4ae6-8da5-8015a0712a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" Oct 31 13:45:44.128717 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:44.128691 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" Oct 31 13:45:44.138237 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:44.138211 5693 generic.go:356] "Generic (PLEG): container finished" podID="a3797b53-75e3-4584-8211-a06b15e03526" containerID="5a5c583549e48c36d054da710bdec34969d42f08049fde50dfe54c6363a55ea4" exitCode=0 Oct 31 13:45:44.138388 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:44.138244 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" event={"ID":"a3797b53-75e3-4584-8211-a06b15e03526","Type":"ContainerDied","Data":"5a5c583549e48c36d054da710bdec34969d42f08049fde50dfe54c6363a55ea4"} Oct 31 13:45:44.138388 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:44.138287 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" event={"ID":"a3797b53-75e3-4584-8211-a06b15e03526","Type":"ContainerDied","Data":"5ea64016e4ecd34c026284dd7fcc6ffa4a85f6bd55a6de801af3e7e8b8feb0b6"} Oct 31 13:45:44.138388 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:44.138304 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm" Oct 31 13:45:44.138503 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:44.138310 5693 scope.go:122] "RemoveContainer" containerID="5a5c583549e48c36d054da710bdec34969d42f08049fde50dfe54c6363a55ea4" Oct 31 13:45:44.148598 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:44.148575 5693 scope.go:122] "RemoveContainer" containerID="5a5c583549e48c36d054da710bdec34969d42f08049fde50dfe54c6363a55ea4" Oct 31 13:45:44.149125 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:45:44.149107 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a5c583549e48c36d054da710bdec34969d42f08049fde50dfe54c6363a55ea4\": container with ID starting with 5a5c583549e48c36d054da710bdec34969d42f08049fde50dfe54c6363a55ea4 not found: ID does not exist" containerID="5a5c583549e48c36d054da710bdec34969d42f08049fde50dfe54c6363a55ea4" Oct 31 13:45:44.149282 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:44.149256 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a5c583549e48c36d054da710bdec34969d42f08049fde50dfe54c6363a55ea4"} err="failed to get container status \"5a5c583549e48c36d054da710bdec34969d42f08049fde50dfe54c6363a55ea4\": rpc error: code = NotFound desc = could not find container \"5a5c583549e48c36d054da710bdec34969d42f08049fde50dfe54c6363a55ea4\": container with ID starting with 5a5c583549e48c36d054da710bdec34969d42f08049fde50dfe54c6363a55ea4 not found: ID does not exist" Oct 31 13:45:44.227064 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:44.227034 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3797b53-75e3-4584-8211-a06b15e03526-istio-token\" (UniqueName: \"kubernetes.io/projected/a3797b53-75e3-4584-8211-a06b15e03526-istio-token\") pod \"a3797b53-75e3-4584-8211-a06b15e03526\" (UID: \"a3797b53-75e3-4584-8211-a06b15e03526\") " Oct 31 13:45:44.227391 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:44.227320 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3797b53-75e3-4584-8211-a06b15e03526-kube-api-access-cfkrh\" (UniqueName: \"kubernetes.io/projected/a3797b53-75e3-4584-8211-a06b15e03526-kube-api-access-cfkrh\") pod \"a3797b53-75e3-4584-8211-a06b15e03526\" (UID: \"a3797b53-75e3-4584-8211-a06b15e03526\") " Oct 31 13:45:44.227598 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:44.227578 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3797b53-75e3-4584-8211-a06b15e03526-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3797b53-75e3-4584-8211-a06b15e03526-istiod-ca-cert\") pod \"a3797b53-75e3-4584-8211-a06b15e03526\" (UID: \"a3797b53-75e3-4584-8211-a06b15e03526\") " Oct 31 13:45:44.227861 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:44.227843 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3797b53-75e3-4584-8211-a06b15e03526-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3797b53-75e3-4584-8211-a06b15e03526-istio-envoy\") pod \"a3797b53-75e3-4584-8211-a06b15e03526\" (UID: \"a3797b53-75e3-4584-8211-a06b15e03526\") " Oct 31 13:45:44.227990 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:44.227973 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3797b53-75e3-4584-8211-a06b15e03526-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3797b53-75e3-4584-8211-a06b15e03526-workload-certs\") pod \"a3797b53-75e3-4584-8211-a06b15e03526\" (UID: \"a3797b53-75e3-4584-8211-a06b15e03526\") " Oct 31 13:45:44.228145 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:44.228128 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3797b53-75e3-4584-8211-a06b15e03526-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3797b53-75e3-4584-8211-a06b15e03526-istio-podinfo\") pod \"a3797b53-75e3-4584-8211-a06b15e03526\" (UID: \"a3797b53-75e3-4584-8211-a06b15e03526\") " Oct 31 13:45:44.228320 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:44.228175 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3797b53-75e3-4584-8211-a06b15e03526-workload-certs" pod "a3797b53-75e3-4584-8211-a06b15e03526" (UID: "a3797b53-75e3-4584-8211-a06b15e03526"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:44.228394 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:44.228284 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3797b53-75e3-4584-8211-a06b15e03526-istiod-ca-cert" pod "a3797b53-75e3-4584-8211-a06b15e03526" (UID: "a3797b53-75e3-4584-8211-a06b15e03526"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:44.228487 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:44.228469 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3797b53-75e3-4584-8211-a06b15e03526-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3797b53-75e3-4584-8211-a06b15e03526-istio-data\") pod \"a3797b53-75e3-4584-8211-a06b15e03526\" (UID: \"a3797b53-75e3-4584-8211-a06b15e03526\") " Oct 31 13:45:44.228619 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:44.228597 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3797b53-75e3-4584-8211-a06b15e03526-istio-data" pod "a3797b53-75e3-4584-8211-a06b15e03526" (UID: "a3797b53-75e3-4584-8211-a06b15e03526"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:44.228757 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:44.228745 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3797b53-75e3-4584-8211-a06b15e03526-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3797b53-75e3-4584-8211-a06b15e03526-workload-socket\") pod \"a3797b53-75e3-4584-8211-a06b15e03526\" (UID: \"a3797b53-75e3-4584-8211-a06b15e03526\") " Oct 31 13:45:44.229023 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:44.229010 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3797b53-75e3-4584-8211-a06b15e03526-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3797b53-75e3-4584-8211-a06b15e03526-credential-socket\") pod \"a3797b53-75e3-4584-8211-a06b15e03526\" (UID: \"a3797b53-75e3-4584-8211-a06b15e03526\") " Oct 31 13:45:44.229393 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:44.228964 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3797b53-75e3-4584-8211-a06b15e03526-workload-socket" pod "a3797b53-75e3-4584-8211-a06b15e03526" (UID: "a3797b53-75e3-4584-8211-a06b15e03526"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:44.229393 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:44.229280 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3797b53-75e3-4584-8211-a06b15e03526-credential-socket" pod "a3797b53-75e3-4584-8211-a06b15e03526" (UID: "a3797b53-75e3-4584-8211-a06b15e03526"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:44.229601 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:44.229575 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3797b53-75e3-4584-8211-a06b15e03526-istio-token" pod "a3797b53-75e3-4584-8211-a06b15e03526" (UID: "a3797b53-75e3-4584-8211-a06b15e03526"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:44.229936 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:44.229923 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3797b53-75e3-4584-8211-a06b15e03526-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:44.230019 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:44.230010 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3797b53-75e3-4584-8211-a06b15e03526-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:44.230073 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:44.230066 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3797b53-75e3-4584-8211-a06b15e03526-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:44.230165 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:44.230153 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3797b53-75e3-4584-8211-a06b15e03526-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:44.230269 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:44.230261 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3797b53-75e3-4584-8211-a06b15e03526-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:44.230326 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:44.230318 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3797b53-75e3-4584-8211-a06b15e03526-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:44.230488 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:44.230469 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3797b53-75e3-4584-8211-a06b15e03526-istio-envoy" pod "a3797b53-75e3-4584-8211-a06b15e03526" (UID: "a3797b53-75e3-4584-8211-a06b15e03526"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:44.230741 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:44.230710 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3797b53-75e3-4584-8211-a06b15e03526-kube-api-access-cfkrh" pod "a3797b53-75e3-4584-8211-a06b15e03526" (UID: "a3797b53-75e3-4584-8211-a06b15e03526"). InnerVolumeSpecName "kube-api-access-cfkrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:44.230741 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:44.230712 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3797b53-75e3-4584-8211-a06b15e03526-istio-podinfo" pod "a3797b53-75e3-4584-8211-a06b15e03526" (UID: "a3797b53-75e3-4584-8211-a06b15e03526"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:45:44.331790 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:44.331743 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfkrh\" (UniqueName: \"kubernetes.io/projected/a3797b53-75e3-4584-8211-a06b15e03526-kube-api-access-cfkrh\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:44.331790 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:44.331771 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3797b53-75e3-4584-8211-a06b15e03526-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:44.331790 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:44.331788 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3797b53-75e3-4584-8211-a06b15e03526-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:44.465337 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:44.465301 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm"] Oct 31 13:45:44.469781 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:44.469756 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-72mhm"] Oct 31 13:45:44.835830 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:44.835730 5693 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body= Oct 31 13:45:44.835830 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:44.835794 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" podUID="5ee4d8a4-7509-4ae6-8da5-8015a0712a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" Oct 31 13:45:45.407629 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:45.407601 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3797b53-75e3-4584-8211-a06b15e03526" path="/var/lib/kubelet/pods/a3797b53-75e3-4584-8211-a06b15e03526/volumes" Oct 31 13:45:45.637631 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:45.637589 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-842ac9-5d74f7f789-w2rqn" podUID="efeec4f0-9411-4add-8d49-586201474004" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:45:45.839763 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:45.839682 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" Oct 31 13:45:45.840253 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:45.840073 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" Oct 31 13:45:45.840760 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:45.840740 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" Oct 31 13:45:48.688304 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:45:48.688267 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:45:55.637695 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:55.637657 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-842ac9-5d74f7f789-w2rqn" podUID="efeec4f0-9411-4add-8d49-586201474004" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:45:56.082755 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:56.082718 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp"] Oct 31 13:45:56.082954 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:56.082930 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp" podUID="a509360d-4d23-4078-b1d8-053338d5de5f" containerName="maas-api" containerID="cri-o://cb4ca31a80de8933ba82b5ee6a176133aab9c55c551172c0be254c082533e260" gracePeriod=30 Oct 31 13:45:56.207076 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:56.207045 5693 generic.go:356] "Generic (PLEG): container finished" podID="a509360d-4d23-4078-b1d8-053338d5de5f" containerID="cb4ca31a80de8933ba82b5ee6a176133aab9c55c551172c0be254c082533e260" exitCode=0 Oct 31 13:45:56.207209 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:56.207111 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp" event={"ID":"a509360d-4d23-4078-b1d8-053338d5de5f","Type":"ContainerDied","Data":"cb4ca31a80de8933ba82b5ee6a176133aab9c55c551172c0be254c082533e260"} Oct 31 13:45:56.339506 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:56.339437 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp" Oct 31 13:45:56.466032 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:56.466001 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a509360d-4d23-4078-b1d8-053338d5de5f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a509360d-4d23-4078-b1d8-053338d5de5f-metrics-certs\") pod \"a509360d-4d23-4078-b1d8-053338d5de5f\" (UID: \"a509360d-4d23-4078-b1d8-053338d5de5f\") " Oct 31 13:45:56.466243 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:56.466170 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a509360d-4d23-4078-b1d8-053338d5de5f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a509360d-4d23-4078-b1d8-053338d5de5f-maas-api-tls\") pod \"a509360d-4d23-4078-b1d8-053338d5de5f\" (UID: \"a509360d-4d23-4078-b1d8-053338d5de5f\") " Oct 31 13:45:56.466243 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:56.466195 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a509360d-4d23-4078-b1d8-053338d5de5f-kube-api-access-hs9vr\" (UniqueName: \"kubernetes.io/projected/a509360d-4d23-4078-b1d8-053338d5de5f-kube-api-access-hs9vr\") pod \"a509360d-4d23-4078-b1d8-053338d5de5f\" (UID: \"a509360d-4d23-4078-b1d8-053338d5de5f\") " Oct 31 13:45:56.466373 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:56.466243 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a509360d-4d23-4078-b1d8-053338d5de5f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a509360d-4d23-4078-b1d8-053338d5de5f-openshift-service-ca\") pod \"a509360d-4d23-4078-b1d8-053338d5de5f\" (UID: \"a509360d-4d23-4078-b1d8-053338d5de5f\") " Oct 31 13:45:56.466829 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:56.466735 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a509360d-4d23-4078-b1d8-053338d5de5f-openshift-service-ca" pod "a509360d-4d23-4078-b1d8-053338d5de5f" (UID: "a509360d-4d23-4078-b1d8-053338d5de5f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:56.467150 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:56.467128 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a509360d-4d23-4078-b1d8-053338d5de5f-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:56.468522 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:56.468492 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a509360d-4d23-4078-b1d8-053338d5de5f-maas-api-tls" pod "a509360d-4d23-4078-b1d8-053338d5de5f" (UID: "a509360d-4d23-4078-b1d8-053338d5de5f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:45:56.468522 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:56.468512 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a509360d-4d23-4078-b1d8-053338d5de5f-metrics-certs" pod "a509360d-4d23-4078-b1d8-053338d5de5f" (UID: "a509360d-4d23-4078-b1d8-053338d5de5f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:45:56.468806 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:56.468785 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a509360d-4d23-4078-b1d8-053338d5de5f-kube-api-access-hs9vr" pod "a509360d-4d23-4078-b1d8-053338d5de5f" (UID: "a509360d-4d23-4078-b1d8-053338d5de5f"). InnerVolumeSpecName "kube-api-access-hs9vr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:56.568177 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:56.568146 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a509360d-4d23-4078-b1d8-053338d5de5f-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:56.568177 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:56.568175 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs9vr\" (UniqueName: \"kubernetes.io/projected/a509360d-4d23-4078-b1d8-053338d5de5f-kube-api-access-hs9vr\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:56.568177 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:56.568186 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a509360d-4d23-4078-b1d8-053338d5de5f-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:57.214032 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:57.213996 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp" Oct 31 13:45:57.214032 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:57.214016 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp" event={"ID":"a509360d-4d23-4078-b1d8-053338d5de5f","Type":"ContainerDied","Data":"329c1dda45e7f8960a0394b82c5472bb1939383bd22881ec911cdf359d95df8c"} Oct 31 13:45:57.214514 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:57.214112 5693 scope.go:122] "RemoveContainer" containerID="cb4ca31a80de8933ba82b5ee6a176133aab9c55c551172c0be254c082533e260" Oct 31 13:45:57.237600 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:57.237568 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp"] Oct 31 13:45:57.243427 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:57.243404 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-15a6a2-69756c78bf-wtcxp"] Oct 31 13:45:57.407512 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:45:57.407481 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a509360d-4d23-4078-b1d8-053338d5de5f" path="/var/lib/kubelet/pods/a509360d-4d23-4078-b1d8-053338d5de5f/volumes" Oct 31 13:46:03.247855 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:03.247820 5693 generic.go:356] "Generic (PLEG): container finished" podID="efeec4f0-9411-4add-8d49-586201474004" containerID="1214bde53139fc4ca67020a820b5c6c27ad78a6dfd75ea99ca6dde757018424b" exitCode=0 Oct 31 13:46:03.248239 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:03.247897 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-842ac9-5d74f7f789-w2rqn" event={"ID":"efeec4f0-9411-4add-8d49-586201474004","Type":"ContainerDied","Data":"1214bde53139fc4ca67020a820b5c6c27ad78a6dfd75ea99ca6dde757018424b"} Oct 31 13:46:03.277856 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:03.277833 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-842ac9-5d74f7f789-w2rqn" Oct 31 13:46:03.340878 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:03.340845 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efeec4f0-9411-4add-8d49-586201474004-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efeec4f0-9411-4add-8d49-586201474004-plugins-config-volume\") pod \"efeec4f0-9411-4add-8d49-586201474004\" (UID: \"efeec4f0-9411-4add-8d49-586201474004\") " Oct 31 13:46:03.341085 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:03.340920 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efeec4f0-9411-4add-8d49-586201474004-kube-api-access-gbh56\" (UniqueName: \"kubernetes.io/projected/efeec4f0-9411-4add-8d49-586201474004-kube-api-access-gbh56\") pod \"efeec4f0-9411-4add-8d49-586201474004\" (UID: \"efeec4f0-9411-4add-8d49-586201474004\") " Oct 31 13:46:03.341299 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:03.341274 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efeec4f0-9411-4add-8d49-586201474004-plugins-config-volume" pod "efeec4f0-9411-4add-8d49-586201474004" (UID: "efeec4f0-9411-4add-8d49-586201474004"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:46:03.341513 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:03.341485 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efeec4f0-9411-4add-8d49-586201474004-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:03.343400 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:03.343374 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efeec4f0-9411-4add-8d49-586201474004-kube-api-access-gbh56" pod "efeec4f0-9411-4add-8d49-586201474004" (UID: "efeec4f0-9411-4add-8d49-586201474004"). InnerVolumeSpecName "kube-api-access-gbh56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:03.442447 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:03.442417 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbh56\" (UniqueName: \"kubernetes.io/projected/efeec4f0-9411-4add-8d49-586201474004-kube-api-access-gbh56\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:04.254355 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:04.254316 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-842ac9-5d74f7f789-w2rqn" event={"ID":"efeec4f0-9411-4add-8d49-586201474004","Type":"ContainerDied","Data":"62efc5f4520528c9d82c8f60d135b4f15ad24df3de8546348a86af24821fb6fe"} Oct 31 13:46:04.254355 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:04.254339 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-842ac9-5d74f7f789-w2rqn" Oct 31 13:46:04.254889 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:04.254378 5693 scope.go:122] "RemoveContainer" containerID="1214bde53139fc4ca67020a820b5c6c27ad78a6dfd75ea99ca6dde757018424b" Oct 31 13:46:04.272276 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:04.272245 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-842ac9-5d74f7f789-w2rqn"] Oct 31 13:46:04.277900 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:04.277872 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-842ac9-5d74f7f789-w2rqn"] Oct 31 13:46:05.407691 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:05.407657 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efeec4f0-9411-4add-8d49-586201474004" path="/var/lib/kubelet/pods/efeec4f0-9411-4add-8d49-586201474004/volumes" Oct 31 13:46:13.379616 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:13.379400 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-6c0612-85c484d865-fvthd"] Oct 31 13:46:13.396514 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:13.396470 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0612-b5b66c795-fdmgl"] Oct 31 13:46:18.381482 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:18.381441 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c0612-85c484d865-fvthd" podUID="32019af4-f819-4127-9d00-d2582affe985" containerName="payload-processing" containerID="cri-o://e91931e26bf74cfd83aa2fb254db66fa562192155e76391641fe4672e4188f00" gracePeriod=25 Oct 31 13:46:18.397990 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:18.397936 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0612-b5b66c795-fdmgl" podUID="fba7a7df-1586-461c-b3d7-63ddcf53f218" containerName="payload-pre-processing" containerID="cri-o://5a08050267cf453575846e766b0837b98554f1abbfe01bab12d04621c8f9d003" gracePeriod=25 Oct 31 13:46:18.688386 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:46:18.688350 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:46:27.653028 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:27.652984 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0612-b5b66c795-fdmgl" podUID="fba7a7df-1586-461c-b3d7-63ddcf53f218" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:46:27.653561 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:27.652984 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c0612-85c484d865-fvthd" podUID="32019af4-f819-4127-9d00-d2582affe985" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:46:31.161538 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:31.161501 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-1e72f6-ddbd6777-k2fm6"] Oct 31 13:46:31.173905 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:31.173878 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1e72f6-64d95f46cf-wxgkn"] Oct 31 13:46:36.162361 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:36.162314 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1e72f6-ddbd6777-k2fm6" podUID="f8f9a320-4770-43a7-9297-b1d659961376" containerName="payload-processing" containerID="cri-o://5fb9ced1279d5809ac9f354b12bc23f7f1af6721b0212c96fa6fd54a4d8ba989" gracePeriod=25 Oct 31 13:46:36.174551 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:36.174512 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1e72f6-64d95f46cf-wxgkn" podUID="c6ff15bc-cbaa-4bbb-a634-415dc98bbb12" containerName="payload-pre-processing" containerID="cri-o://b70b0035e6482367f73184c3110ea7e48ee0a538ad1899f3b4083988f550040b" gracePeriod=25 Oct 31 13:46:37.653338 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:37.653294 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0612-b5b66c795-fdmgl" podUID="fba7a7df-1586-461c-b3d7-63ddcf53f218" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:46:37.653811 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:37.653302 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c0612-85c484d865-fvthd" podUID="32019af4-f819-4127-9d00-d2582affe985" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:46:38.103543 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:38.103440 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1e72f6-ddbd6777-k2fm6" podUID="f8f9a320-4770-43a7-9297-b1d659961376" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:46:38.103543 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:38.103454 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1e72f6-64d95f46cf-wxgkn" podUID="c6ff15bc-cbaa-4bbb-a634-415dc98bbb12" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:46:38.674218 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:38.674193 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c0612-85c484d865-fvthd" Oct 31 13:46:38.703753 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:38.703730 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0612-b5b66c795-fdmgl" Oct 31 13:46:38.717616 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:38.717582 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32019af4-f819-4127-9d00-d2582affe985-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32019af4-f819-4127-9d00-d2582affe985-plugins-config-volume\") pod \"32019af4-f819-4127-9d00-d2582affe985\" (UID: \"32019af4-f819-4127-9d00-d2582affe985\") " Oct 31 13:46:38.717616 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:38.717620 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32019af4-f819-4127-9d00-d2582affe985-kube-api-access-zf84d\" (UniqueName: \"kubernetes.io/projected/32019af4-f819-4127-9d00-d2582affe985-kube-api-access-zf84d\") pod \"32019af4-f819-4127-9d00-d2582affe985\" (UID: \"32019af4-f819-4127-9d00-d2582affe985\") " Oct 31 13:46:38.717994 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:38.717964 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32019af4-f819-4127-9d00-d2582affe985-plugins-config-volume" pod "32019af4-f819-4127-9d00-d2582affe985" (UID: "32019af4-f819-4127-9d00-d2582affe985"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:46:38.719929 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:38.719904 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32019af4-f819-4127-9d00-d2582affe985-kube-api-access-zf84d" pod "32019af4-f819-4127-9d00-d2582affe985" (UID: "32019af4-f819-4127-9d00-d2582affe985"). InnerVolumeSpecName "kube-api-access-zf84d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:38.818695 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:38.818666 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fba7a7df-1586-461c-b3d7-63ddcf53f218-kube-api-access-pcplm\" (UniqueName: \"kubernetes.io/projected/fba7a7df-1586-461c-b3d7-63ddcf53f218-kube-api-access-pcplm\") pod \"fba7a7df-1586-461c-b3d7-63ddcf53f218\" (UID: \"fba7a7df-1586-461c-b3d7-63ddcf53f218\") " Oct 31 13:46:38.818872 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:38.818789 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fba7a7df-1586-461c-b3d7-63ddcf53f218-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fba7a7df-1586-461c-b3d7-63ddcf53f218-plugins-config-volume\") pod \"fba7a7df-1586-461c-b3d7-63ddcf53f218\" (UID: \"fba7a7df-1586-461c-b3d7-63ddcf53f218\") " Oct 31 13:46:38.819192 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:38.819169 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32019af4-f819-4127-9d00-d2582affe985-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:38.819460 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:38.819187 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fba7a7df-1586-461c-b3d7-63ddcf53f218-plugins-config-volume" pod "fba7a7df-1586-461c-b3d7-63ddcf53f218" (UID: "fba7a7df-1586-461c-b3d7-63ddcf53f218"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:46:38.819460 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:38.819196 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf84d\" (UniqueName: \"kubernetes.io/projected/32019af4-f819-4127-9d00-d2582affe985-kube-api-access-zf84d\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:38.820854 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:38.820830 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fba7a7df-1586-461c-b3d7-63ddcf53f218-kube-api-access-pcplm" pod "fba7a7df-1586-461c-b3d7-63ddcf53f218" (UID: "fba7a7df-1586-461c-b3d7-63ddcf53f218"). InnerVolumeSpecName "kube-api-access-pcplm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:38.920859 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:38.920779 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fba7a7df-1586-461c-b3d7-63ddcf53f218-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:38.920859 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:38.920804 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcplm\" (UniqueName: \"kubernetes.io/projected/fba7a7df-1586-461c-b3d7-63ddcf53f218-kube-api-access-pcplm\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:39.442145 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:39.442119 5693 generic.go:356] "Generic (PLEG): container finished" podID="32019af4-f819-4127-9d00-d2582affe985" containerID="e91931e26bf74cfd83aa2fb254db66fa562192155e76391641fe4672e4188f00" exitCode=0 Oct 31 13:46:39.442303 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:39.442166 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c0612-85c484d865-fvthd" Oct 31 13:46:39.442303 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:39.442204 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c0612-85c484d865-fvthd" event={"ID":"32019af4-f819-4127-9d00-d2582affe985","Type":"ContainerDied","Data":"e91931e26bf74cfd83aa2fb254db66fa562192155e76391641fe4672e4188f00"} Oct 31 13:46:39.442303 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:39.442259 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c0612-85c484d865-fvthd" event={"ID":"32019af4-f819-4127-9d00-d2582affe985","Type":"ContainerDied","Data":"98054c763cf9d482e1dad2ccd9567bb938c09205da9135527da8d7977f075bed"} Oct 31 13:46:39.442303 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:39.442289 5693 scope.go:122] "RemoveContainer" containerID="e91931e26bf74cfd83aa2fb254db66fa562192155e76391641fe4672e4188f00" Oct 31 13:46:39.443631 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:39.443607 5693 generic.go:356] "Generic (PLEG): container finished" podID="fba7a7df-1586-461c-b3d7-63ddcf53f218" containerID="5a08050267cf453575846e766b0837b98554f1abbfe01bab12d04621c8f9d003" exitCode=0 Oct 31 13:46:39.443773 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:39.443652 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0612-b5b66c795-fdmgl" event={"ID":"fba7a7df-1586-461c-b3d7-63ddcf53f218","Type":"ContainerDied","Data":"5a08050267cf453575846e766b0837b98554f1abbfe01bab12d04621c8f9d003"} Oct 31 13:46:39.443773 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:39.443671 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0612-b5b66c795-fdmgl" Oct 31 13:46:39.443773 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:39.443702 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0612-b5b66c795-fdmgl" event={"ID":"fba7a7df-1586-461c-b3d7-63ddcf53f218","Type":"ContainerDied","Data":"7cdcd77277bb1a3af661aa46fed8786b2fbfe401ce0dcd1e755bb247e5a5aa86"} Oct 31 13:46:39.452153 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:39.452135 5693 scope.go:122] "RemoveContainer" containerID="e91931e26bf74cfd83aa2fb254db66fa562192155e76391641fe4672e4188f00" Oct 31 13:46:39.452417 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:46:39.452395 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e91931e26bf74cfd83aa2fb254db66fa562192155e76391641fe4672e4188f00\": container with ID starting with e91931e26bf74cfd83aa2fb254db66fa562192155e76391641fe4672e4188f00 not found: ID does not exist" containerID="e91931e26bf74cfd83aa2fb254db66fa562192155e76391641fe4672e4188f00" Oct 31 13:46:39.452507 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:39.452427 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e91931e26bf74cfd83aa2fb254db66fa562192155e76391641fe4672e4188f00"} err="failed to get container status \"e91931e26bf74cfd83aa2fb254db66fa562192155e76391641fe4672e4188f00\": rpc error: code = NotFound desc = could not find container \"e91931e26bf74cfd83aa2fb254db66fa562192155e76391641fe4672e4188f00\": container with ID starting with e91931e26bf74cfd83aa2fb254db66fa562192155e76391641fe4672e4188f00 not found: ID does not exist" Oct 31 13:46:39.452507 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:39.452448 5693 scope.go:122] "RemoveContainer" containerID="5a08050267cf453575846e766b0837b98554f1abbfe01bab12d04621c8f9d003" Oct 31 13:46:39.460724 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:39.460704 5693 scope.go:122] "RemoveContainer" containerID="5a08050267cf453575846e766b0837b98554f1abbfe01bab12d04621c8f9d003" Oct 31 13:46:39.461005 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:46:39.460983 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a08050267cf453575846e766b0837b98554f1abbfe01bab12d04621c8f9d003\": container with ID starting with 5a08050267cf453575846e766b0837b98554f1abbfe01bab12d04621c8f9d003 not found: ID does not exist" containerID="5a08050267cf453575846e766b0837b98554f1abbfe01bab12d04621c8f9d003" Oct 31 13:46:39.461077 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:39.461012 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a08050267cf453575846e766b0837b98554f1abbfe01bab12d04621c8f9d003"} err="failed to get container status \"5a08050267cf453575846e766b0837b98554f1abbfe01bab12d04621c8f9d003\": rpc error: code = NotFound desc = could not find container \"5a08050267cf453575846e766b0837b98554f1abbfe01bab12d04621c8f9d003\": container with ID starting with 5a08050267cf453575846e766b0837b98554f1abbfe01bab12d04621c8f9d003 not found: ID does not exist" Oct 31 13:46:39.465592 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:39.465569 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0612-b5b66c795-fdmgl"] Oct 31 13:46:39.468975 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:39.468951 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0612-b5b66c795-fdmgl"] Oct 31 13:46:39.481835 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:39.481815 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-6c0612-85c484d865-fvthd"] Oct 31 13:46:39.485173 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:39.485149 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-6c0612-85c484d865-fvthd"] Oct 31 13:46:40.196716 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:40.196684 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5"] Oct 31 13:46:40.197276 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:40.196919 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" podUID="5ee4d8a4-7509-4ae6-8da5-8015a0712a5e" containerName="istio-proxy" containerID="cri-o://c094d479298ab544e8871572783108ab2c089a8a4d87607557cce4fd6f65d1a4" gracePeriod=30 Oct 31 13:46:41.406855 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:41.406827 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32019af4-f819-4127-9d00-d2582affe985" path="/var/lib/kubelet/pods/32019af4-f819-4127-9d00-d2582affe985/volumes" Oct 31 13:46:41.407249 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:41.407227 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fba7a7df-1586-461c-b3d7-63ddcf53f218" path="/var/lib/kubelet/pods/fba7a7df-1586-461c-b3d7-63ddcf53f218/volumes" Oct 31 13:46:45.463113 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:45.463055 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" Oct 31 13:46:45.478782 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:45.478752 5693 generic.go:356] "Generic (PLEG): container finished" podID="5ee4d8a4-7509-4ae6-8da5-8015a0712a5e" containerID="c094d479298ab544e8871572783108ab2c089a8a4d87607557cce4fd6f65d1a4" exitCode=0 Oct 31 13:46:45.478946 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:45.478823 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" Oct 31 13:46:45.478946 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:45.478845 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" event={"ID":"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e","Type":"ContainerDied","Data":"c094d479298ab544e8871572783108ab2c089a8a4d87607557cce4fd6f65d1a4"} Oct 31 13:46:45.478946 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:45.478923 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5" event={"ID":"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e","Type":"ContainerDied","Data":"59ba8a36fb6286e3e1e88eae971c813646275d1dbba21fc40a1961da154969e9"} Oct 31 13:46:45.479372 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:45.478956 5693 scope.go:122] "RemoveContainer" containerID="c094d479298ab544e8871572783108ab2c089a8a4d87607557cce4fd6f65d1a4" Oct 31 13:46:45.490308 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:45.490287 5693 scope.go:122] "RemoveContainer" containerID="c094d479298ab544e8871572783108ab2c089a8a4d87607557cce4fd6f65d1a4" Oct 31 13:46:45.490652 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:46:45.490616 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c094d479298ab544e8871572783108ab2c089a8a4d87607557cce4fd6f65d1a4\": container with ID starting with c094d479298ab544e8871572783108ab2c089a8a4d87607557cce4fd6f65d1a4 not found: ID does not exist" containerID="c094d479298ab544e8871572783108ab2c089a8a4d87607557cce4fd6f65d1a4" Oct 31 13:46:45.490734 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:45.490664 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c094d479298ab544e8871572783108ab2c089a8a4d87607557cce4fd6f65d1a4"} err="failed to get container status \"c094d479298ab544e8871572783108ab2c089a8a4d87607557cce4fd6f65d1a4\": rpc error: code = NotFound desc = could not find container \"c094d479298ab544e8871572783108ab2c089a8a4d87607557cce4fd6f65d1a4\": container with ID starting with c094d479298ab544e8871572783108ab2c089a8a4d87607557cce4fd6f65d1a4 not found: ID does not exist" Oct 31 13:46:45.589238 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:45.589202 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-istio-token\" (UniqueName: \"kubernetes.io/projected/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-istio-token\") pod \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\" (UID: \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\") " Oct 31 13:46:45.589238 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:45.589244 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-credential-socket\") pod \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\" (UID: \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\") " Oct 31 13:46:45.589492 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:45.589272 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-workload-certs\") pod \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\" (UID: \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\") " Oct 31 13:46:45.589492 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:45.589297 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-kube-api-access-cwkpq\" (UniqueName: \"kubernetes.io/projected/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-kube-api-access-cwkpq\") pod \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\" (UID: \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\") " Oct 31 13:46:45.589492 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:45.589336 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-istio-envoy\") pod \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\" (UID: \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\") " Oct 31 13:46:45.589492 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:45.589367 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-istio-data\") pod \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\" (UID: \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\") " Oct 31 13:46:45.589492 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:45.589399 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-istiod-ca-cert\") pod \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\" (UID: \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\") " Oct 31 13:46:45.589492 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:45.589436 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-istio-podinfo\") pod \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\" (UID: \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\") " Oct 31 13:46:45.589492 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:45.589487 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-workload-socket\") pod \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\" (UID: \"5ee4d8a4-7509-4ae6-8da5-8015a0712a5e\") " Oct 31 13:46:45.589934 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:45.589540 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-credential-socket" pod "5ee4d8a4-7509-4ae6-8da5-8015a0712a5e" (UID: "5ee4d8a4-7509-4ae6-8da5-8015a0712a5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:45.589934 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:45.589592 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-workload-certs" pod "5ee4d8a4-7509-4ae6-8da5-8015a0712a5e" (UID: "5ee4d8a4-7509-4ae6-8da5-8015a0712a5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:45.589934 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:45.589814 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-istio-data" pod "5ee4d8a4-7509-4ae6-8da5-8015a0712a5e" (UID: "5ee4d8a4-7509-4ae6-8da5-8015a0712a5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:45.589934 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:45.589825 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-workload-socket" pod "5ee4d8a4-7509-4ae6-8da5-8015a0712a5e" (UID: "5ee4d8a4-7509-4ae6-8da5-8015a0712a5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:45.589934 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:45.589892 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-istiod-ca-cert" pod "5ee4d8a4-7509-4ae6-8da5-8015a0712a5e" (UID: "5ee4d8a4-7509-4ae6-8da5-8015a0712a5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:46:45.590376 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:45.590306 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:45.590376 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:45.590328 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:45.590376 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:45.590344 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:45.590376 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:45.590357 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:45.590376 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:45.590372 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:45.591916 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:45.591887 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-istio-envoy" pod "5ee4d8a4-7509-4ae6-8da5-8015a0712a5e" (UID: "5ee4d8a4-7509-4ae6-8da5-8015a0712a5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:45.591916 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:45.591911 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-istio-token" pod "5ee4d8a4-7509-4ae6-8da5-8015a0712a5e" (UID: "5ee4d8a4-7509-4ae6-8da5-8015a0712a5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:45.592086 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:45.591924 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-istio-podinfo" pod "5ee4d8a4-7509-4ae6-8da5-8015a0712a5e" (UID: "5ee4d8a4-7509-4ae6-8da5-8015a0712a5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:46:45.592086 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:45.591980 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-kube-api-access-cwkpq" pod "5ee4d8a4-7509-4ae6-8da5-8015a0712a5e" (UID: "5ee4d8a4-7509-4ae6-8da5-8015a0712a5e"). InnerVolumeSpecName "kube-api-access-cwkpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:45.673514 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:45.673474 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv"] Oct 31 13:46:45.691357 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:45.691315 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:45.691357 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:45.691349 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwkpq\" (UniqueName: \"kubernetes.io/projected/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-kube-api-access-cwkpq\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:45.691873 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:45.691366 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:45.691873 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:45.691382 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:45.693604 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:45.693582 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44"] Oct 31 13:46:45.702886 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:45.702859 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j"] Oct 31 13:46:45.806038 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:45.806004 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5"] Oct 31 13:46:45.807757 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:45.807736 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-l9kb5"] Oct 31 13:46:47.406935 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:47.406899 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ee4d8a4-7509-4ae6-8da5-8015a0712a5e" path="/var/lib/kubelet/pods/5ee4d8a4-7509-4ae6-8da5-8015a0712a5e/volumes" Oct 31 13:46:48.103435 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:48.103399 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1e72f6-ddbd6777-k2fm6" podUID="f8f9a320-4770-43a7-9297-b1d659961376" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:46:48.103687 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:48.103548 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1e72f6-64d95f46cf-wxgkn" podUID="c6ff15bc-cbaa-4bbb-a634-415dc98bbb12" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:46:48.691699 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:46:48.691625 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:46:56.444042 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:56.444011 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1e72f6-ddbd6777-k2fm6" Oct 31 13:46:56.474978 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:56.474950 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1e72f6-64d95f46cf-wxgkn" Oct 31 13:46:56.539020 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:56.538978 5693 generic.go:356] "Generic (PLEG): container finished" podID="f8f9a320-4770-43a7-9297-b1d659961376" containerID="5fb9ced1279d5809ac9f354b12bc23f7f1af6721b0212c96fa6fd54a4d8ba989" exitCode=0 Oct 31 13:46:56.539020 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:56.539019 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1e72f6-ddbd6777-k2fm6" Oct 31 13:46:56.539274 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:56.539068 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1e72f6-ddbd6777-k2fm6" event={"ID":"f8f9a320-4770-43a7-9297-b1d659961376","Type":"ContainerDied","Data":"5fb9ced1279d5809ac9f354b12bc23f7f1af6721b0212c96fa6fd54a4d8ba989"} Oct 31 13:46:56.539274 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:56.539122 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1e72f6-ddbd6777-k2fm6" event={"ID":"f8f9a320-4770-43a7-9297-b1d659961376","Type":"ContainerDied","Data":"57c61d8a664f9371df453084cd870f8d46f6e5a7428b5b0c15ca7b817d797cd9"} Oct 31 13:46:56.539274 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:56.539149 5693 scope.go:122] "RemoveContainer" containerID="5fb9ced1279d5809ac9f354b12bc23f7f1af6721b0212c96fa6fd54a4d8ba989" Oct 31 13:46:56.540509 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:56.540487 5693 generic.go:356] "Generic (PLEG): container finished" podID="c6ff15bc-cbaa-4bbb-a634-415dc98bbb12" containerID="b70b0035e6482367f73184c3110ea7e48ee0a538ad1899f3b4083988f550040b" exitCode=0 Oct 31 13:46:56.540626 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:56.540528 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1e72f6-64d95f46cf-wxgkn" Oct 31 13:46:56.540626 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:56.540570 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1e72f6-64d95f46cf-wxgkn" event={"ID":"c6ff15bc-cbaa-4bbb-a634-415dc98bbb12","Type":"ContainerDied","Data":"b70b0035e6482367f73184c3110ea7e48ee0a538ad1899f3b4083988f550040b"} Oct 31 13:46:56.540781 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:56.540631 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1e72f6-64d95f46cf-wxgkn" event={"ID":"c6ff15bc-cbaa-4bbb-a634-415dc98bbb12","Type":"ContainerDied","Data":"f2778342ddc724b66601353b7da97d4baa79138d70cafd34ff841c8b6d2c2bfc"} Oct 31 13:46:56.550053 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:56.549835 5693 scope.go:122] "RemoveContainer" containerID="5fb9ced1279d5809ac9f354b12bc23f7f1af6721b0212c96fa6fd54a4d8ba989" Oct 31 13:46:56.550522 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:46:56.550378 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fb9ced1279d5809ac9f354b12bc23f7f1af6721b0212c96fa6fd54a4d8ba989\": container with ID starting with 5fb9ced1279d5809ac9f354b12bc23f7f1af6721b0212c96fa6fd54a4d8ba989 not found: ID does not exist" containerID="5fb9ced1279d5809ac9f354b12bc23f7f1af6721b0212c96fa6fd54a4d8ba989" Oct 31 13:46:56.550522 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:56.550417 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fb9ced1279d5809ac9f354b12bc23f7f1af6721b0212c96fa6fd54a4d8ba989"} err="failed to get container status \"5fb9ced1279d5809ac9f354b12bc23f7f1af6721b0212c96fa6fd54a4d8ba989\": rpc error: code = NotFound desc = could not find container \"5fb9ced1279d5809ac9f354b12bc23f7f1af6721b0212c96fa6fd54a4d8ba989\": container with ID starting with 5fb9ced1279d5809ac9f354b12bc23f7f1af6721b0212c96fa6fd54a4d8ba989 not found: ID does not exist" Oct 31 13:46:56.550522 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:56.550441 5693 scope.go:122] "RemoveContainer" containerID="b70b0035e6482367f73184c3110ea7e48ee0a538ad1899f3b4083988f550040b" Oct 31 13:46:56.562125 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:56.562105 5693 scope.go:122] "RemoveContainer" containerID="b70b0035e6482367f73184c3110ea7e48ee0a538ad1899f3b4083988f550040b" Oct 31 13:46:56.562425 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:46:56.562404 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b70b0035e6482367f73184c3110ea7e48ee0a538ad1899f3b4083988f550040b\": container with ID starting with b70b0035e6482367f73184c3110ea7e48ee0a538ad1899f3b4083988f550040b not found: ID does not exist" containerID="b70b0035e6482367f73184c3110ea7e48ee0a538ad1899f3b4083988f550040b" Oct 31 13:46:56.562510 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:56.562436 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b70b0035e6482367f73184c3110ea7e48ee0a538ad1899f3b4083988f550040b"} err="failed to get container status \"b70b0035e6482367f73184c3110ea7e48ee0a538ad1899f3b4083988f550040b\": rpc error: code = NotFound desc = could not find container \"b70b0035e6482367f73184c3110ea7e48ee0a538ad1899f3b4083988f550040b\": container with ID starting with b70b0035e6482367f73184c3110ea7e48ee0a538ad1899f3b4083988f550040b not found: ID does not exist" Oct 31 13:46:56.606476 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:56.606435 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6ff15bc-cbaa-4bbb-a634-415dc98bbb12-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6ff15bc-cbaa-4bbb-a634-415dc98bbb12-plugins-config-volume\") pod \"c6ff15bc-cbaa-4bbb-a634-415dc98bbb12\" (UID: \"c6ff15bc-cbaa-4bbb-a634-415dc98bbb12\") " Oct 31 13:46:56.606476 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:56.606470 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6ff15bc-cbaa-4bbb-a634-415dc98bbb12-kube-api-access-n7pmv\" (UniqueName: \"kubernetes.io/projected/c6ff15bc-cbaa-4bbb-a634-415dc98bbb12-kube-api-access-n7pmv\") pod \"c6ff15bc-cbaa-4bbb-a634-415dc98bbb12\" (UID: \"c6ff15bc-cbaa-4bbb-a634-415dc98bbb12\") " Oct 31 13:46:56.606743 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:56.606588 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8f9a320-4770-43a7-9297-b1d659961376-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8f9a320-4770-43a7-9297-b1d659961376-plugins-config-volume\") pod \"f8f9a320-4770-43a7-9297-b1d659961376\" (UID: \"f8f9a320-4770-43a7-9297-b1d659961376\") " Oct 31 13:46:56.606743 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:56.606660 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8f9a320-4770-43a7-9297-b1d659961376-kube-api-access-2vk66\" (UniqueName: \"kubernetes.io/projected/f8f9a320-4770-43a7-9297-b1d659961376-kube-api-access-2vk66\") pod \"f8f9a320-4770-43a7-9297-b1d659961376\" (UID: \"f8f9a320-4770-43a7-9297-b1d659961376\") " Oct 31 13:46:56.606908 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:56.606886 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6ff15bc-cbaa-4bbb-a634-415dc98bbb12-plugins-config-volume" pod "c6ff15bc-cbaa-4bbb-a634-415dc98bbb12" (UID: "c6ff15bc-cbaa-4bbb-a634-415dc98bbb12"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:46:56.607022 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:56.607002 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8f9a320-4770-43a7-9297-b1d659961376-plugins-config-volume" pod "f8f9a320-4770-43a7-9297-b1d659961376" (UID: "f8f9a320-4770-43a7-9297-b1d659961376"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:46:56.607164 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:56.607147 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8f9a320-4770-43a7-9297-b1d659961376-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:56.607232 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:56.607171 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6ff15bc-cbaa-4bbb-a634-415dc98bbb12-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:56.608970 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:56.608946 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6ff15bc-cbaa-4bbb-a634-415dc98bbb12-kube-api-access-n7pmv" pod "c6ff15bc-cbaa-4bbb-a634-415dc98bbb12" (UID: "c6ff15bc-cbaa-4bbb-a634-415dc98bbb12"). InnerVolumeSpecName "kube-api-access-n7pmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:56.609049 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:56.609013 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8f9a320-4770-43a7-9297-b1d659961376-kube-api-access-2vk66" pod "f8f9a320-4770-43a7-9297-b1d659961376" (UID: "f8f9a320-4770-43a7-9297-b1d659961376"). InnerVolumeSpecName "kube-api-access-2vk66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:56.708705 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:56.708578 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vk66\" (UniqueName: \"kubernetes.io/projected/f8f9a320-4770-43a7-9297-b1d659961376-kube-api-access-2vk66\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:56.708705 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:56.708610 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7pmv\" (UniqueName: \"kubernetes.io/projected/c6ff15bc-cbaa-4bbb-a634-415dc98bbb12-kube-api-access-n7pmv\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:56.868730 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:56.868698 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1e72f6-64d95f46cf-wxgkn"] Oct 31 13:46:56.893122 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:56.893086 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1e72f6-64d95f46cf-wxgkn"] Oct 31 13:46:56.905920 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:56.905893 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-1e72f6-ddbd6777-k2fm6"] Oct 31 13:46:56.912591 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:56.912562 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-1e72f6-ddbd6777-k2fm6"] Oct 31 13:46:57.408180 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:57.408148 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6ff15bc-cbaa-4bbb-a634-415dc98bbb12" path="/var/lib/kubelet/pods/c6ff15bc-cbaa-4bbb-a634-415dc98bbb12/volumes" Oct 31 13:46:57.408542 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:46:57.408528 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8f9a320-4770-43a7-9297-b1d659961376" path="/var/lib/kubelet/pods/f8f9a320-4770-43a7-9297-b1d659961376/volumes" Oct 31 13:47:00.694761 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:00.694720 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" podUID="f2b3766f-058c-40e7-9a2d-a7f5028e2b1d" containerName="main" containerID="cri-o://f3dde38c04b1e25c3a22ff0bd13f65bf7679718b2dec1e1fb665093273c6b508" gracePeriod=45 Oct 31 13:47:00.718086 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:00.718055 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" podUID="8fb66a94-c399-43fb-945c-610ca63f0d76" containerName="main" containerID="cri-o://83b955b87f7980f30ebb11e76b8cee966e5f12bb55771e2c2198ccad8f80b64d" gracePeriod=45 Oct 31 13:47:00.723303 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:00.723275 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" podUID="e5a40b79-9879-4981-ab9f-446e1750a013" containerName="main" containerID="cri-o://a74d6f1cfb955ab0647c84c8822d7d82daee55c15cbdb3e092a9a5387c31b086" gracePeriod=45 Oct 31 13:47:00.775012 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:00.774984 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" podUID="8fb66a94-c399-43fb-945c-610ca63f0d76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": dial tcp 10.133.0.66:8000: connect: connection refused" Oct 31 13:47:00.978445 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:00.978416 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" Oct 31 13:47:01.015539 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.015183 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" Oct 31 13:47:01.030722 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.030694 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" Oct 31 13:47:01.152187 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.152153 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fb66a94-c399-43fb-945c-610ca63f0d76-kube-api-access-wx2dm\" (UniqueName: \"kubernetes.io/projected/8fb66a94-c399-43fb-945c-610ca63f0d76-kube-api-access-wx2dm\") pod \"8fb66a94-c399-43fb-945c-610ca63f0d76\" (UID: \"8fb66a94-c399-43fb-945c-610ca63f0d76\") " Oct 31 13:47:01.152357 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.152200 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-tls-certs\") pod \"f2b3766f-058c-40e7-9a2d-a7f5028e2b1d\" (UID: \"f2b3766f-058c-40e7-9a2d-a7f5028e2b1d\") " Oct 31 13:47:01.152357 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.152226 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb66a94-c399-43fb-945c-610ca63f0d76-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fb66a94-c399-43fb-945c-610ca63f0d76-tmp-dir\") pod \"8fb66a94-c399-43fb-945c-610ca63f0d76\" (UID: \"8fb66a94-c399-43fb-945c-610ca63f0d76\") " Oct 31 13:47:01.152465 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.152398 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-kube-api-access-8kchh\" (UniqueName: \"kubernetes.io/projected/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-kube-api-access-8kchh\") pod \"f2b3766f-058c-40e7-9a2d-a7f5028e2b1d\" (UID: \"f2b3766f-058c-40e7-9a2d-a7f5028e2b1d\") " Oct 31 13:47:01.152465 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.152450 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb66a94-c399-43fb-945c-610ca63f0d76-tmp-dir" pod "8fb66a94-c399-43fb-945c-610ca63f0d76" (UID: "8fb66a94-c399-43fb-945c-610ca63f0d76"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:47:01.152592 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.152473 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5a40b79-9879-4981-ab9f-446e1750a013-kube-api-access-z65xd\" (UniqueName: \"kubernetes.io/projected/e5a40b79-9879-4981-ab9f-446e1750a013-kube-api-access-z65xd\") pod \"e5a40b79-9879-4981-ab9f-446e1750a013\" (UID: \"e5a40b79-9879-4981-ab9f-446e1750a013\") " Oct 31 13:47:01.152592 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.152517 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5a40b79-9879-4981-ab9f-446e1750a013-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5a40b79-9879-4981-ab9f-446e1750a013-tmp-dir\") pod \"e5a40b79-9879-4981-ab9f-446e1750a013\" (UID: \"e5a40b79-9879-4981-ab9f-446e1750a013\") " Oct 31 13:47:01.152592 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.152547 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb66a94-c399-43fb-945c-610ca63f0d76-home\" (UniqueName: \"kubernetes.io/empty-dir/8fb66a94-c399-43fb-945c-610ca63f0d76-home\") pod \"8fb66a94-c399-43fb-945c-610ca63f0d76\" (UID: \"8fb66a94-c399-43fb-945c-610ca63f0d76\") " Oct 31 13:47:01.152592 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.152576 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5a40b79-9879-4981-ab9f-446e1750a013-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5a40b79-9879-4981-ab9f-446e1750a013-model-cache\") pod \"e5a40b79-9879-4981-ab9f-446e1750a013\" (UID: \"e5a40b79-9879-4981-ab9f-446e1750a013\") " Oct 31 13:47:01.152868 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.152621 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5a40b79-9879-4981-ab9f-446e1750a013-home\" (UniqueName: \"kubernetes.io/empty-dir/e5a40b79-9879-4981-ab9f-446e1750a013-home\") pod \"e5a40b79-9879-4981-ab9f-446e1750a013\" (UID: \"e5a40b79-9879-4981-ab9f-446e1750a013\") " Oct 31 13:47:01.152868 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.152696 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb66a94-c399-43fb-945c-610ca63f0d76-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fb66a94-c399-43fb-945c-610ca63f0d76-model-cache\") pod \"8fb66a94-c399-43fb-945c-610ca63f0d76\" (UID: \"8fb66a94-c399-43fb-945c-610ca63f0d76\") " Oct 31 13:47:01.152868 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.152727 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb66a94-c399-43fb-945c-610ca63f0d76-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fb66a94-c399-43fb-945c-610ca63f0d76-dshm\") pod \"8fb66a94-c399-43fb-945c-610ca63f0d76\" (UID: \"8fb66a94-c399-43fb-945c-610ca63f0d76\") " Oct 31 13:47:01.152868 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.152790 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5a40b79-9879-4981-ab9f-446e1750a013-tls-certs\" (UniqueName: \"kubernetes.io/secret/e5a40b79-9879-4981-ab9f-446e1750a013-tls-certs\") pod \"e5a40b79-9879-4981-ab9f-446e1750a013\" (UID: \"e5a40b79-9879-4981-ab9f-446e1750a013\") " Oct 31 13:47:01.152868 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.152847 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-home\" (UniqueName: \"kubernetes.io/empty-dir/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-home\") pod \"f2b3766f-058c-40e7-9a2d-a7f5028e2b1d\" (UID: \"f2b3766f-058c-40e7-9a2d-a7f5028e2b1d\") " Oct 31 13:47:01.152868 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.152844 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a40b79-9879-4981-ab9f-446e1750a013-tmp-dir" pod "e5a40b79-9879-4981-ab9f-446e1750a013" (UID: "e5a40b79-9879-4981-ab9f-446e1750a013"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:47:01.153383 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.152867 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a40b79-9879-4981-ab9f-446e1750a013-model-cache" pod "e5a40b79-9879-4981-ab9f-446e1750a013" (UID: "e5a40b79-9879-4981-ab9f-446e1750a013"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:47:01.153383 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.152879 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-dshm\") pod \"f2b3766f-058c-40e7-9a2d-a7f5028e2b1d\" (UID: \"f2b3766f-058c-40e7-9a2d-a7f5028e2b1d\") " Oct 31 13:47:01.153383 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.152910 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-tmp-dir\") pod \"f2b3766f-058c-40e7-9a2d-a7f5028e2b1d\" (UID: \"f2b3766f-058c-40e7-9a2d-a7f5028e2b1d\") " Oct 31 13:47:01.153383 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.153109 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fb66a94-c399-43fb-945c-610ca63f0d76-tls-certs\" (UniqueName: \"kubernetes.io/secret/8fb66a94-c399-43fb-945c-610ca63f0d76-tls-certs\") pod \"8fb66a94-c399-43fb-945c-610ca63f0d76\" (UID: \"8fb66a94-c399-43fb-945c-610ca63f0d76\") " Oct 31 13:47:01.153383 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.153156 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5a40b79-9879-4981-ab9f-446e1750a013-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5a40b79-9879-4981-ab9f-446e1750a013-dshm\") pod \"e5a40b79-9879-4981-ab9f-446e1750a013\" (UID: \"e5a40b79-9879-4981-ab9f-446e1750a013\") " Oct 31 13:47:01.153383 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.153185 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-model-cache\") pod \"f2b3766f-058c-40e7-9a2d-a7f5028e2b1d\" (UID: \"f2b3766f-058c-40e7-9a2d-a7f5028e2b1d\") " Oct 31 13:47:01.153383 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.153299 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb66a94-c399-43fb-945c-610ca63f0d76-home" pod "8fb66a94-c399-43fb-945c-610ca63f0d76" (UID: "8fb66a94-c399-43fb-945c-610ca63f0d76"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:47:01.153383 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.153307 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-home" pod "f2b3766f-058c-40e7-9a2d-a7f5028e2b1d" (UID: "f2b3766f-058c-40e7-9a2d-a7f5028e2b1d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:47:01.153876 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.153473 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-tmp-dir" pod "f2b3766f-058c-40e7-9a2d-a7f5028e2b1d" (UID: "f2b3766f-058c-40e7-9a2d-a7f5028e2b1d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:47:01.153876 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.153677 5693 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5a40b79-9879-4981-ab9f-446e1750a013-tmp-dir\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:01.153876 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.153699 5693 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fb66a94-c399-43fb-945c-610ca63f0d76-home\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:01.153876 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.153712 5693 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5a40b79-9879-4981-ab9f-446e1750a013-model-cache\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:01.153876 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.153727 5693 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-home\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:01.153876 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.153740 5693 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-tmp-dir\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:01.153876 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.153753 5693 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fb66a94-c399-43fb-945c-610ca63f0d76-tmp-dir\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:01.153876 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.153758 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a40b79-9879-4981-ab9f-446e1750a013-home" pod "e5a40b79-9879-4981-ab9f-446e1750a013" (UID: "e5a40b79-9879-4981-ab9f-446e1750a013"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:47:01.154529 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.153910 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb66a94-c399-43fb-945c-610ca63f0d76-model-cache" pod "8fb66a94-c399-43fb-945c-610ca63f0d76" (UID: "8fb66a94-c399-43fb-945c-610ca63f0d76"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:47:01.154529 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.154096 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-model-cache" pod "f2b3766f-058c-40e7-9a2d-a7f5028e2b1d" (UID: "f2b3766f-058c-40e7-9a2d-a7f5028e2b1d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:47:01.155279 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.155243 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-tls-certs" pod "f2b3766f-058c-40e7-9a2d-a7f5028e2b1d" (UID: "f2b3766f-058c-40e7-9a2d-a7f5028e2b1d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:47:01.155539 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.155513 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-kube-api-access-8kchh" pod "f2b3766f-058c-40e7-9a2d-a7f5028e2b1d" (UID: "f2b3766f-058c-40e7-9a2d-a7f5028e2b1d"). InnerVolumeSpecName "kube-api-access-8kchh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:47:01.156336 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.156305 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5a40b79-9879-4981-ab9f-446e1750a013-kube-api-access-z65xd" pod "e5a40b79-9879-4981-ab9f-446e1750a013" (UID: "e5a40b79-9879-4981-ab9f-446e1750a013"). InnerVolumeSpecName "kube-api-access-z65xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:47:01.156483 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.156354 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb66a94-c399-43fb-945c-610ca63f0d76-kube-api-access-wx2dm" pod "8fb66a94-c399-43fb-945c-610ca63f0d76" (UID: "8fb66a94-c399-43fb-945c-610ca63f0d76"). InnerVolumeSpecName "kube-api-access-wx2dm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:47:01.156483 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.156375 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fb66a94-c399-43fb-945c-610ca63f0d76-tls-certs" pod "8fb66a94-c399-43fb-945c-610ca63f0d76" (UID: "8fb66a94-c399-43fb-945c-610ca63f0d76"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:47:01.156483 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.156376 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb66a94-c399-43fb-945c-610ca63f0d76-dshm" pod "8fb66a94-c399-43fb-945c-610ca63f0d76" (UID: "8fb66a94-c399-43fb-945c-610ca63f0d76"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:47:01.156483 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.156460 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-dshm" pod "f2b3766f-058c-40e7-9a2d-a7f5028e2b1d" (UID: "f2b3766f-058c-40e7-9a2d-a7f5028e2b1d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:47:01.156775 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.156635 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a40b79-9879-4981-ab9f-446e1750a013-dshm" pod "e5a40b79-9879-4981-ab9f-446e1750a013" (UID: "e5a40b79-9879-4981-ab9f-446e1750a013"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:47:01.157095 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.157079 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5a40b79-9879-4981-ab9f-446e1750a013-tls-certs" pod "e5a40b79-9879-4981-ab9f-446e1750a013" (UID: "e5a40b79-9879-4981-ab9f-446e1750a013"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:47:01.255058 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.254985 5693 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-dshm\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:01.255058 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.255008 5693 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fb66a94-c399-43fb-945c-610ca63f0d76-tls-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:01.255058 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.255018 5693 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5a40b79-9879-4981-ab9f-446e1750a013-dshm\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:01.255058 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.255026 5693 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-model-cache\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:01.255058 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.255036 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx2dm\" (UniqueName: \"kubernetes.io/projected/8fb66a94-c399-43fb-945c-610ca63f0d76-kube-api-access-wx2dm\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:01.255058 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.255044 5693 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-tls-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:01.255058 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.255053 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kchh\" (UniqueName: \"kubernetes.io/projected/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d-kube-api-access-8kchh\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:01.255058 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.255062 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z65xd\" (UniqueName: \"kubernetes.io/projected/e5a40b79-9879-4981-ab9f-446e1750a013-kube-api-access-z65xd\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:01.255434 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.255070 5693 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5a40b79-9879-4981-ab9f-446e1750a013-home\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:01.255434 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.255078 5693 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fb66a94-c399-43fb-945c-610ca63f0d76-model-cache\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:01.255434 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.255086 5693 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fb66a94-c399-43fb-945c-610ca63f0d76-dshm\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:01.255434 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.255094 5693 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5a40b79-9879-4981-ab9f-446e1750a013-tls-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:01.568955 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.568868 5693 generic.go:356] "Generic (PLEG): container finished" podID="f2b3766f-058c-40e7-9a2d-a7f5028e2b1d" containerID="f3dde38c04b1e25c3a22ff0bd13f65bf7679718b2dec1e1fb665093273c6b508" exitCode=0 Oct 31 13:47:01.568955 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.568914 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" Oct 31 13:47:01.569180 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.568946 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" event={"ID":"f2b3766f-058c-40e7-9a2d-a7f5028e2b1d","Type":"ContainerDied","Data":"f3dde38c04b1e25c3a22ff0bd13f65bf7679718b2dec1e1fb665093273c6b508"} Oct 31 13:47:01.569180 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.568986 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv" event={"ID":"f2b3766f-058c-40e7-9a2d-a7f5028e2b1d","Type":"ContainerDied","Data":"19551b0daedad477182f85862e846b25e8e75f423c1800f401e27c197fb87d56"} Oct 31 13:47:01.569180 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.569004 5693 scope.go:122] "RemoveContainer" containerID="f3dde38c04b1e25c3a22ff0bd13f65bf7679718b2dec1e1fb665093273c6b508" Oct 31 13:47:01.570410 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.570387 5693 generic.go:356] "Generic (PLEG): container finished" podID="8fb66a94-c399-43fb-945c-610ca63f0d76" containerID="83b955b87f7980f30ebb11e76b8cee966e5f12bb55771e2c2198ccad8f80b64d" exitCode=0 Oct 31 13:47:01.570547 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.570423 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" event={"ID":"8fb66a94-c399-43fb-945c-610ca63f0d76","Type":"ContainerDied","Data":"83b955b87f7980f30ebb11e76b8cee966e5f12bb55771e2c2198ccad8f80b64d"} Oct 31 13:47:01.570547 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.570457 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" Oct 31 13:47:01.570547 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.570465 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44" event={"ID":"8fb66a94-c399-43fb-945c-610ca63f0d76","Type":"ContainerDied","Data":"aa94e953b4e213dde0da4a3d2142ad42e95be062a966ccee6cdb6863d97511e4"} Oct 31 13:47:01.572068 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.572041 5693 generic.go:356] "Generic (PLEG): container finished" podID="e5a40b79-9879-4981-ab9f-446e1750a013" containerID="a74d6f1cfb955ab0647c84c8822d7d82daee55c15cbdb3e092a9a5387c31b086" exitCode=0 Oct 31 13:47:01.572180 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.572069 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" event={"ID":"e5a40b79-9879-4981-ab9f-446e1750a013","Type":"ContainerDied","Data":"a74d6f1cfb955ab0647c84c8822d7d82daee55c15cbdb3e092a9a5387c31b086"} Oct 31 13:47:01.572180 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.572084 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" Oct 31 13:47:01.572180 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.572103 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" event={"ID":"e5a40b79-9879-4981-ab9f-446e1750a013","Type":"ContainerDied","Data":"05699a910ce6a66a9709b3f09ff2f34d5c28aba16a03f7787f841bd190fee7f1"} Oct 31 13:47:01.580008 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.579984 5693 scope.go:122] "RemoveContainer" containerID="f3dde38c04b1e25c3a22ff0bd13f65bf7679718b2dec1e1fb665093273c6b508" Oct 31 13:47:01.580306 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:47:01.580285 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3dde38c04b1e25c3a22ff0bd13f65bf7679718b2dec1e1fb665093273c6b508\": container with ID starting with f3dde38c04b1e25c3a22ff0bd13f65bf7679718b2dec1e1fb665093273c6b508 not found: ID does not exist" containerID="f3dde38c04b1e25c3a22ff0bd13f65bf7679718b2dec1e1fb665093273c6b508" Oct 31 13:47:01.580393 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.580314 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3dde38c04b1e25c3a22ff0bd13f65bf7679718b2dec1e1fb665093273c6b508"} err="failed to get container status \"f3dde38c04b1e25c3a22ff0bd13f65bf7679718b2dec1e1fb665093273c6b508\": rpc error: code = NotFound desc = could not find container \"f3dde38c04b1e25c3a22ff0bd13f65bf7679718b2dec1e1fb665093273c6b508\": container with ID starting with f3dde38c04b1e25c3a22ff0bd13f65bf7679718b2dec1e1fb665093273c6b508 not found: ID does not exist" Oct 31 13:47:01.580393 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.580330 5693 scope.go:122] "RemoveContainer" containerID="83b955b87f7980f30ebb11e76b8cee966e5f12bb55771e2c2198ccad8f80b64d" Oct 31 13:47:01.589356 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.589336 5693 scope.go:122] "RemoveContainer" containerID="83b955b87f7980f30ebb11e76b8cee966e5f12bb55771e2c2198ccad8f80b64d" Oct 31 13:47:01.589608 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:47:01.589587 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83b955b87f7980f30ebb11e76b8cee966e5f12bb55771e2c2198ccad8f80b64d\": container with ID starting with 83b955b87f7980f30ebb11e76b8cee966e5f12bb55771e2c2198ccad8f80b64d not found: ID does not exist" containerID="83b955b87f7980f30ebb11e76b8cee966e5f12bb55771e2c2198ccad8f80b64d" Oct 31 13:47:01.589713 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.589620 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83b955b87f7980f30ebb11e76b8cee966e5f12bb55771e2c2198ccad8f80b64d"} err="failed to get container status \"83b955b87f7980f30ebb11e76b8cee966e5f12bb55771e2c2198ccad8f80b64d\": rpc error: code = NotFound desc = could not find container \"83b955b87f7980f30ebb11e76b8cee966e5f12bb55771e2c2198ccad8f80b64d\": container with ID starting with 83b955b87f7980f30ebb11e76b8cee966e5f12bb55771e2c2198ccad8f80b64d not found: ID does not exist" Oct 31 13:47:01.589713 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.589662 5693 scope.go:122] "RemoveContainer" containerID="a74d6f1cfb955ab0647c84c8822d7d82daee55c15cbdb3e092a9a5387c31b086" Oct 31 13:47:01.594544 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.594524 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv"] Oct 31 13:47:01.598920 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.598901 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7ae9e7/distinct-2-w4-7ae9e7-kserve-75496cc8b4-qvrfv"] Oct 31 13:47:01.598992 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.598919 5693 scope.go:122] "RemoveContainer" containerID="a74d6f1cfb955ab0647c84c8822d7d82daee55c15cbdb3e092a9a5387c31b086" Oct 31 13:47:01.599216 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:47:01.599196 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a74d6f1cfb955ab0647c84c8822d7d82daee55c15cbdb3e092a9a5387c31b086\": container with ID starting with a74d6f1cfb955ab0647c84c8822d7d82daee55c15cbdb3e092a9a5387c31b086 not found: ID does not exist" containerID="a74d6f1cfb955ab0647c84c8822d7d82daee55c15cbdb3e092a9a5387c31b086" Oct 31 13:47:01.599266 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.599223 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a74d6f1cfb955ab0647c84c8822d7d82daee55c15cbdb3e092a9a5387c31b086"} err="failed to get container status \"a74d6f1cfb955ab0647c84c8822d7d82daee55c15cbdb3e092a9a5387c31b086\": rpc error: code = NotFound desc = could not find container \"a74d6f1cfb955ab0647c84c8822d7d82daee55c15cbdb3e092a9a5387c31b086\": container with ID starting with a74d6f1cfb955ab0647c84c8822d7d82daee55c15cbdb3e092a9a5387c31b086 not found: ID does not exist" Oct 31 13:47:01.614856 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.614833 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44"] Oct 31 13:47:01.618870 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.618848 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7ae9e7/premium-w4-7ae9e7-kserve-6b7f497cf6-9zp44"] Oct 31 13:47:01.631046 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.631020 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j"] Oct 31 13:47:01.636867 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.636846 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j"] Oct 31 13:47:01.932088 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:01.932043 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/simulator-w4-7ae9e7-kserve-5d9dcbbdbf-s7n9j" podUID="e5a40b79-9879-4981-ab9f-446e1750a013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:47:03.406685 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:03.406632 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fb66a94-c399-43fb-945c-610ca63f0d76" path="/var/lib/kubelet/pods/8fb66a94-c399-43fb-945c-610ca63f0d76/volumes" Oct 31 13:47:03.407085 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:03.407073 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5a40b79-9879-4981-ab9f-446e1750a013" path="/var/lib/kubelet/pods/e5a40b79-9879-4981-ab9f-446e1750a013/volumes" Oct 31 13:47:03.407433 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:03.407419 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2b3766f-058c-40e7-9a2d-a7f5028e2b1d" path="/var/lib/kubelet/pods/f2b3766f-058c-40e7-9a2d-a7f5028e2b1d/volumes" Oct 31 13:47:03.623699 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:03.623634 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj"] Oct 31 13:47:03.623919 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:03.623879 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj" podUID="71e7cbcd-adef-4e0e-8944-b3c62e6407bc" containerName="maas-api" containerID="cri-o://e763633f260c89b77a7c7f1c6375ca2f7d4bfc336c59c1058c64c9e79c60af73" gracePeriod=30 Oct 31 13:47:03.891738 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:03.891713 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj" Oct 31 13:47:03.982666 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:03.982603 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71e7cbcd-adef-4e0e-8944-b3c62e6407bc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/71e7cbcd-adef-4e0e-8944-b3c62e6407bc-metrics-certs\") pod \"71e7cbcd-adef-4e0e-8944-b3c62e6407bc\" (UID: \"71e7cbcd-adef-4e0e-8944-b3c62e6407bc\") " Oct 31 13:47:03.982868 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:03.982690 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71e7cbcd-adef-4e0e-8944-b3c62e6407bc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e7cbcd-adef-4e0e-8944-b3c62e6407bc-openshift-service-ca\") pod \"71e7cbcd-adef-4e0e-8944-b3c62e6407bc\" (UID: \"71e7cbcd-adef-4e0e-8944-b3c62e6407bc\") " Oct 31 13:47:03.982868 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:03.982762 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71e7cbcd-adef-4e0e-8944-b3c62e6407bc-kube-api-access-2lpmb\" (UniqueName: \"kubernetes.io/projected/71e7cbcd-adef-4e0e-8944-b3c62e6407bc-kube-api-access-2lpmb\") pod \"71e7cbcd-adef-4e0e-8944-b3c62e6407bc\" (UID: \"71e7cbcd-adef-4e0e-8944-b3c62e6407bc\") " Oct 31 13:47:03.983119 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:03.983091 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71e7cbcd-adef-4e0e-8944-b3c62e6407bc-openshift-service-ca" pod "71e7cbcd-adef-4e0e-8944-b3c62e6407bc" (UID: "71e7cbcd-adef-4e0e-8944-b3c62e6407bc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:47:03.984965 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:03.984937 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71e7cbcd-adef-4e0e-8944-b3c62e6407bc-metrics-certs" pod "71e7cbcd-adef-4e0e-8944-b3c62e6407bc" (UID: "71e7cbcd-adef-4e0e-8944-b3c62e6407bc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:47:03.985081 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:03.984973 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71e7cbcd-adef-4e0e-8944-b3c62e6407bc-kube-api-access-2lpmb" pod "71e7cbcd-adef-4e0e-8944-b3c62e6407bc" (UID: "71e7cbcd-adef-4e0e-8944-b3c62e6407bc"). InnerVolumeSpecName "kube-api-access-2lpmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:47:04.083406 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:04.083374 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71e7cbcd-adef-4e0e-8944-b3c62e6407bc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71e7cbcd-adef-4e0e-8944-b3c62e6407bc-maas-api-tls\") pod \"71e7cbcd-adef-4e0e-8944-b3c62e6407bc\" (UID: \"71e7cbcd-adef-4e0e-8944-b3c62e6407bc\") " Oct 31 13:47:04.083598 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:04.083586 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lpmb\" (UniqueName: \"kubernetes.io/projected/71e7cbcd-adef-4e0e-8944-b3c62e6407bc-kube-api-access-2lpmb\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:04.083660 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:04.083601 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71e7cbcd-adef-4e0e-8944-b3c62e6407bc-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:04.083660 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:04.083612 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e7cbcd-adef-4e0e-8944-b3c62e6407bc-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:04.085410 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:04.085385 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71e7cbcd-adef-4e0e-8944-b3c62e6407bc-maas-api-tls" pod "71e7cbcd-adef-4e0e-8944-b3c62e6407bc" (UID: "71e7cbcd-adef-4e0e-8944-b3c62e6407bc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:47:04.184963 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:04.184918 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71e7cbcd-adef-4e0e-8944-b3c62e6407bc-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:04.590823 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:04.590789 5693 generic.go:356] "Generic (PLEG): container finished" podID="71e7cbcd-adef-4e0e-8944-b3c62e6407bc" containerID="e763633f260c89b77a7c7f1c6375ca2f7d4bfc336c59c1058c64c9e79c60af73" exitCode=0 Oct 31 13:47:04.591270 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:04.590833 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj" Oct 31 13:47:04.591270 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:04.590870 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj" event={"ID":"71e7cbcd-adef-4e0e-8944-b3c62e6407bc","Type":"ContainerDied","Data":"e763633f260c89b77a7c7f1c6375ca2f7d4bfc336c59c1058c64c9e79c60af73"} Oct 31 13:47:04.591270 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:04.590927 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj" event={"ID":"71e7cbcd-adef-4e0e-8944-b3c62e6407bc","Type":"ContainerDied","Data":"9013b8d8e9e17cbd03fe2dca702661f2a59154e55278f383d187eddbe1e298fc"} Oct 31 13:47:04.591270 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:04.590950 5693 scope.go:122] "RemoveContainer" containerID="e763633f260c89b77a7c7f1c6375ca2f7d4bfc336c59c1058c64c9e79c60af73" Oct 31 13:47:04.601293 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:04.601277 5693 scope.go:122] "RemoveContainer" containerID="e763633f260c89b77a7c7f1c6375ca2f7d4bfc336c59c1058c64c9e79c60af73" Oct 31 13:47:04.601554 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:47:04.601535 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e763633f260c89b77a7c7f1c6375ca2f7d4bfc336c59c1058c64c9e79c60af73\": container with ID starting with e763633f260c89b77a7c7f1c6375ca2f7d4bfc336c59c1058c64c9e79c60af73 not found: ID does not exist" containerID="e763633f260c89b77a7c7f1c6375ca2f7d4bfc336c59c1058c64c9e79c60af73" Oct 31 13:47:04.601592 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:04.601564 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e763633f260c89b77a7c7f1c6375ca2f7d4bfc336c59c1058c64c9e79c60af73"} err="failed to get container status \"e763633f260c89b77a7c7f1c6375ca2f7d4bfc336c59c1058c64c9e79c60af73\": rpc error: code = NotFound desc = could not find container \"e763633f260c89b77a7c7f1c6375ca2f7d4bfc336c59c1058c64c9e79c60af73\": container with ID starting with e763633f260c89b77a7c7f1c6375ca2f7d4bfc336c59c1058c64c9e79c60af73 not found: ID does not exist" Oct 31 13:47:04.612720 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:04.612698 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj"] Oct 31 13:47:04.618752 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:04.618732 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ae9e7-67f49f46c-wx7wj"] Oct 31 13:47:05.406973 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:05.406938 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71e7cbcd-adef-4e0e-8944-b3c62e6407bc" path="/var/lib/kubelet/pods/71e7cbcd-adef-4e0e-8944-b3c62e6407bc/volumes" Oct 31 13:47:13.254803 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:13.254766 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp"] Oct 31 13:47:13.256931 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:13.254963 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" podUID="a4b60d0a-c847-439d-a156-d6111ffb40a7" containerName="istio-proxy" containerID="cri-o://fadceadd0330b24524f6a9ee5bcdb5925def479bd27453d8c1c0d45bde3b934d" gracePeriod=30 Oct 31 13:47:14.217130 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:14.217102 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-9c9ccc756-vngkm_56840e04-370d-44ef-92a1-46d4bb15f9ec/authorino/0.log" Oct 31 13:47:14.825299 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:14.825268 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-b2vpf_86e98514-c296-467f-ba33-cbcdf689b48b/kuadrant-console-plugin/0.log" Oct 31 13:47:16.635420 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:16.635389 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6df7b9f889-66dn6_5c3a62d1-ddd7-4459-8342-2ed0eedd3d8d/manager/0.log" Oct 31 13:47:17.246078 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:17.246042 5693 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:47:17.246258 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:17.246115 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" podUID="a4b60d0a-c847-439d-a156-d6111ffb40a7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:47:17.769202 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:17.769166 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7767cbb97d-2mzd8_052fb6af-f18c-429c-8946-2022c122adfd/manager/0.log" Oct 31 13:47:18.355788 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:18.355746 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-2wppt_75062797-634c-4d93-9c96-7c0060e0f684/proxy/0.log" Oct 31 13:47:18.511818 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:18.511795 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" Oct 31 13:47:18.614012 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:18.613935 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4b60d0a-c847-439d-a156-d6111ffb40a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4b60d0a-c847-439d-a156-d6111ffb40a7-istiod-ca-cert\") pod \"a4b60d0a-c847-439d-a156-d6111ffb40a7\" (UID: \"a4b60d0a-c847-439d-a156-d6111ffb40a7\") " Oct 31 13:47:18.614012 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:18.613969 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4b60d0a-c847-439d-a156-d6111ffb40a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4b60d0a-c847-439d-a156-d6111ffb40a7-workload-certs\") pod \"a4b60d0a-c847-439d-a156-d6111ffb40a7\" (UID: \"a4b60d0a-c847-439d-a156-d6111ffb40a7\") " Oct 31 13:47:18.614253 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:18.614017 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4b60d0a-c847-439d-a156-d6111ffb40a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4b60d0a-c847-439d-a156-d6111ffb40a7-istio-podinfo\") pod \"a4b60d0a-c847-439d-a156-d6111ffb40a7\" (UID: \"a4b60d0a-c847-439d-a156-d6111ffb40a7\") " Oct 31 13:47:18.614253 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:18.614048 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4b60d0a-c847-439d-a156-d6111ffb40a7-istio-token\" (UniqueName: \"kubernetes.io/projected/a4b60d0a-c847-439d-a156-d6111ffb40a7-istio-token\") pod \"a4b60d0a-c847-439d-a156-d6111ffb40a7\" (UID: \"a4b60d0a-c847-439d-a156-d6111ffb40a7\") " Oct 31 13:47:18.614253 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:18.614109 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4b60d0a-c847-439d-a156-d6111ffb40a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4b60d0a-c847-439d-a156-d6111ffb40a7-istio-envoy\") pod \"a4b60d0a-c847-439d-a156-d6111ffb40a7\" (UID: \"a4b60d0a-c847-439d-a156-d6111ffb40a7\") " Oct 31 13:47:18.614253 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:18.614177 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4b60d0a-c847-439d-a156-d6111ffb40a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4b60d0a-c847-439d-a156-d6111ffb40a7-istio-data\") pod \"a4b60d0a-c847-439d-a156-d6111ffb40a7\" (UID: \"a4b60d0a-c847-439d-a156-d6111ffb40a7\") " Oct 31 13:47:18.614253 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:18.614213 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4b60d0a-c847-439d-a156-d6111ffb40a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b60d0a-c847-439d-a156-d6111ffb40a7-credential-socket\") pod \"a4b60d0a-c847-439d-a156-d6111ffb40a7\" (UID: \"a4b60d0a-c847-439d-a156-d6111ffb40a7\") " Oct 31 13:47:18.614253 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:18.614238 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4b60d0a-c847-439d-a156-d6111ffb40a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b60d0a-c847-439d-a156-d6111ffb40a7-workload-socket\") pod \"a4b60d0a-c847-439d-a156-d6111ffb40a7\" (UID: \"a4b60d0a-c847-439d-a156-d6111ffb40a7\") " Oct 31 13:47:18.614602 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:18.614321 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4b60d0a-c847-439d-a156-d6111ffb40a7-kube-api-access-4nq6w\" (UniqueName: \"kubernetes.io/projected/a4b60d0a-c847-439d-a156-d6111ffb40a7-kube-api-access-4nq6w\") pod \"a4b60d0a-c847-439d-a156-d6111ffb40a7\" (UID: \"a4b60d0a-c847-439d-a156-d6111ffb40a7\") " Oct 31 13:47:18.614602 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:18.614325 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4b60d0a-c847-439d-a156-d6111ffb40a7-workload-certs" pod "a4b60d0a-c847-439d-a156-d6111ffb40a7" (UID: "a4b60d0a-c847-439d-a156-d6111ffb40a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:47:18.614602 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:18.614399 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4b60d0a-c847-439d-a156-d6111ffb40a7-istiod-ca-cert" pod "a4b60d0a-c847-439d-a156-d6111ffb40a7" (UID: "a4b60d0a-c847-439d-a156-d6111ffb40a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:47:18.614602 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:18.614553 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4b60d0a-c847-439d-a156-d6111ffb40a7-istio-data" pod "a4b60d0a-c847-439d-a156-d6111ffb40a7" (UID: "a4b60d0a-c847-439d-a156-d6111ffb40a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:47:18.614602 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:18.614562 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4b60d0a-c847-439d-a156-d6111ffb40a7-credential-socket" pod "a4b60d0a-c847-439d-a156-d6111ffb40a7" (UID: "a4b60d0a-c847-439d-a156-d6111ffb40a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:47:18.615102 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:18.614785 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4b60d0a-c847-439d-a156-d6111ffb40a7-workload-socket" pod "a4b60d0a-c847-439d-a156-d6111ffb40a7" (UID: "a4b60d0a-c847-439d-a156-d6111ffb40a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:47:18.615102 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:18.614960 5693 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4b60d0a-c847-439d-a156-d6111ffb40a7-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:18.615102 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:18.614982 5693 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4b60d0a-c847-439d-a156-d6111ffb40a7-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:18.615102 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:18.614997 5693 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4b60d0a-c847-439d-a156-d6111ffb40a7-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:18.615102 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:18.615013 5693 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b60d0a-c847-439d-a156-d6111ffb40a7-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:18.615102 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:18.615029 5693 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b60d0a-c847-439d-a156-d6111ffb40a7-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:18.616356 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:18.616333 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4b60d0a-c847-439d-a156-d6111ffb40a7-istio-podinfo" pod "a4b60d0a-c847-439d-a156-d6111ffb40a7" (UID: "a4b60d0a-c847-439d-a156-d6111ffb40a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:47:18.616476 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:18.616385 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4b60d0a-c847-439d-a156-d6111ffb40a7-istio-envoy" pod "a4b60d0a-c847-439d-a156-d6111ffb40a7" (UID: "a4b60d0a-c847-439d-a156-d6111ffb40a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:47:18.616525 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:18.616502 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4b60d0a-c847-439d-a156-d6111ffb40a7-istio-token" pod "a4b60d0a-c847-439d-a156-d6111ffb40a7" (UID: "a4b60d0a-c847-439d-a156-d6111ffb40a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:47:18.616581 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:18.616565 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4b60d0a-c847-439d-a156-d6111ffb40a7-kube-api-access-4nq6w" pod "a4b60d0a-c847-439d-a156-d6111ffb40a7" (UID: "a4b60d0a-c847-439d-a156-d6111ffb40a7"). InnerVolumeSpecName "kube-api-access-4nq6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:47:18.669132 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:18.669100 5693 generic.go:356] "Generic (PLEG): container finished" podID="a4b60d0a-c847-439d-a156-d6111ffb40a7" containerID="fadceadd0330b24524f6a9ee5bcdb5925def479bd27453d8c1c0d45bde3b934d" exitCode=0 Oct 31 13:47:18.669331 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:18.669143 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" event={"ID":"a4b60d0a-c847-439d-a156-d6111ffb40a7","Type":"ContainerDied","Data":"fadceadd0330b24524f6a9ee5bcdb5925def479bd27453d8c1c0d45bde3b934d"} Oct 31 13:47:18.669331 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:18.669158 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" Oct 31 13:47:18.669331 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:18.669219 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp" event={"ID":"a4b60d0a-c847-439d-a156-d6111ffb40a7","Type":"ContainerDied","Data":"fb2ec01e46c607c5fbbc0cb95e33813144be8f205ff5ee8f8002f37d44d59647"} Oct 31 13:47:18.669331 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:18.669242 5693 scope.go:122] "RemoveContainer" containerID="fadceadd0330b24524f6a9ee5bcdb5925def479bd27453d8c1c0d45bde3b934d" Oct 31 13:47:18.679747 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:18.679631 5693 scope.go:122] "RemoveContainer" containerID="fadceadd0330b24524f6a9ee5bcdb5925def479bd27453d8c1c0d45bde3b934d" Oct 31 13:47:18.680046 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:47:18.680025 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fadceadd0330b24524f6a9ee5bcdb5925def479bd27453d8c1c0d45bde3b934d\": container with ID starting with fadceadd0330b24524f6a9ee5bcdb5925def479bd27453d8c1c0d45bde3b934d not found: ID does not exist" containerID="fadceadd0330b24524f6a9ee5bcdb5925def479bd27453d8c1c0d45bde3b934d" Oct 31 13:47:18.680123 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:18.680054 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fadceadd0330b24524f6a9ee5bcdb5925def479bd27453d8c1c0d45bde3b934d"} err="failed to get container status \"fadceadd0330b24524f6a9ee5bcdb5925def479bd27453d8c1c0d45bde3b934d\": rpc error: code = NotFound desc = could not find container \"fadceadd0330b24524f6a9ee5bcdb5925def479bd27453d8c1c0d45bde3b934d\": container with ID starting with fadceadd0330b24524f6a9ee5bcdb5925def479bd27453d8c1c0d45bde3b934d not found: ID does not exist" Oct 31 13:47:18.693167 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:18.693137 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp"] Oct 31 13:47:18.698623 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:47:18.698593 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:47:18.699692 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:18.699669 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-jgcfp"] Oct 31 13:47:18.715613 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:18.715586 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nq6w\" (UniqueName: \"kubernetes.io/projected/a4b60d0a-c847-439d-a156-d6111ffb40a7-kube-api-access-4nq6w\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:18.715613 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:18.715614 5693 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4b60d0a-c847-439d-a156-d6111ffb40a7-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:18.715805 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:18.715629 5693 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4b60d0a-c847-439d-a156-d6111ffb40a7-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:18.715805 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:18.715666 5693 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4b60d0a-c847-439d-a156-d6111ffb40a7-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:19.406267 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:19.406237 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4b60d0a-c847-439d-a156-d6111ffb40a7" path="/var/lib/kubelet/pods/a4b60d0a-c847-439d-a156-d6111ffb40a7/volumes" Oct 31 13:47:48.686553 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:47:48.686477 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:47:59.741489 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:59.741455 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7767cbb97d-2mzd8"] Oct 31 13:47:59.742003 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:59.741703 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7767cbb97d-2mzd8" podUID="052fb6af-f18c-429c-8946-2022c122adfd" containerName="manager" containerID="cri-o://1b2bf9172fab3059a265f2ad41e1444280d9efe1ec24d57a1654ebeb84874d5b" gracePeriod=30 Oct 31 13:47:59.874112 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:59.874081 5693 generic.go:356] "Generic (PLEG): container finished" podID="052fb6af-f18c-429c-8946-2022c122adfd" containerID="1b2bf9172fab3059a265f2ad41e1444280d9efe1ec24d57a1654ebeb84874d5b" exitCode=0 Oct 31 13:47:59.874245 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:59.874147 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7767cbb97d-2mzd8" event={"ID":"052fb6af-f18c-429c-8946-2022c122adfd","Type":"ContainerDied","Data":"1b2bf9172fab3059a265f2ad41e1444280d9efe1ec24d57a1654ebeb84874d5b"} Oct 31 13:47:59.994547 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:47:59.994489 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7767cbb97d-2mzd8" Oct 31 13:48:00.024333 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:00.024298 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/052fb6af-f18c-429c-8946-2022c122adfd-kube-api-access-grwzv\" (UniqueName: \"kubernetes.io/projected/052fb6af-f18c-429c-8946-2022c122adfd-kube-api-access-grwzv\") pod \"052fb6af-f18c-429c-8946-2022c122adfd\" (UID: \"052fb6af-f18c-429c-8946-2022c122adfd\") " Oct 31 13:48:00.024805 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:00.024394 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/052fb6af-f18c-429c-8946-2022c122adfd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/052fb6af-f18c-429c-8946-2022c122adfd-metrics-certs\") pod \"052fb6af-f18c-429c-8946-2022c122adfd\" (UID: \"052fb6af-f18c-429c-8946-2022c122adfd\") " Oct 31 13:48:00.024805 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:00.024446 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/052fb6af-f18c-429c-8946-2022c122adfd-webhook-cert\" (UniqueName: \"kubernetes.io/secret/052fb6af-f18c-429c-8946-2022c122adfd-webhook-cert\") pod \"052fb6af-f18c-429c-8946-2022c122adfd\" (UID: \"052fb6af-f18c-429c-8946-2022c122adfd\") " Oct 31 13:48:00.027677 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:00.027615 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/052fb6af-f18c-429c-8946-2022c122adfd-metrics-certs" pod "052fb6af-f18c-429c-8946-2022c122adfd" (UID: "052fb6af-f18c-429c-8946-2022c122adfd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:48:00.027880 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:00.027855 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/052fb6af-f18c-429c-8946-2022c122adfd-webhook-cert" pod "052fb6af-f18c-429c-8946-2022c122adfd" (UID: "052fb6af-f18c-429c-8946-2022c122adfd"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:48:00.028999 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:00.028970 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/052fb6af-f18c-429c-8946-2022c122adfd-kube-api-access-grwzv" pod "052fb6af-f18c-429c-8946-2022c122adfd" (UID: "052fb6af-f18c-429c-8946-2022c122adfd"). InnerVolumeSpecName "kube-api-access-grwzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:48:00.125453 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:00.125417 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grwzv\" (UniqueName: \"kubernetes.io/projected/052fb6af-f18c-429c-8946-2022c122adfd-kube-api-access-grwzv\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:48:00.125453 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:00.125448 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/052fb6af-f18c-429c-8946-2022c122adfd-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:48:00.125453 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:00.125459 5693 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/052fb6af-f18c-429c-8946-2022c122adfd-webhook-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:48:00.880033 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:00.879994 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7767cbb97d-2mzd8" event={"ID":"052fb6af-f18c-429c-8946-2022c122adfd","Type":"ContainerDied","Data":"4a0cde9a35dcb2a6baf260d405defac000c55dcf34f52515f4e49e7f81084394"} Oct 31 13:48:00.880509 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:00.880065 5693 scope.go:122] "RemoveContainer" containerID="1b2bf9172fab3059a265f2ad41e1444280d9efe1ec24d57a1654ebeb84874d5b" Oct 31 13:48:00.880509 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:00.880008 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7767cbb97d-2mzd8" Oct 31 13:48:00.906736 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:00.906712 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7767cbb97d-2mzd8"] Oct 31 13:48:00.913366 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:00.913341 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7767cbb97d-2mzd8"] Oct 31 13:48:01.407654 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:01.407610 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="052fb6af-f18c-429c-8946-2022c122adfd" path="/var/lib/kubelet/pods/052fb6af-f18c-429c-8946-2022c122adfd/volumes" Oct 31 13:48:18.686174 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:48:18.686134 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:48:23.792589 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.792549 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr"] Oct 31 13:48:23.793135 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793116 5693 cpu_manager.go:414] "containerMap: removing container" podUID="5ee4d8a4-7509-4ae6-8da5-8015a0712a5e" containerName="istio-proxy" Oct 31 13:48:23.793188 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793140 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ee4d8a4-7509-4ae6-8da5-8015a0712a5e" containerName="istio-proxy" Oct 31 13:48:23.793188 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793161 5693 cpu_manager.go:414] "containerMap: removing container" podUID="f8f9a320-4770-43a7-9297-b1d659961376" containerName="payload-processing" Oct 31 13:48:23.793188 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793169 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8f9a320-4770-43a7-9297-b1d659961376" containerName="payload-processing" Oct 31 13:48:23.793309 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793187 5693 cpu_manager.go:414] "containerMap: removing container" podUID="f2b3766f-058c-40e7-9a2d-a7f5028e2b1d" containerName="main" Oct 31 13:48:23.793309 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793197 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2b3766f-058c-40e7-9a2d-a7f5028e2b1d" containerName="main" Oct 31 13:48:23.793309 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793212 5693 cpu_manager.go:414] "containerMap: removing container" podUID="71e7cbcd-adef-4e0e-8944-b3c62e6407bc" containerName="maas-api" Oct 31 13:48:23.793309 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793218 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71e7cbcd-adef-4e0e-8944-b3c62e6407bc" containerName="maas-api" Oct 31 13:48:23.793309 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793228 5693 cpu_manager.go:414] "containerMap: removing container" podUID="c6ff15bc-cbaa-4bbb-a634-415dc98bbb12" containerName="payload-pre-processing" Oct 31 13:48:23.793309 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793235 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6ff15bc-cbaa-4bbb-a634-415dc98bbb12" containerName="payload-pre-processing" Oct 31 13:48:23.793309 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793250 5693 cpu_manager.go:414] "containerMap: removing container" podUID="fba7a7df-1586-461c-b3d7-63ddcf53f218" containerName="payload-pre-processing" Oct 31 13:48:23.793309 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793256 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fba7a7df-1586-461c-b3d7-63ddcf53f218" containerName="payload-pre-processing" Oct 31 13:48:23.793309 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793265 5693 cpu_manager.go:414] "containerMap: removing container" podUID="052fb6af-f18c-429c-8946-2022c122adfd" containerName="manager" Oct 31 13:48:23.793309 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793270 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="052fb6af-f18c-429c-8946-2022c122adfd" containerName="manager" Oct 31 13:48:23.793309 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793277 5693 cpu_manager.go:414] "containerMap: removing container" podUID="a3797b53-75e3-4584-8211-a06b15e03526" containerName="istio-proxy" Oct 31 13:48:23.793309 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793281 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3797b53-75e3-4584-8211-a06b15e03526" containerName="istio-proxy" Oct 31 13:48:23.793309 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793289 5693 cpu_manager.go:414] "containerMap: removing container" podUID="8fb66a94-c399-43fb-945c-610ca63f0d76" containerName="main" Oct 31 13:48:23.793309 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793293 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fb66a94-c399-43fb-945c-610ca63f0d76" containerName="main" Oct 31 13:48:23.793309 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793301 5693 cpu_manager.go:414] "containerMap: removing container" podUID="a4b60d0a-c847-439d-a156-d6111ffb40a7" containerName="istio-proxy" Oct 31 13:48:23.793309 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793305 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4b60d0a-c847-439d-a156-d6111ffb40a7" containerName="istio-proxy" Oct 31 13:48:23.793309 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793315 5693 cpu_manager.go:414] "containerMap: removing container" podUID="32019af4-f819-4127-9d00-d2582affe985" containerName="payload-processing" Oct 31 13:48:23.793309 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793320 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32019af4-f819-4127-9d00-d2582affe985" containerName="payload-processing" Oct 31 13:48:23.793965 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793326 5693 cpu_manager.go:414] "containerMap: removing container" podUID="a509360d-4d23-4078-b1d8-053338d5de5f" containerName="maas-api" Oct 31 13:48:23.793965 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793331 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a509360d-4d23-4078-b1d8-053338d5de5f" containerName="maas-api" Oct 31 13:48:23.793965 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793339 5693 cpu_manager.go:414] "containerMap: removing container" podUID="e5a40b79-9879-4981-ab9f-446e1750a013" containerName="main" Oct 31 13:48:23.793965 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793343 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5a40b79-9879-4981-ab9f-446e1750a013" containerName="main" Oct 31 13:48:23.793965 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793350 5693 cpu_manager.go:414] "containerMap: removing container" podUID="efeec4f0-9411-4add-8d49-586201474004" containerName="payload-processing" Oct 31 13:48:23.793965 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793355 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efeec4f0-9411-4add-8d49-586201474004" containerName="payload-processing" Oct 31 13:48:23.793965 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793429 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ee4d8a4-7509-4ae6-8da5-8015a0712a5e" containerName="istio-proxy" Oct 31 13:48:23.793965 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793439 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8f9a320-4770-43a7-9297-b1d659961376" containerName="payload-processing" Oct 31 13:48:23.793965 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793445 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2b3766f-058c-40e7-9a2d-a7f5028e2b1d" containerName="main" Oct 31 13:48:23.793965 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793451 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="71e7cbcd-adef-4e0e-8944-b3c62e6407bc" containerName="maas-api" Oct 31 13:48:23.793965 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793456 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6ff15bc-cbaa-4bbb-a634-415dc98bbb12" containerName="payload-pre-processing" Oct 31 13:48:23.793965 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793461 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="fba7a7df-1586-461c-b3d7-63ddcf53f218" containerName="payload-pre-processing" Oct 31 13:48:23.793965 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793466 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="052fb6af-f18c-429c-8946-2022c122adfd" containerName="manager" Oct 31 13:48:23.793965 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793471 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3797b53-75e3-4584-8211-a06b15e03526" containerName="istio-proxy" Oct 31 13:48:23.793965 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793476 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fb66a94-c399-43fb-945c-610ca63f0d76" containerName="main" Oct 31 13:48:23.793965 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793481 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4b60d0a-c847-439d-a156-d6111ffb40a7" containerName="istio-proxy" Oct 31 13:48:23.793965 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793487 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="32019af4-f819-4127-9d00-d2582affe985" containerName="payload-processing" Oct 31 13:48:23.793965 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793492 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="a509360d-4d23-4078-b1d8-053338d5de5f" containerName="maas-api" Oct 31 13:48:23.793965 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793497 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5a40b79-9879-4981-ab9f-446e1750a013" containerName="main" Oct 31 13:48:23.793965 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.793502 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="efeec4f0-9411-4add-8d49-586201474004" containerName="payload-processing" Oct 31 13:48:23.796620 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.796603 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" Oct 31 13:48:23.810197 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.810167 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr"] Oct 31 13:48:23.967412 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.967375 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/073cc2ff-2dc7-4376-8449-f45cd637eeea-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-bc5lr\" (UID: \"073cc2ff-2dc7-4376-8449-f45cd637eeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" Oct 31 13:48:23.967613 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.967423 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/073cc2ff-2dc7-4376-8449-f45cd637eeea-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-bc5lr\" (UID: \"073cc2ff-2dc7-4376-8449-f45cd637eeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" Oct 31 13:48:23.967613 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.967444 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/073cc2ff-2dc7-4376-8449-f45cd637eeea-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-bc5lr\" (UID: \"073cc2ff-2dc7-4376-8449-f45cd637eeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" Oct 31 13:48:23.967613 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.967570 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/073cc2ff-2dc7-4376-8449-f45cd637eeea-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-bc5lr\" (UID: \"073cc2ff-2dc7-4376-8449-f45cd637eeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" Oct 31 13:48:23.967821 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.967633 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/073cc2ff-2dc7-4376-8449-f45cd637eeea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-bc5lr\" (UID: \"073cc2ff-2dc7-4376-8449-f45cd637eeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" Oct 31 13:48:23.967821 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.967700 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/073cc2ff-2dc7-4376-8449-f45cd637eeea-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-bc5lr\" (UID: \"073cc2ff-2dc7-4376-8449-f45cd637eeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" Oct 31 13:48:23.967821 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.967743 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/073cc2ff-2dc7-4376-8449-f45cd637eeea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-bc5lr\" (UID: \"073cc2ff-2dc7-4376-8449-f45cd637eeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" Oct 31 13:48:23.967821 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.967797 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbz7q\" (UniqueName: \"kubernetes.io/projected/073cc2ff-2dc7-4376-8449-f45cd637eeea-kube-api-access-zbz7q\") pod \"maas-default-gateway-openshift-default-fc8c56547-bc5lr\" (UID: \"073cc2ff-2dc7-4376-8449-f45cd637eeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" Oct 31 13:48:23.968007 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:23.967824 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/073cc2ff-2dc7-4376-8449-f45cd637eeea-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-bc5lr\" (UID: \"073cc2ff-2dc7-4376-8449-f45cd637eeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" Oct 31 13:48:24.068532 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:24.068449 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/073cc2ff-2dc7-4376-8449-f45cd637eeea-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-bc5lr\" (UID: \"073cc2ff-2dc7-4376-8449-f45cd637eeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" Oct 31 13:48:24.068532 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:24.068489 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/073cc2ff-2dc7-4376-8449-f45cd637eeea-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-bc5lr\" (UID: \"073cc2ff-2dc7-4376-8449-f45cd637eeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" Oct 31 13:48:24.068532 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:24.068531 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/073cc2ff-2dc7-4376-8449-f45cd637eeea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-bc5lr\" (UID: \"073cc2ff-2dc7-4376-8449-f45cd637eeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" Oct 31 13:48:24.068801 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:24.068552 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/073cc2ff-2dc7-4376-8449-f45cd637eeea-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-bc5lr\" (UID: \"073cc2ff-2dc7-4376-8449-f45cd637eeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" Oct 31 13:48:24.068801 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:24.068570 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/073cc2ff-2dc7-4376-8449-f45cd637eeea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-bc5lr\" (UID: \"073cc2ff-2dc7-4376-8449-f45cd637eeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" Oct 31 13:48:24.068801 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:24.068615 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbz7q\" (UniqueName: \"kubernetes.io/projected/073cc2ff-2dc7-4376-8449-f45cd637eeea-kube-api-access-zbz7q\") pod \"maas-default-gateway-openshift-default-fc8c56547-bc5lr\" (UID: \"073cc2ff-2dc7-4376-8449-f45cd637eeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" Oct 31 13:48:24.068801 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:24.068664 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/073cc2ff-2dc7-4376-8449-f45cd637eeea-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-bc5lr\" (UID: \"073cc2ff-2dc7-4376-8449-f45cd637eeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" Oct 31 13:48:24.068801 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:24.068747 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/073cc2ff-2dc7-4376-8449-f45cd637eeea-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-bc5lr\" (UID: \"073cc2ff-2dc7-4376-8449-f45cd637eeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" Oct 31 13:48:24.068801 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:24.068797 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/073cc2ff-2dc7-4376-8449-f45cd637eeea-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-bc5lr\" (UID: \"073cc2ff-2dc7-4376-8449-f45cd637eeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" Oct 31 13:48:24.069098 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:24.068933 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/073cc2ff-2dc7-4376-8449-f45cd637eeea-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-bc5lr\" (UID: \"073cc2ff-2dc7-4376-8449-f45cd637eeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" Oct 31 13:48:24.069098 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:24.068995 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/073cc2ff-2dc7-4376-8449-f45cd637eeea-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-bc5lr\" (UID: \"073cc2ff-2dc7-4376-8449-f45cd637eeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" Oct 31 13:48:24.069375 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:24.069330 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/073cc2ff-2dc7-4376-8449-f45cd637eeea-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-bc5lr\" (UID: \"073cc2ff-2dc7-4376-8449-f45cd637eeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" Oct 31 13:48:24.069502 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:24.069419 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/073cc2ff-2dc7-4376-8449-f45cd637eeea-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-bc5lr\" (UID: \"073cc2ff-2dc7-4376-8449-f45cd637eeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" Oct 31 13:48:24.069710 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:24.069688 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/073cc2ff-2dc7-4376-8449-f45cd637eeea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-bc5lr\" (UID: \"073cc2ff-2dc7-4376-8449-f45cd637eeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" Oct 31 13:48:24.070917 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:24.070899 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/073cc2ff-2dc7-4376-8449-f45cd637eeea-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-bc5lr\" (UID: \"073cc2ff-2dc7-4376-8449-f45cd637eeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" Oct 31 13:48:24.071216 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:24.071196 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/073cc2ff-2dc7-4376-8449-f45cd637eeea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-bc5lr\" (UID: \"073cc2ff-2dc7-4376-8449-f45cd637eeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" Oct 31 13:48:24.081695 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:24.081668 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/073cc2ff-2dc7-4376-8449-f45cd637eeea-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-bc5lr\" (UID: \"073cc2ff-2dc7-4376-8449-f45cd637eeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" Oct 31 13:48:24.081818 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:24.081717 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbz7q\" (UniqueName: \"kubernetes.io/projected/073cc2ff-2dc7-4376-8449-f45cd637eeea-kube-api-access-zbz7q\") pod \"maas-default-gateway-openshift-default-fc8c56547-bc5lr\" (UID: \"073cc2ff-2dc7-4376-8449-f45cd637eeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" Oct 31 13:48:24.109287 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:24.109267 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" Oct 31 13:48:24.396096 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:24.396010 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr"] Oct 31 13:48:24.398383 ip-10-0-131-157 kubenswrapper[5693]: W1031 13:48:24.398354 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod073cc2ff_2dc7_4376_8449_f45cd637eeea.slice/crio-6e729ea269ea5b11745b704347a702d83bedee924f7c435dcf87fdb57cdb7826 WatchSource:0}: Error finding container 6e729ea269ea5b11745b704347a702d83bedee924f7c435dcf87fdb57cdb7826: Status 404 returned error can't find the container with id 6e729ea269ea5b11745b704347a702d83bedee924f7c435dcf87fdb57cdb7826 Oct 31 13:48:24.400622 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:24.400587 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:48:24.400746 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:24.400702 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:48:24.400796 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:24.400752 5693 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 13:48:25.010120 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:25.010081 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" event={"ID":"073cc2ff-2dc7-4376-8449-f45cd637eeea","Type":"ContainerStarted","Data":"f4fe5a2ffa1e9d2b86fadb5568d74eac66460f19dc425fba8f0936e8d70a74df"} Oct 31 13:48:25.010525 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:25.010181 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" event={"ID":"073cc2ff-2dc7-4376-8449-f45cd637eeea","Type":"ContainerStarted","Data":"6e729ea269ea5b11745b704347a702d83bedee924f7c435dcf87fdb57cdb7826"} Oct 31 13:48:25.040876 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:25.040816 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" podStartSLOduration=2.040800427 podStartE2EDuration="2.040800427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:48:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:48:25.040439578 +0000 UTC m=+4555.748025872" watchObservedRunningTime="2026-10-31 13:48:25.040800427 +0000 UTC m=+4555.748386724" Oct 31 13:48:25.109561 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:25.109525 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" Oct 31 13:48:25.111286 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:25.111257 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-bc5lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body= Oct 31 13:48:25.111420 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:25.111306 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" podUID="073cc2ff-2dc7-4376-8449-f45cd637eeea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" Oct 31 13:48:26.110537 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:26.110505 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-bc5lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body= Oct 31 13:48:26.110945 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:26.110556 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" podUID="073cc2ff-2dc7-4376-8449-f45cd637eeea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" Oct 31 13:48:27.110472 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:27.110431 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-bc5lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body= Oct 31 13:48:27.110693 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:27.110497 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" podUID="073cc2ff-2dc7-4376-8449-f45cd637eeea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" Oct 31 13:48:28.110172 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:28.110132 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-bc5lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body= Oct 31 13:48:28.110374 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:28.110190 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" podUID="073cc2ff-2dc7-4376-8449-f45cd637eeea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" Oct 31 13:48:29.109700 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:29.109624 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-bc5lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body= Oct 31 13:48:29.110159 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:29.109723 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" podUID="073cc2ff-2dc7-4376-8449-f45cd637eeea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" Oct 31 13:48:30.109822 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:30.109786 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-bc5lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body= Oct 31 13:48:30.110236 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:30.109855 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" podUID="073cc2ff-2dc7-4376-8449-f45cd637eeea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" Oct 31 13:48:31.110305 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:31.110265 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-bc5lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body= Oct 31 13:48:31.110718 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:31.110325 5693 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" podUID="073cc2ff-2dc7-4376-8449-f45cd637eeea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" Oct 31 13:48:32.801407 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:32.801374 5693 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" Oct 31 13:48:32.802027 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:32.801998 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" Oct 31 13:48:32.802847 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:48:32.802828 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" Oct 31 13:48:48.686539 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:48:48.686506 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:49:18.688594 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:49:18.688515 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:49:48.685565 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:49:48.685529 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:50:12.024357 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:50:12.024324 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4f3bc9f-5tf48"] Oct 31 13:50:12.026707 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:50:12.026681 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4f3bc9f-5tf48"] Oct 31 13:50:13.406147 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:50:13.406117 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27ece4ce-6746-4405-b30b-65656e868f4b" path="/var/lib/kubelet/pods/27ece4ce-6746-4405-b30b-65656e868f4b/volumes" Oct 31 13:50:18.689761 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:50:18.689719 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:50:48.761997 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:50:48.761894 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:50:48.848947 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:50:48.848886 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-bc5lr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:50:48.849161 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:50:48.849012 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" podUID="073cc2ff-2dc7-4376-8449-f45cd637eeea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:51:11.963857 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:51:11.963811 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fv482 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:51:11.964376 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:51:11.963870 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" podUID="c0bdd69c-691b-47b2-ba17-2e0ac7185643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:51:18.686202 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:51:18.686165 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:51:32.286960 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:51:32.286929 5693 scope.go:122] "RemoveContainer" containerID="242c61ad9c4983e7a1310fd6ddded6eacd9bd828c34628c261a30d2cfff86e3a" Oct 31 13:51:48.686698 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:51:48.686656 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:52:18.685788 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:52:18.685748 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:52:48.686055 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:52:48.686017 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:53:18.686812 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:53:18.686775 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:53:48.761045 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:53:48.760857 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:53:48.851885 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:53:48.851845 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-bc5lr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:53:48.852251 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:53:48.852186 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" podUID="073cc2ff-2dc7-4376-8449-f45cd637eeea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:54:18.686184 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:54:18.686144 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:54:48.686616 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:54:48.686579 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:54:58.708322 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:54:58.708285 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-n4pg9"] Oct 31 13:55:03.709538 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:55:03.709441 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n4pg9" podUID="3d3532b0-6567-4331-a327-a7b917fceeec" containerName="payload-pre-processing" containerID="cri-o://92da70c80bf808ad69af71268efcc43f46f0cde4c1ae0fbe23677134cf6e2823" gracePeriod=25 Oct 31 13:55:03.964715 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:55:03.964631 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n4pg9" Oct 31 13:55:04.066009 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:55:04.065978 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d3532b0-6567-4331-a327-a7b917fceeec-kube-api-access-4q8z6\" (UniqueName: \"kubernetes.io/projected/3d3532b0-6567-4331-a327-a7b917fceeec-kube-api-access-4q8z6\") pod \"3d3532b0-6567-4331-a327-a7b917fceeec\" (UID: \"3d3532b0-6567-4331-a327-a7b917fceeec\") " Oct 31 13:55:04.066308 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:55:04.066033 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d3532b0-6567-4331-a327-a7b917fceeec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d3532b0-6567-4331-a327-a7b917fceeec-plugins-config-volume\") pod \"3d3532b0-6567-4331-a327-a7b917fceeec\" (UID: \"3d3532b0-6567-4331-a327-a7b917fceeec\") " Oct 31 13:55:04.066433 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:55:04.066406 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d3532b0-6567-4331-a327-a7b917fceeec-plugins-config-volume" pod "3d3532b0-6567-4331-a327-a7b917fceeec" (UID: "3d3532b0-6567-4331-a327-a7b917fceeec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:55:04.068062 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:55:04.068044 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d3532b0-6567-4331-a327-a7b917fceeec-kube-api-access-4q8z6" pod "3d3532b0-6567-4331-a327-a7b917fceeec" (UID: "3d3532b0-6567-4331-a327-a7b917fceeec"). InnerVolumeSpecName "kube-api-access-4q8z6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:55:04.111275 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:55:04.111244 5693 generic.go:356] "Generic (PLEG): container finished" podID="3d3532b0-6567-4331-a327-a7b917fceeec" containerID="92da70c80bf808ad69af71268efcc43f46f0cde4c1ae0fbe23677134cf6e2823" exitCode=0 Oct 31 13:55:04.111476 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:55:04.111306 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n4pg9" Oct 31 13:55:04.111476 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:55:04.111342 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n4pg9" event={"ID":"3d3532b0-6567-4331-a327-a7b917fceeec","Type":"ContainerDied","Data":"92da70c80bf808ad69af71268efcc43f46f0cde4c1ae0fbe23677134cf6e2823"} Oct 31 13:55:04.111476 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:55:04.111398 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n4pg9" event={"ID":"3d3532b0-6567-4331-a327-a7b917fceeec","Type":"ContainerDied","Data":"bff0c6a0fb36d9d7e6f2b3073fcdbf57a94c7c592632466e311fdffa771bb406"} Oct 31 13:55:04.111476 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:55:04.111424 5693 scope.go:122] "RemoveContainer" containerID="92da70c80bf808ad69af71268efcc43f46f0cde4c1ae0fbe23677134cf6e2823" Oct 31 13:55:04.121340 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:55:04.121321 5693 scope.go:122] "RemoveContainer" containerID="92da70c80bf808ad69af71268efcc43f46f0cde4c1ae0fbe23677134cf6e2823" Oct 31 13:55:04.121603 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:55:04.121585 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92da70c80bf808ad69af71268efcc43f46f0cde4c1ae0fbe23677134cf6e2823\": container with ID starting with 92da70c80bf808ad69af71268efcc43f46f0cde4c1ae0fbe23677134cf6e2823 not found: ID does not exist" containerID="92da70c80bf808ad69af71268efcc43f46f0cde4c1ae0fbe23677134cf6e2823" Oct 31 13:55:04.121704 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:55:04.121612 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92da70c80bf808ad69af71268efcc43f46f0cde4c1ae0fbe23677134cf6e2823"} err="failed to get container status \"92da70c80bf808ad69af71268efcc43f46f0cde4c1ae0fbe23677134cf6e2823\": rpc error: code = NotFound desc = could not find container \"92da70c80bf808ad69af71268efcc43f46f0cde4c1ae0fbe23677134cf6e2823\": container with ID starting with 92da70c80bf808ad69af71268efcc43f46f0cde4c1ae0fbe23677134cf6e2823 not found: ID does not exist" Oct 31 13:55:04.133117 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:55:04.133092 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-n4pg9"] Oct 31 13:55:04.136768 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:55:04.136746 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-n4pg9"] Oct 31 13:55:04.167203 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:55:04.167182 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q8z6\" (UniqueName: \"kubernetes.io/projected/3d3532b0-6567-4331-a327-a7b917fceeec-kube-api-access-4q8z6\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:04.167203 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:55:04.167200 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d3532b0-6567-4331-a327-a7b917fceeec-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:05.406955 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:55:05.406922 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d3532b0-6567-4331-a327-a7b917fceeec" path="/var/lib/kubelet/pods/3d3532b0-6567-4331-a327-a7b917fceeec/volumes" Oct 31 13:55:12.794388 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:55:12.794278 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-w7mmf"] Oct 31 13:55:17.794992 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:55:17.794911 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-55fbbbf9dc-w7mmf" podUID="6485629e-cfde-468c-93b0-9f68ca0ded66" containerName="payload-processing" containerID="cri-o://dbceae1e79dcb72382b653c4cb4a44bb4f272da55ae75e0678ec7551bdcbc2e7" gracePeriod=25 Oct 31 13:55:18.050314 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:55:18.050247 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-w7mmf" Oct 31 13:55:18.182539 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:55:18.182507 5693 generic.go:356] "Generic (PLEG): container finished" podID="6485629e-cfde-468c-93b0-9f68ca0ded66" containerID="dbceae1e79dcb72382b653c4cb4a44bb4f272da55ae75e0678ec7551bdcbc2e7" exitCode=0 Oct 31 13:55:18.182776 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:55:18.182552 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-w7mmf" Oct 31 13:55:18.182776 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:55:18.182566 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-w7mmf" event={"ID":"6485629e-cfde-468c-93b0-9f68ca0ded66","Type":"ContainerDied","Data":"dbceae1e79dcb72382b653c4cb4a44bb4f272da55ae75e0678ec7551bdcbc2e7"} Oct 31 13:55:18.182776 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:55:18.182613 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-w7mmf" event={"ID":"6485629e-cfde-468c-93b0-9f68ca0ded66","Type":"ContainerDied","Data":"69907cde19ec9a563dcf05df485ea39dd0a6ea2595b1ab049f6fe41bb56e445b"} Oct 31 13:55:18.182776 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:55:18.182664 5693 scope.go:122] "RemoveContainer" containerID="dbceae1e79dcb72382b653c4cb4a44bb4f272da55ae75e0678ec7551bdcbc2e7" Oct 31 13:55:18.191475 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:55:18.191456 5693 scope.go:122] "RemoveContainer" containerID="dbceae1e79dcb72382b653c4cb4a44bb4f272da55ae75e0678ec7551bdcbc2e7" Oct 31 13:55:18.191783 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:55:18.191762 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbceae1e79dcb72382b653c4cb4a44bb4f272da55ae75e0678ec7551bdcbc2e7\": container with ID starting with dbceae1e79dcb72382b653c4cb4a44bb4f272da55ae75e0678ec7551bdcbc2e7 not found: ID does not exist" containerID="dbceae1e79dcb72382b653c4cb4a44bb4f272da55ae75e0678ec7551bdcbc2e7" Oct 31 13:55:18.191863 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:55:18.191800 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbceae1e79dcb72382b653c4cb4a44bb4f272da55ae75e0678ec7551bdcbc2e7"} err="failed to get container status \"dbceae1e79dcb72382b653c4cb4a44bb4f272da55ae75e0678ec7551bdcbc2e7\": rpc error: code = NotFound desc = could not find container \"dbceae1e79dcb72382b653c4cb4a44bb4f272da55ae75e0678ec7551bdcbc2e7\": container with ID starting with dbceae1e79dcb72382b653c4cb4a44bb4f272da55ae75e0678ec7551bdcbc2e7 not found: ID does not exist" Oct 31 13:55:18.219394 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:55:18.219364 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6485629e-cfde-468c-93b0-9f68ca0ded66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6485629e-cfde-468c-93b0-9f68ca0ded66-plugins-config-volume\") pod \"6485629e-cfde-468c-93b0-9f68ca0ded66\" (UID: \"6485629e-cfde-468c-93b0-9f68ca0ded66\") " Oct 31 13:55:18.219536 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:55:18.219482 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6485629e-cfde-468c-93b0-9f68ca0ded66-kube-api-access-7tdhw\" (UniqueName: \"kubernetes.io/projected/6485629e-cfde-468c-93b0-9f68ca0ded66-kube-api-access-7tdhw\") pod \"6485629e-cfde-468c-93b0-9f68ca0ded66\" (UID: \"6485629e-cfde-468c-93b0-9f68ca0ded66\") " Oct 31 13:55:18.219730 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:55:18.219710 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6485629e-cfde-468c-93b0-9f68ca0ded66-plugins-config-volume" pod "6485629e-cfde-468c-93b0-9f68ca0ded66" (UID: "6485629e-cfde-468c-93b0-9f68ca0ded66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:55:18.219929 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:55:18.219911 5693 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6485629e-cfde-468c-93b0-9f68ca0ded66-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:18.221728 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:55:18.221704 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6485629e-cfde-468c-93b0-9f68ca0ded66-kube-api-access-7tdhw" pod "6485629e-cfde-468c-93b0-9f68ca0ded66" (UID: "6485629e-cfde-468c-93b0-9f68ca0ded66"). InnerVolumeSpecName "kube-api-access-7tdhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:55:18.320747 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:55:18.320631 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tdhw\" (UniqueName: \"kubernetes.io/projected/6485629e-cfde-468c-93b0-9f68ca0ded66-kube-api-access-7tdhw\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:18.505543 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:55:18.505513 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-w7mmf"] Oct 31 13:55:18.513125 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:55:18.513086 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-w7mmf"] Oct 31 13:55:18.686552 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:55:18.686516 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:55:19.406432 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:55:19.406401 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6485629e-cfde-468c-93b0-9f68ca0ded66" path="/var/lib/kubelet/pods/6485629e-cfde-468c-93b0-9f68ca0ded66/volumes" Oct 31 13:55:33.849899 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:55:33.849830 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-bc5lr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:55:33.850583 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:55:33.850547 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" podUID="073cc2ff-2dc7-4376-8449-f45cd637eeea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:55:48.686864 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:55:48.686827 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:56:18.686359 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:56:18.686316 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:56:48.687015 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:56:48.686937 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:57:18.686442 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:57:18.686342 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:57:48.754385 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:57:48.754307 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:57:48.846392 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:57:48.845809 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-bc5lr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:57:48.846392 ip-10-0-131-157 kubenswrapper[5693]: I1031 13:57:48.845897 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-bc5lr" podUID="073cc2ff-2dc7-4376-8449-f45cd637eeea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:58:18.686875 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:58:18.686789 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:58:48.687381 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:58:48.687336 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:59:18.686471 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:59:18.686433 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 13:59:48.686582 ip-10-0-131-157 kubenswrapper[5693]: E1031 13:59:48.686490 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 14:00:18.692543 ip-10-0-131-157 kubenswrapper[5693]: E1031 14:00:18.692502 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 14:00:48.687214 ip-10-0-131-157 kubenswrapper[5693]: E1031 14:00:48.687174 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 14:01:18.686024 ip-10-0-131-157 kubenswrapper[5693]: E1031 14:01:18.685985 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 14:01:48.687034 ip-10-0-131-157 kubenswrapper[5693]: E1031 14:01:48.686987 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 14:02:18.686785 ip-10-0-131-157 kubenswrapper[5693]: E1031 14:02:18.686687 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 14:02:48.686598 ip-10-0-131-157 kubenswrapper[5693]: E1031 14:02:48.686505 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 14:03:18.686517 ip-10-0-131-157 kubenswrapper[5693]: E1031 14:03:18.686483 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 14:03:48.686818 ip-10-0-131-157 kubenswrapper[5693]: E1031 14:03:48.686783 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 14:04:18.687054 ip-10-0-131-157 kubenswrapper[5693]: E1031 14:04:18.686971 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 14:04:41.941027 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:04:41.940972 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fv482 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 14:04:41.941727 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:04:41.941109 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" podUID="c0bdd69c-691b-47b2-ba17-2e0ac7185643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 14:04:48.687513 ip-10-0-131-157 kubenswrapper[5693]: E1031 14:04:48.687475 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 14:05:18.687941 ip-10-0-131-157 kubenswrapper[5693]: E1031 14:05:18.687906 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 14:05:26.966839 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:26.966794 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fv482 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 14:05:26.967251 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:26.966870 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" podUID="c0bdd69c-691b-47b2-ba17-2e0ac7185643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 14:05:28.061062 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:28.061026 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-547b4fbdb9-zxdgg"] Oct 31 14:05:28.061549 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:28.061242 5693 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-zxdgg" podUID="796a4469-22c6-4d99-b8e4-34f417c8661d" containerName="maas-api" containerID="cri-o://dfe7a513520cc608018c50c5998734f71a6c2f738cfa1ccb9c436d3498cebe5d" gracePeriod=30 Oct 31 14:05:29.328033 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.328004 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-zxdgg" Oct 31 14:05:29.395414 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.395387 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/796a4469-22c6-4d99-b8e4-34f417c8661d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/796a4469-22c6-4d99-b8e4-34f417c8661d-maas-api-tls\") pod \"796a4469-22c6-4d99-b8e4-34f417c8661d\" (UID: \"796a4469-22c6-4d99-b8e4-34f417c8661d\") " Oct 31 14:05:29.395603 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.395437 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/796a4469-22c6-4d99-b8e4-34f417c8661d-kube-api-access-rmq9s\" (UniqueName: \"kubernetes.io/projected/796a4469-22c6-4d99-b8e4-34f417c8661d-kube-api-access-rmq9s\") pod \"796a4469-22c6-4d99-b8e4-34f417c8661d\" (UID: \"796a4469-22c6-4d99-b8e4-34f417c8661d\") " Oct 31 14:05:29.395603 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.395479 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/796a4469-22c6-4d99-b8e4-34f417c8661d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/796a4469-22c6-4d99-b8e4-34f417c8661d-openshift-service-ca\") pod \"796a4469-22c6-4d99-b8e4-34f417c8661d\" (UID: \"796a4469-22c6-4d99-b8e4-34f417c8661d\") " Oct 31 14:05:29.395603 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.395581 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/796a4469-22c6-4d99-b8e4-34f417c8661d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/796a4469-22c6-4d99-b8e4-34f417c8661d-metrics-certs\") pod \"796a4469-22c6-4d99-b8e4-34f417c8661d\" (UID: \"796a4469-22c6-4d99-b8e4-34f417c8661d\") " Oct 31 14:05:29.395990 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.395962 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/796a4469-22c6-4d99-b8e4-34f417c8661d-openshift-service-ca" pod "796a4469-22c6-4d99-b8e4-34f417c8661d" (UID: "796a4469-22c6-4d99-b8e4-34f417c8661d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:05:29.397878 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.397847 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/796a4469-22c6-4d99-b8e4-34f417c8661d-maas-api-tls" pod "796a4469-22c6-4d99-b8e4-34f417c8661d" (UID: "796a4469-22c6-4d99-b8e4-34f417c8661d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:05:29.397992 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.397905 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/796a4469-22c6-4d99-b8e4-34f417c8661d-metrics-certs" pod "796a4469-22c6-4d99-b8e4-34f417c8661d" (UID: "796a4469-22c6-4d99-b8e4-34f417c8661d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:05:29.397992 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.397945 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/796a4469-22c6-4d99-b8e4-34f417c8661d-kube-api-access-rmq9s" pod "796a4469-22c6-4d99-b8e4-34f417c8661d" (UID: "796a4469-22c6-4d99-b8e4-34f417c8661d"). InnerVolumeSpecName "kube-api-access-rmq9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:05:29.425383 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.425355 5693 generic.go:356] "Generic (PLEG): container finished" podID="796a4469-22c6-4d99-b8e4-34f417c8661d" containerID="dfe7a513520cc608018c50c5998734f71a6c2f738cfa1ccb9c436d3498cebe5d" exitCode=0 Oct 31 14:05:29.425515 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.425410 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-zxdgg" Oct 31 14:05:29.425515 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.425437 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-zxdgg" event={"ID":"796a4469-22c6-4d99-b8e4-34f417c8661d","Type":"ContainerDied","Data":"dfe7a513520cc608018c50c5998734f71a6c2f738cfa1ccb9c436d3498cebe5d"} Oct 31 14:05:29.425667 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.425524 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-547b4fbdb9-zxdgg" event={"ID":"796a4469-22c6-4d99-b8e4-34f417c8661d","Type":"ContainerDied","Data":"04738d56d09d925a4bc4a1cd8d62b0136eca8c6114357b97ca4ae561df696d5a"} Oct 31 14:05:29.425667 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.425555 5693 scope.go:122] "RemoveContainer" containerID="dfe7a513520cc608018c50c5998734f71a6c2f738cfa1ccb9c436d3498cebe5d" Oct 31 14:05:29.434565 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.434548 5693 scope.go:122] "RemoveContainer" containerID="dfe7a513520cc608018c50c5998734f71a6c2f738cfa1ccb9c436d3498cebe5d" Oct 31 14:05:29.434876 ip-10-0-131-157 kubenswrapper[5693]: E1031 14:05:29.434857 5693 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfe7a513520cc608018c50c5998734f71a6c2f738cfa1ccb9c436d3498cebe5d\": container with ID starting with dfe7a513520cc608018c50c5998734f71a6c2f738cfa1ccb9c436d3498cebe5d not found: ID does not exist" containerID="dfe7a513520cc608018c50c5998734f71a6c2f738cfa1ccb9c436d3498cebe5d" Oct 31 14:05:29.434933 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.434884 5693 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfe7a513520cc608018c50c5998734f71a6c2f738cfa1ccb9c436d3498cebe5d"} err="failed to get container status \"dfe7a513520cc608018c50c5998734f71a6c2f738cfa1ccb9c436d3498cebe5d\": rpc error: code = NotFound desc = could not find container \"dfe7a513520cc608018c50c5998734f71a6c2f738cfa1ccb9c436d3498cebe5d\": container with ID starting with dfe7a513520cc608018c50c5998734f71a6c2f738cfa1ccb9c436d3498cebe5d not found: ID does not exist" Oct 31 14:05:29.442971 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.442951 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-547b4fbdb9-zxdgg"] Oct 31 14:05:29.446656 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.446621 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-547b4fbdb9-zxdgg"] Oct 31 14:05:29.496737 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.496704 5693 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/796a4469-22c6-4d99-b8e4-34f417c8661d-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:05:29.496737 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.496729 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/796a4469-22c6-4d99-b8e4-34f417c8661d-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:05:29.496737 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.496742 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmq9s\" (UniqueName: \"kubernetes.io/projected/796a4469-22c6-4d99-b8e4-34f417c8661d-kube-api-access-rmq9s\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:05:29.496961 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.496771 5693 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/796a4469-22c6-4d99-b8e4-34f417c8661d-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:05:29.529371 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.529341 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6f2c4d4-852rr"] Oct 31 14:05:29.529713 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.529700 5693 cpu_manager.go:414] "containerMap: removing container" podUID="796a4469-22c6-4d99-b8e4-34f417c8661d" containerName="maas-api" Oct 31 14:05:29.529764 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.529714 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="796a4469-22c6-4d99-b8e4-34f417c8661d" containerName="maas-api" Oct 31 14:05:29.529764 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.529734 5693 cpu_manager.go:414] "containerMap: removing container" podUID="6485629e-cfde-468c-93b0-9f68ca0ded66" containerName="payload-processing" Oct 31 14:05:29.529764 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.529739 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6485629e-cfde-468c-93b0-9f68ca0ded66" containerName="payload-processing" Oct 31 14:05:29.529764 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.529752 5693 cpu_manager.go:414] "containerMap: removing container" podUID="3d3532b0-6567-4331-a327-a7b917fceeec" containerName="payload-pre-processing" Oct 31 14:05:29.529764 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.529757 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d3532b0-6567-4331-a327-a7b917fceeec" containerName="payload-pre-processing" Oct 31 14:05:29.529927 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.529831 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="796a4469-22c6-4d99-b8e4-34f417c8661d" containerName="maas-api" Oct 31 14:05:29.529927 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.529840 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="6485629e-cfde-468c-93b0-9f68ca0ded66" containerName="payload-processing" Oct 31 14:05:29.529927 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.529848 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d3532b0-6567-4331-a327-a7b917fceeec" containerName="payload-pre-processing" Oct 31 14:05:29.534217 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.534199 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f2c4d4-852rr" Oct 31 14:05:29.541209 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.541188 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:05:29.541446 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.541421 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 14:05:29.542417 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.542396 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6f2c4d4-852rr"] Oct 31 14:05:29.598061 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.598031 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vvd7\" (UniqueName: \"kubernetes.io/projected/3b3398fa-0028-4349-abd3-99d1cfe8247c-kube-api-access-6vvd7\") pod \"maas-api-revoke-sub-c6f2c4d4-852rr\" (UID: \"3b3398fa-0028-4349-abd3-99d1cfe8247c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f2c4d4-852rr" Oct 31 14:05:29.598245 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.598154 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b3398fa-0028-4349-abd3-99d1cfe8247c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6f2c4d4-852rr\" (UID: \"3b3398fa-0028-4349-abd3-99d1cfe8247c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f2c4d4-852rr" Oct 31 14:05:29.598245 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.598183 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b3398fa-0028-4349-abd3-99d1cfe8247c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6f2c4d4-852rr\" (UID: \"3b3398fa-0028-4349-abd3-99d1cfe8247c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f2c4d4-852rr" Oct 31 14:05:29.699255 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.699169 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vvd7\" (UniqueName: \"kubernetes.io/projected/3b3398fa-0028-4349-abd3-99d1cfe8247c-kube-api-access-6vvd7\") pod \"maas-api-revoke-sub-c6f2c4d4-852rr\" (UID: \"3b3398fa-0028-4349-abd3-99d1cfe8247c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f2c4d4-852rr" Oct 31 14:05:29.699255 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.699243 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b3398fa-0028-4349-abd3-99d1cfe8247c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6f2c4d4-852rr\" (UID: \"3b3398fa-0028-4349-abd3-99d1cfe8247c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f2c4d4-852rr" Oct 31 14:05:29.699501 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.699265 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b3398fa-0028-4349-abd3-99d1cfe8247c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6f2c4d4-852rr\" (UID: \"3b3398fa-0028-4349-abd3-99d1cfe8247c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f2c4d4-852rr" Oct 31 14:05:29.699928 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.699905 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b3398fa-0028-4349-abd3-99d1cfe8247c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6f2c4d4-852rr\" (UID: \"3b3398fa-0028-4349-abd3-99d1cfe8247c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f2c4d4-852rr" Oct 31 14:05:29.701732 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.701715 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b3398fa-0028-4349-abd3-99d1cfe8247c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6f2c4d4-852rr\" (UID: \"3b3398fa-0028-4349-abd3-99d1cfe8247c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f2c4d4-852rr" Oct 31 14:05:29.711659 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.711620 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vvd7\" (UniqueName: \"kubernetes.io/projected/3b3398fa-0028-4349-abd3-99d1cfe8247c-kube-api-access-6vvd7\") pod \"maas-api-revoke-sub-c6f2c4d4-852rr\" (UID: \"3b3398fa-0028-4349-abd3-99d1cfe8247c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f2c4d4-852rr" Oct 31 14:05:29.849061 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:29.849025 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f2c4d4-852rr" Oct 31 14:05:30.124227 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:30.124196 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6f2c4d4-852rr"] Oct 31 14:05:30.126206 ip-10-0-131-157 kubenswrapper[5693]: W1031 14:05:30.126175 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b3398fa_0028_4349_abd3_99d1cfe8247c.slice/crio-0c1db6aa442adb3125ff6d7fd2f9f74546ec06f4dc3a89d40fb54249dd112f11 WatchSource:0}: Error finding container 0c1db6aa442adb3125ff6d7fd2f9f74546ec06f4dc3a89d40fb54249dd112f11: Status 404 returned error can't find the container with id 0c1db6aa442adb3125ff6d7fd2f9f74546ec06f4dc3a89d40fb54249dd112f11 Oct 31 14:05:30.432267 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:30.432227 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f2c4d4-852rr" event={"ID":"3b3398fa-0028-4349-abd3-99d1cfe8247c","Type":"ContainerStarted","Data":"34dd2944ebc099df349325409c751ba471e167204763be1d7ff69e7e15fd74e3"} Oct 31 14:05:30.432714 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:30.432295 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f2c4d4-852rr" event={"ID":"3b3398fa-0028-4349-abd3-99d1cfe8247c","Type":"ContainerStarted","Data":"0c1db6aa442adb3125ff6d7fd2f9f74546ec06f4dc3a89d40fb54249dd112f11"} Oct 31 14:05:30.450579 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:30.450532 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f2c4d4-852rr" podStartSLOduration=1.450519739 podStartE2EDuration="1.450519739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:05:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:05:30.4491101 +0000 UTC m=+5581.156696396" watchObservedRunningTime="2026-10-31 14:05:30.450519739 +0000 UTC m=+5581.158106084" Oct 31 14:05:31.406246 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:31.406216 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="796a4469-22c6-4d99-b8e4-34f417c8661d" path="/var/lib/kubelet/pods/796a4469-22c6-4d99-b8e4-34f417c8661d/volumes" Oct 31 14:05:31.438998 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:31.438967 5693 generic.go:356] "Generic (PLEG): container finished" podID="3b3398fa-0028-4349-abd3-99d1cfe8247c" containerID="34dd2944ebc099df349325409c751ba471e167204763be1d7ff69e7e15fd74e3" exitCode=0 Oct 31 14:05:31.439360 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:31.439043 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f2c4d4-852rr" event={"ID":"3b3398fa-0028-4349-abd3-99d1cfe8247c","Type":"ContainerDied","Data":"34dd2944ebc099df349325409c751ba471e167204763be1d7ff69e7e15fd74e3"} Oct 31 14:05:32.580984 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:32.580960 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f2c4d4-852rr" Oct 31 14:05:32.728237 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:32.728155 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b3398fa-0028-4349-abd3-99d1cfe8247c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b3398fa-0028-4349-abd3-99d1cfe8247c-maas-api-openshift-ca\") pod \"3b3398fa-0028-4349-abd3-99d1cfe8247c\" (UID: \"3b3398fa-0028-4349-abd3-99d1cfe8247c\") " Oct 31 14:05:32.728237 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:32.728226 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b3398fa-0028-4349-abd3-99d1cfe8247c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b3398fa-0028-4349-abd3-99d1cfe8247c-maas-api-xks-ca\") pod \"3b3398fa-0028-4349-abd3-99d1cfe8247c\" (UID: \"3b3398fa-0028-4349-abd3-99d1cfe8247c\") " Oct 31 14:05:32.728499 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:32.728312 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b3398fa-0028-4349-abd3-99d1cfe8247c-kube-api-access-6vvd7\" (UniqueName: \"kubernetes.io/projected/3b3398fa-0028-4349-abd3-99d1cfe8247c-kube-api-access-6vvd7\") pod \"3b3398fa-0028-4349-abd3-99d1cfe8247c\" (UID: \"3b3398fa-0028-4349-abd3-99d1cfe8247c\") " Oct 31 14:05:32.728574 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:32.728548 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b3398fa-0028-4349-abd3-99d1cfe8247c-maas-api-openshift-ca" pod "3b3398fa-0028-4349-abd3-99d1cfe8247c" (UID: "3b3398fa-0028-4349-abd3-99d1cfe8247c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:05:32.728785 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:32.728760 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b3398fa-0028-4349-abd3-99d1cfe8247c-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:05:32.730549 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:32.730527 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b3398fa-0028-4349-abd3-99d1cfe8247c-maas-api-xks-ca" pod "3b3398fa-0028-4349-abd3-99d1cfe8247c" (UID: "3b3398fa-0028-4349-abd3-99d1cfe8247c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:05:32.730688 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:32.730572 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b3398fa-0028-4349-abd3-99d1cfe8247c-kube-api-access-6vvd7" pod "3b3398fa-0028-4349-abd3-99d1cfe8247c" (UID: "3b3398fa-0028-4349-abd3-99d1cfe8247c"). InnerVolumeSpecName "kube-api-access-6vvd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:05:32.830330 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:32.830301 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b3398fa-0028-4349-abd3-99d1cfe8247c-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:05:32.830330 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:32.830329 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vvd7\" (UniqueName: \"kubernetes.io/projected/3b3398fa-0028-4349-abd3-99d1cfe8247c-kube-api-access-6vvd7\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:05:33.450533 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:33.450503 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f2c4d4-852rr" Oct 31 14:05:33.450533 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:33.450523 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f2c4d4-852rr" event={"ID":"3b3398fa-0028-4349-abd3-99d1cfe8247c","Type":"ContainerDied","Data":"0c1db6aa442adb3125ff6d7fd2f9f74546ec06f4dc3a89d40fb54249dd112f11"} Oct 31 14:05:33.450822 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:33.450570 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c1db6aa442adb3125ff6d7fd2f9f74546ec06f4dc3a89d40fb54249dd112f11" Oct 31 14:05:39.051723 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:39.051625 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9dcc22e-hcz8n"] Oct 31 14:05:39.052073 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:39.052043 5693 cpu_manager.go:414] "containerMap: removing container" podUID="3b3398fa-0028-4349-abd3-99d1cfe8247c" containerName="revoke-keys" Oct 31 14:05:39.052073 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:39.052058 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b3398fa-0028-4349-abd3-99d1cfe8247c" containerName="revoke-keys" Oct 31 14:05:39.052140 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:39.052135 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b3398fa-0028-4349-abd3-99d1cfe8247c" containerName="revoke-keys" Oct 31 14:05:39.056920 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:39.056897 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9dcc22e-hcz8n" Oct 31 14:05:39.067212 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:39.067153 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:05:39.067212 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:39.067155 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 14:05:39.069302 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:39.069277 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9dcc22e-hcz8n"] Oct 31 14:05:39.197428 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:39.197396 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/909bf5d0-6f7f-487d-9a4a-b3db681d664b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9dcc22e-hcz8n\" (UID: \"909bf5d0-6f7f-487d-9a4a-b3db681d664b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9dcc22e-hcz8n" Oct 31 14:05:39.197623 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:39.197570 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzjwb\" (UniqueName: \"kubernetes.io/projected/909bf5d0-6f7f-487d-9a4a-b3db681d664b-kube-api-access-nzjwb\") pod \"maas-api-revoke-sub-f9dcc22e-hcz8n\" (UID: \"909bf5d0-6f7f-487d-9a4a-b3db681d664b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9dcc22e-hcz8n" Oct 31 14:05:39.197623 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:39.197606 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/909bf5d0-6f7f-487d-9a4a-b3db681d664b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9dcc22e-hcz8n\" (UID: \"909bf5d0-6f7f-487d-9a4a-b3db681d664b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9dcc22e-hcz8n" Oct 31 14:05:39.298816 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:39.298780 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzjwb\" (UniqueName: \"kubernetes.io/projected/909bf5d0-6f7f-487d-9a4a-b3db681d664b-kube-api-access-nzjwb\") pod \"maas-api-revoke-sub-f9dcc22e-hcz8n\" (UID: \"909bf5d0-6f7f-487d-9a4a-b3db681d664b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9dcc22e-hcz8n" Oct 31 14:05:39.298982 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:39.298823 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/909bf5d0-6f7f-487d-9a4a-b3db681d664b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9dcc22e-hcz8n\" (UID: \"909bf5d0-6f7f-487d-9a4a-b3db681d664b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9dcc22e-hcz8n" Oct 31 14:05:39.298982 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:39.298847 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/909bf5d0-6f7f-487d-9a4a-b3db681d664b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9dcc22e-hcz8n\" (UID: \"909bf5d0-6f7f-487d-9a4a-b3db681d664b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9dcc22e-hcz8n" Oct 31 14:05:39.299464 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:39.299444 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/909bf5d0-6f7f-487d-9a4a-b3db681d664b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9dcc22e-hcz8n\" (UID: \"909bf5d0-6f7f-487d-9a4a-b3db681d664b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9dcc22e-hcz8n" Oct 31 14:05:39.301352 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:39.301321 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/909bf5d0-6f7f-487d-9a4a-b3db681d664b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9dcc22e-hcz8n\" (UID: \"909bf5d0-6f7f-487d-9a4a-b3db681d664b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9dcc22e-hcz8n" Oct 31 14:05:39.320051 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:39.319996 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzjwb\" (UniqueName: \"kubernetes.io/projected/909bf5d0-6f7f-487d-9a4a-b3db681d664b-kube-api-access-nzjwb\") pod \"maas-api-revoke-sub-f9dcc22e-hcz8n\" (UID: \"909bf5d0-6f7f-487d-9a4a-b3db681d664b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9dcc22e-hcz8n" Oct 31 14:05:39.374517 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:39.374484 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9dcc22e-hcz8n" Oct 31 14:05:39.649203 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:39.649087 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9dcc22e-hcz8n"] Oct 31 14:05:39.652090 ip-10-0-131-157 kubenswrapper[5693]: W1031 14:05:39.652057 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod909bf5d0_6f7f_487d_9a4a_b3db681d664b.slice/crio-433429d6ee750fd7318ecb1638fdbdd22954b40bfa68a2001bbedf39a956cd15 WatchSource:0}: Error finding container 433429d6ee750fd7318ecb1638fdbdd22954b40bfa68a2001bbedf39a956cd15: Status 404 returned error can't find the container with id 433429d6ee750fd7318ecb1638fdbdd22954b40bfa68a2001bbedf39a956cd15 Oct 31 14:05:40.483009 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:40.482968 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9dcc22e-hcz8n" event={"ID":"909bf5d0-6f7f-487d-9a4a-b3db681d664b","Type":"ContainerStarted","Data":"95bfdc49eed21bc57ade63550496c251dd648baf70ecdff4f538cd5668262a6d"} Oct 31 14:05:40.483421 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:40.483063 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9dcc22e-hcz8n" event={"ID":"909bf5d0-6f7f-487d-9a4a-b3db681d664b","Type":"ContainerStarted","Data":"433429d6ee750fd7318ecb1638fdbdd22954b40bfa68a2001bbedf39a956cd15"} Oct 31 14:05:40.501772 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:40.501728 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9dcc22e-hcz8n" podStartSLOduration=1.501716965 podStartE2EDuration="1.501716965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:05:40.500524743 +0000 UTC m=+5591.208111039" watchObservedRunningTime="2026-10-31 14:05:40.501716965 +0000 UTC m=+5591.209303261" Oct 31 14:05:41.488440 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:41.488407 5693 generic.go:356] "Generic (PLEG): container finished" podID="909bf5d0-6f7f-487d-9a4a-b3db681d664b" containerID="95bfdc49eed21bc57ade63550496c251dd648baf70ecdff4f538cd5668262a6d" exitCode=0 Oct 31 14:05:41.488864 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:41.488464 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9dcc22e-hcz8n" event={"ID":"909bf5d0-6f7f-487d-9a4a-b3db681d664b","Type":"ContainerDied","Data":"95bfdc49eed21bc57ade63550496c251dd648baf70ecdff4f538cd5668262a6d"} Oct 31 14:05:42.629129 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:42.629104 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9dcc22e-hcz8n" Oct 31 14:05:42.632166 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:42.632149 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/909bf5d0-6f7f-487d-9a4a-b3db681d664b-kube-api-access-nzjwb\" (UniqueName: \"kubernetes.io/projected/909bf5d0-6f7f-487d-9a4a-b3db681d664b-kube-api-access-nzjwb\") pod \"909bf5d0-6f7f-487d-9a4a-b3db681d664b\" (UID: \"909bf5d0-6f7f-487d-9a4a-b3db681d664b\") " Oct 31 14:05:42.632263 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:42.632217 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/909bf5d0-6f7f-487d-9a4a-b3db681d664b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/909bf5d0-6f7f-487d-9a4a-b3db681d664b-maas-api-xks-ca\") pod \"909bf5d0-6f7f-487d-9a4a-b3db681d664b\" (UID: \"909bf5d0-6f7f-487d-9a4a-b3db681d664b\") " Oct 31 14:05:42.632327 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:42.632282 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/909bf5d0-6f7f-487d-9a4a-b3db681d664b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/909bf5d0-6f7f-487d-9a4a-b3db681d664b-maas-api-openshift-ca\") pod \"909bf5d0-6f7f-487d-9a4a-b3db681d664b\" (UID: \"909bf5d0-6f7f-487d-9a4a-b3db681d664b\") " Oct 31 14:05:42.632668 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:42.632619 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/909bf5d0-6f7f-487d-9a4a-b3db681d664b-maas-api-openshift-ca" pod "909bf5d0-6f7f-487d-9a4a-b3db681d664b" (UID: "909bf5d0-6f7f-487d-9a4a-b3db681d664b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:05:42.634520 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:42.634499 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/909bf5d0-6f7f-487d-9a4a-b3db681d664b-maas-api-xks-ca" pod "909bf5d0-6f7f-487d-9a4a-b3db681d664b" (UID: "909bf5d0-6f7f-487d-9a4a-b3db681d664b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:05:42.634614 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:42.634594 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/909bf5d0-6f7f-487d-9a4a-b3db681d664b-kube-api-access-nzjwb" pod "909bf5d0-6f7f-487d-9a4a-b3db681d664b" (UID: "909bf5d0-6f7f-487d-9a4a-b3db681d664b"). InnerVolumeSpecName "kube-api-access-nzjwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:05:42.733944 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:42.733906 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/909bf5d0-6f7f-487d-9a4a-b3db681d664b-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:05:42.733944 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:42.733938 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzjwb\" (UniqueName: \"kubernetes.io/projected/909bf5d0-6f7f-487d-9a4a-b3db681d664b-kube-api-access-nzjwb\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:05:42.733944 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:42.733948 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/909bf5d0-6f7f-487d-9a4a-b3db681d664b-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:05:43.497758 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:43.497726 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9dcc22e-hcz8n" Oct 31 14:05:43.497903 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:43.497759 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9dcc22e-hcz8n" event={"ID":"909bf5d0-6f7f-487d-9a4a-b3db681d664b","Type":"ContainerDied","Data":"433429d6ee750fd7318ecb1638fdbdd22954b40bfa68a2001bbedf39a956cd15"} Oct 31 14:05:43.497903 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:43.497804 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="433429d6ee750fd7318ecb1638fdbdd22954b40bfa68a2001bbedf39a956cd15" Oct 31 14:05:48.686159 ip-10-0-131-157 kubenswrapper[5693]: E1031 14:05:48.686121 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 14:05:59.037988 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:59.037957 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11881524-dvn9m"] Oct 31 14:05:59.038625 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:59.038536 5693 cpu_manager.go:414] "containerMap: removing container" podUID="909bf5d0-6f7f-487d-9a4a-b3db681d664b" containerName="revoke-keys" Oct 31 14:05:59.038625 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:59.038556 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="909bf5d0-6f7f-487d-9a4a-b3db681d664b" containerName="revoke-keys" Oct 31 14:05:59.038802 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:59.038694 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="909bf5d0-6f7f-487d-9a4a-b3db681d664b" containerName="revoke-keys" Oct 31 14:05:59.041826 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:59.041802 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11881524-dvn9m" Oct 31 14:05:59.045493 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:59.045468 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:05:59.046469 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:59.046443 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 14:05:59.051459 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:59.051436 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11881524-dvn9m"] Oct 31 14:05:59.094269 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:59.094236 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhxls\" (UniqueName: \"kubernetes.io/projected/208b7c2f-0a32-4cfc-8100-26764ae45e26-kube-api-access-lhxls\") pod \"maas-api-revoke-sub-11881524-dvn9m\" (UID: \"208b7c2f-0a32-4cfc-8100-26764ae45e26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11881524-dvn9m" Oct 31 14:05:59.094438 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:59.094365 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208b7c2f-0a32-4cfc-8100-26764ae45e26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11881524-dvn9m\" (UID: \"208b7c2f-0a32-4cfc-8100-26764ae45e26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11881524-dvn9m" Oct 31 14:05:59.094438 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:59.094390 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208b7c2f-0a32-4cfc-8100-26764ae45e26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11881524-dvn9m\" (UID: \"208b7c2f-0a32-4cfc-8100-26764ae45e26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11881524-dvn9m" Oct 31 14:05:59.195746 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:59.195706 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208b7c2f-0a32-4cfc-8100-26764ae45e26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11881524-dvn9m\" (UID: \"208b7c2f-0a32-4cfc-8100-26764ae45e26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11881524-dvn9m" Oct 31 14:05:59.195746 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:59.195748 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208b7c2f-0a32-4cfc-8100-26764ae45e26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11881524-dvn9m\" (UID: \"208b7c2f-0a32-4cfc-8100-26764ae45e26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11881524-dvn9m" Oct 31 14:05:59.195966 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:59.195804 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhxls\" (UniqueName: \"kubernetes.io/projected/208b7c2f-0a32-4cfc-8100-26764ae45e26-kube-api-access-lhxls\") pod \"maas-api-revoke-sub-11881524-dvn9m\" (UID: \"208b7c2f-0a32-4cfc-8100-26764ae45e26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11881524-dvn9m" Oct 31 14:05:59.196511 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:59.196487 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208b7c2f-0a32-4cfc-8100-26764ae45e26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11881524-dvn9m\" (UID: \"208b7c2f-0a32-4cfc-8100-26764ae45e26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11881524-dvn9m" Oct 31 14:05:59.198360 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:59.198338 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208b7c2f-0a32-4cfc-8100-26764ae45e26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11881524-dvn9m\" (UID: \"208b7c2f-0a32-4cfc-8100-26764ae45e26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11881524-dvn9m" Oct 31 14:05:59.208370 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:59.208346 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhxls\" (UniqueName: \"kubernetes.io/projected/208b7c2f-0a32-4cfc-8100-26764ae45e26-kube-api-access-lhxls\") pod \"maas-api-revoke-sub-11881524-dvn9m\" (UID: \"208b7c2f-0a32-4cfc-8100-26764ae45e26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11881524-dvn9m" Oct 31 14:05:59.358362 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:59.358275 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11881524-dvn9m" Oct 31 14:05:59.635039 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:05:59.634995 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11881524-dvn9m"] Oct 31 14:05:59.637859 ip-10-0-131-157 kubenswrapper[5693]: W1031 14:05:59.637830 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod208b7c2f_0a32_4cfc_8100_26764ae45e26.slice/crio-12cbf8f45c6f0e7424128d75a73297b7ec10a1f19fd33c862d9a33d852401b78 WatchSource:0}: Error finding container 12cbf8f45c6f0e7424128d75a73297b7ec10a1f19fd33c862d9a33d852401b78: Status 404 returned error can't find the container with id 12cbf8f45c6f0e7424128d75a73297b7ec10a1f19fd33c862d9a33d852401b78 Oct 31 14:06:00.596115 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:00.596078 5693 generic.go:356] "Generic (PLEG): container finished" podID="208b7c2f-0a32-4cfc-8100-26764ae45e26" containerID="ba647d678a14059f78cfb89f135875b18404316c3c1b2fdbc51dd34f855f39fd" exitCode=0 Oct 31 14:06:00.596519 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:00.596174 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11881524-dvn9m" event={"ID":"208b7c2f-0a32-4cfc-8100-26764ae45e26","Type":"ContainerDied","Data":"ba647d678a14059f78cfb89f135875b18404316c3c1b2fdbc51dd34f855f39fd"} Oct 31 14:06:00.596519 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:00.596232 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11881524-dvn9m" event={"ID":"208b7c2f-0a32-4cfc-8100-26764ae45e26","Type":"ContainerStarted","Data":"12cbf8f45c6f0e7424128d75a73297b7ec10a1f19fd33c862d9a33d852401b78"} Oct 31 14:06:01.738551 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:01.738522 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11881524-dvn9m" Oct 31 14:06:01.824571 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:01.824531 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/208b7c2f-0a32-4cfc-8100-26764ae45e26-kube-api-access-lhxls\" (UniqueName: \"kubernetes.io/projected/208b7c2f-0a32-4cfc-8100-26764ae45e26-kube-api-access-lhxls\") pod \"208b7c2f-0a32-4cfc-8100-26764ae45e26\" (UID: \"208b7c2f-0a32-4cfc-8100-26764ae45e26\") " Oct 31 14:06:01.824813 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:01.824626 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/208b7c2f-0a32-4cfc-8100-26764ae45e26-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208b7c2f-0a32-4cfc-8100-26764ae45e26-maas-api-openshift-ca\") pod \"208b7c2f-0a32-4cfc-8100-26764ae45e26\" (UID: \"208b7c2f-0a32-4cfc-8100-26764ae45e26\") " Oct 31 14:06:01.824813 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:01.824736 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/208b7c2f-0a32-4cfc-8100-26764ae45e26-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208b7c2f-0a32-4cfc-8100-26764ae45e26-maas-api-xks-ca\") pod \"208b7c2f-0a32-4cfc-8100-26764ae45e26\" (UID: \"208b7c2f-0a32-4cfc-8100-26764ae45e26\") " Oct 31 14:06:01.825047 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:01.825021 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/208b7c2f-0a32-4cfc-8100-26764ae45e26-maas-api-openshift-ca" pod "208b7c2f-0a32-4cfc-8100-26764ae45e26" (UID: "208b7c2f-0a32-4cfc-8100-26764ae45e26"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:06:01.826851 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:01.826825 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/208b7c2f-0a32-4cfc-8100-26764ae45e26-maas-api-xks-ca" pod "208b7c2f-0a32-4cfc-8100-26764ae45e26" (UID: "208b7c2f-0a32-4cfc-8100-26764ae45e26"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:06:01.826973 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:01.826924 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/208b7c2f-0a32-4cfc-8100-26764ae45e26-kube-api-access-lhxls" pod "208b7c2f-0a32-4cfc-8100-26764ae45e26" (UID: "208b7c2f-0a32-4cfc-8100-26764ae45e26"). InnerVolumeSpecName "kube-api-access-lhxls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:06:01.925680 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:01.925623 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208b7c2f-0a32-4cfc-8100-26764ae45e26-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:06:01.925680 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:01.925675 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208b7c2f-0a32-4cfc-8100-26764ae45e26-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:06:01.925680 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:01.925686 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhxls\" (UniqueName: \"kubernetes.io/projected/208b7c2f-0a32-4cfc-8100-26764ae45e26-kube-api-access-lhxls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:06:02.606107 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:02.606075 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11881524-dvn9m" Oct 31 14:06:02.606107 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:02.606103 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11881524-dvn9m" event={"ID":"208b7c2f-0a32-4cfc-8100-26764ae45e26","Type":"ContainerDied","Data":"12cbf8f45c6f0e7424128d75a73297b7ec10a1f19fd33c862d9a33d852401b78"} Oct 31 14:06:02.606337 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:02.606150 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12cbf8f45c6f0e7424128d75a73297b7ec10a1f19fd33c862d9a33d852401b78" Oct 31 14:06:18.686470 ip-10-0-131-157 kubenswrapper[5693]: E1031 14:06:18.686429 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 14:06:25.247974 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:25.247932 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abecc70c-k6hx5"] Oct 31 14:06:25.248566 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:25.248546 5693 cpu_manager.go:414] "containerMap: removing container" podUID="208b7c2f-0a32-4cfc-8100-26764ae45e26" containerName="revoke-keys" Oct 31 14:06:25.248620 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:25.248571 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="208b7c2f-0a32-4cfc-8100-26764ae45e26" containerName="revoke-keys" Oct 31 14:06:25.248757 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:25.248739 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="208b7c2f-0a32-4cfc-8100-26764ae45e26" containerName="revoke-keys" Oct 31 14:06:25.252399 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:25.252380 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abecc70c-k6hx5" Oct 31 14:06:25.256767 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:25.256724 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:06:25.257682 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:25.257633 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 14:06:25.264499 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:25.264106 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abecc70c-k6hx5"] Oct 31 14:06:25.405505 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:25.405477 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c05c1144-fffc-47f3-8b46-9b708b7d3c5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abecc70c-k6hx5\" (UID: \"c05c1144-fffc-47f3-8b46-9b708b7d3c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abecc70c-k6hx5" Oct 31 14:06:25.405697 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:25.405530 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c05c1144-fffc-47f3-8b46-9b708b7d3c5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abecc70c-k6hx5\" (UID: \"c05c1144-fffc-47f3-8b46-9b708b7d3c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abecc70c-k6hx5" Oct 31 14:06:25.405761 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:25.405738 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8l2r\" (UniqueName: \"kubernetes.io/projected/c05c1144-fffc-47f3-8b46-9b708b7d3c5d-kube-api-access-g8l2r\") pod \"maas-api-revoke-sub-abecc70c-k6hx5\" (UID: \"c05c1144-fffc-47f3-8b46-9b708b7d3c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abecc70c-k6hx5" Oct 31 14:06:25.507060 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:25.506981 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8l2r\" (UniqueName: \"kubernetes.io/projected/c05c1144-fffc-47f3-8b46-9b708b7d3c5d-kube-api-access-g8l2r\") pod \"maas-api-revoke-sub-abecc70c-k6hx5\" (UID: \"c05c1144-fffc-47f3-8b46-9b708b7d3c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abecc70c-k6hx5" Oct 31 14:06:25.507060 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:25.507032 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c05c1144-fffc-47f3-8b46-9b708b7d3c5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abecc70c-k6hx5\" (UID: \"c05c1144-fffc-47f3-8b46-9b708b7d3c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abecc70c-k6hx5" Oct 31 14:06:25.507279 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:25.507080 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c05c1144-fffc-47f3-8b46-9b708b7d3c5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abecc70c-k6hx5\" (UID: \"c05c1144-fffc-47f3-8b46-9b708b7d3c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abecc70c-k6hx5" Oct 31 14:06:25.507831 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:25.507801 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c05c1144-fffc-47f3-8b46-9b708b7d3c5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abecc70c-k6hx5\" (UID: \"c05c1144-fffc-47f3-8b46-9b708b7d3c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abecc70c-k6hx5" Oct 31 14:06:25.509654 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:25.509617 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c05c1144-fffc-47f3-8b46-9b708b7d3c5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abecc70c-k6hx5\" (UID: \"c05c1144-fffc-47f3-8b46-9b708b7d3c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abecc70c-k6hx5" Oct 31 14:06:25.519346 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:25.519325 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8l2r\" (UniqueName: \"kubernetes.io/projected/c05c1144-fffc-47f3-8b46-9b708b7d3c5d-kube-api-access-g8l2r\") pod \"maas-api-revoke-sub-abecc70c-k6hx5\" (UID: \"c05c1144-fffc-47f3-8b46-9b708b7d3c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abecc70c-k6hx5" Oct 31 14:06:25.571322 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:25.571293 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abecc70c-k6hx5" Oct 31 14:06:25.845970 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:25.845945 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abecc70c-k6hx5"] Oct 31 14:06:25.848572 ip-10-0-131-157 kubenswrapper[5693]: W1031 14:06:25.848543 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc05c1144_fffc_47f3_8b46_9b708b7d3c5d.slice/crio-19a1895c349f03b98b73ef1e8b615547cadf3d9121de90a0a3785c33cf004823 WatchSource:0}: Error finding container 19a1895c349f03b98b73ef1e8b615547cadf3d9121de90a0a3785c33cf004823: Status 404 returned error can't find the container with id 19a1895c349f03b98b73ef1e8b615547cadf3d9121de90a0a3785c33cf004823 Oct 31 14:06:26.729580 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:26.729541 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abecc70c-k6hx5" event={"ID":"c05c1144-fffc-47f3-8b46-9b708b7d3c5d","Type":"ContainerStarted","Data":"927629ad44e5e7bdf87ade667563e54f22373477c712fe0a78ff87c2751af2f8"} Oct 31 14:06:26.730050 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:26.729594 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abecc70c-k6hx5" event={"ID":"c05c1144-fffc-47f3-8b46-9b708b7d3c5d","Type":"ContainerStarted","Data":"19a1895c349f03b98b73ef1e8b615547cadf3d9121de90a0a3785c33cf004823"} Oct 31 14:06:26.749531 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:26.749455 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abecc70c-k6hx5" podStartSLOduration=1.749443915 podStartE2EDuration="1.749443915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:06:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:06:26.747233467 +0000 UTC m=+5637.454819764" watchObservedRunningTime="2026-10-31 14:06:26.749443915 +0000 UTC m=+5637.457030209" Oct 31 14:06:27.743321 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:27.743281 5693 generic.go:356] "Generic (PLEG): container finished" podID="c05c1144-fffc-47f3-8b46-9b708b7d3c5d" containerID="927629ad44e5e7bdf87ade667563e54f22373477c712fe0a78ff87c2751af2f8" exitCode=0 Oct 31 14:06:27.743790 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:27.743369 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abecc70c-k6hx5" event={"ID":"c05c1144-fffc-47f3-8b46-9b708b7d3c5d","Type":"ContainerDied","Data":"927629ad44e5e7bdf87ade667563e54f22373477c712fe0a78ff87c2751af2f8"} Oct 31 14:06:28.882204 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:28.882171 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abecc70c-k6hx5" Oct 31 14:06:29.041338 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:29.041249 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c05c1144-fffc-47f3-8b46-9b708b7d3c5d-kube-api-access-g8l2r\" (UniqueName: \"kubernetes.io/projected/c05c1144-fffc-47f3-8b46-9b708b7d3c5d-kube-api-access-g8l2r\") pod \"c05c1144-fffc-47f3-8b46-9b708b7d3c5d\" (UID: \"c05c1144-fffc-47f3-8b46-9b708b7d3c5d\") " Oct 31 14:06:29.041338 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:29.041288 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c05c1144-fffc-47f3-8b46-9b708b7d3c5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c05c1144-fffc-47f3-8b46-9b708b7d3c5d-maas-api-openshift-ca\") pod \"c05c1144-fffc-47f3-8b46-9b708b7d3c5d\" (UID: \"c05c1144-fffc-47f3-8b46-9b708b7d3c5d\") " Oct 31 14:06:29.041636 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:29.041421 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c05c1144-fffc-47f3-8b46-9b708b7d3c5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c05c1144-fffc-47f3-8b46-9b708b7d3c5d-maas-api-xks-ca\") pod \"c05c1144-fffc-47f3-8b46-9b708b7d3c5d\" (UID: \"c05c1144-fffc-47f3-8b46-9b708b7d3c5d\") " Oct 31 14:06:29.041761 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:29.041742 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c05c1144-fffc-47f3-8b46-9b708b7d3c5d-maas-api-openshift-ca" pod "c05c1144-fffc-47f3-8b46-9b708b7d3c5d" (UID: "c05c1144-fffc-47f3-8b46-9b708b7d3c5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:06:29.043516 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:29.043477 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c05c1144-fffc-47f3-8b46-9b708b7d3c5d-maas-api-xks-ca" pod "c05c1144-fffc-47f3-8b46-9b708b7d3c5d" (UID: "c05c1144-fffc-47f3-8b46-9b708b7d3c5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:06:29.043623 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:29.043608 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c05c1144-fffc-47f3-8b46-9b708b7d3c5d-kube-api-access-g8l2r" pod "c05c1144-fffc-47f3-8b46-9b708b7d3c5d" (UID: "c05c1144-fffc-47f3-8b46-9b708b7d3c5d"). InnerVolumeSpecName "kube-api-access-g8l2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:06:29.142863 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:29.142830 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c05c1144-fffc-47f3-8b46-9b708b7d3c5d-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:06:29.142863 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:29.142860 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8l2r\" (UniqueName: \"kubernetes.io/projected/c05c1144-fffc-47f3-8b46-9b708b7d3c5d-kube-api-access-g8l2r\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:06:29.142863 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:29.142871 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c05c1144-fffc-47f3-8b46-9b708b7d3c5d-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:06:29.754922 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:29.754839 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abecc70c-k6hx5" Oct 31 14:06:29.754922 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:29.754870 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abecc70c-k6hx5" event={"ID":"c05c1144-fffc-47f3-8b46-9b708b7d3c5d","Type":"ContainerDied","Data":"19a1895c349f03b98b73ef1e8b615547cadf3d9121de90a0a3785c33cf004823"} Oct 31 14:06:29.754922 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:29.754918 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19a1895c349f03b98b73ef1e8b615547cadf3d9121de90a0a3785c33cf004823" Oct 31 14:06:35.471983 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:35.471949 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b74cb0f-4p8v5"] Oct 31 14:06:35.472504 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:35.472487 5693 cpu_manager.go:414] "containerMap: removing container" podUID="c05c1144-fffc-47f3-8b46-9b708b7d3c5d" containerName="revoke-keys" Oct 31 14:06:35.472560 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:35.472509 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c05c1144-fffc-47f3-8b46-9b708b7d3c5d" containerName="revoke-keys" Oct 31 14:06:35.472652 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:35.472629 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="c05c1144-fffc-47f3-8b46-9b708b7d3c5d" containerName="revoke-keys" Oct 31 14:06:35.481671 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:35.477335 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b74cb0f-4p8v5" Oct 31 14:06:35.485532 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:35.485475 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b74cb0f-4p8v5"] Oct 31 14:06:35.486055 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:35.485980 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 14:06:35.486055 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:35.486021 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:06:35.503972 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:35.503939 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be4e85f9-a566-4083-b068-5a9ce3222bc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b74cb0f-4p8v5\" (UID: \"be4e85f9-a566-4083-b068-5a9ce3222bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b74cb0f-4p8v5" Oct 31 14:06:35.504144 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:35.504068 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4682x\" (UniqueName: \"kubernetes.io/projected/be4e85f9-a566-4083-b068-5a9ce3222bc6-kube-api-access-4682x\") pod \"maas-api-revoke-sub-5b74cb0f-4p8v5\" (UID: \"be4e85f9-a566-4083-b068-5a9ce3222bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b74cb0f-4p8v5" Oct 31 14:06:35.504211 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:35.504185 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be4e85f9-a566-4083-b068-5a9ce3222bc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b74cb0f-4p8v5\" (UID: \"be4e85f9-a566-4083-b068-5a9ce3222bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b74cb0f-4p8v5" Oct 31 14:06:35.604756 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:35.604723 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be4e85f9-a566-4083-b068-5a9ce3222bc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b74cb0f-4p8v5\" (UID: \"be4e85f9-a566-4083-b068-5a9ce3222bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b74cb0f-4p8v5" Oct 31 14:06:35.604920 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:35.604792 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be4e85f9-a566-4083-b068-5a9ce3222bc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b74cb0f-4p8v5\" (UID: \"be4e85f9-a566-4083-b068-5a9ce3222bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b74cb0f-4p8v5" Oct 31 14:06:35.604920 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:35.604838 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4682x\" (UniqueName: \"kubernetes.io/projected/be4e85f9-a566-4083-b068-5a9ce3222bc6-kube-api-access-4682x\") pod \"maas-api-revoke-sub-5b74cb0f-4p8v5\" (UID: \"be4e85f9-a566-4083-b068-5a9ce3222bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b74cb0f-4p8v5" Oct 31 14:06:35.605520 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:35.605499 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be4e85f9-a566-4083-b068-5a9ce3222bc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b74cb0f-4p8v5\" (UID: \"be4e85f9-a566-4083-b068-5a9ce3222bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b74cb0f-4p8v5" Oct 31 14:06:35.607279 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:35.607251 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be4e85f9-a566-4083-b068-5a9ce3222bc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b74cb0f-4p8v5\" (UID: \"be4e85f9-a566-4083-b068-5a9ce3222bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b74cb0f-4p8v5" Oct 31 14:06:35.617175 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:35.617148 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4682x\" (UniqueName: \"kubernetes.io/projected/be4e85f9-a566-4083-b068-5a9ce3222bc6-kube-api-access-4682x\") pod \"maas-api-revoke-sub-5b74cb0f-4p8v5\" (UID: \"be4e85f9-a566-4083-b068-5a9ce3222bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b74cb0f-4p8v5" Oct 31 14:06:35.804021 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:35.803942 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b74cb0f-4p8v5" Oct 31 14:06:36.077664 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:36.077484 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b74cb0f-4p8v5"] Oct 31 14:06:36.080566 ip-10-0-131-157 kubenswrapper[5693]: W1031 14:06:36.080536 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe4e85f9_a566_4083_b068_5a9ce3222bc6.slice/crio-83de1f0ab266230dbe79cff66d94422c2503aeb94345dba0520b58a387933981 WatchSource:0}: Error finding container 83de1f0ab266230dbe79cff66d94422c2503aeb94345dba0520b58a387933981: Status 404 returned error can't find the container with id 83de1f0ab266230dbe79cff66d94422c2503aeb94345dba0520b58a387933981 Oct 31 14:06:36.791390 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:36.791357 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b74cb0f-4p8v5" event={"ID":"be4e85f9-a566-4083-b068-5a9ce3222bc6","Type":"ContainerStarted","Data":"316759f9f44b37b2c1405786415a0ad9d6ac6df3d61e64be7c55602b3536abb4"} Oct 31 14:06:36.791839 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:36.791415 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b74cb0f-4p8v5" event={"ID":"be4e85f9-a566-4083-b068-5a9ce3222bc6","Type":"ContainerStarted","Data":"83de1f0ab266230dbe79cff66d94422c2503aeb94345dba0520b58a387933981"} Oct 31 14:06:36.809265 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:36.809206 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b74cb0f-4p8v5" podStartSLOduration=1.809188461 podStartE2EDuration="1.809188461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:06:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:06:36.808362659 +0000 UTC m=+5647.515948953" watchObservedRunningTime="2026-10-31 14:06:36.809188461 +0000 UTC m=+5647.516774757" Oct 31 14:06:37.797625 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:37.797592 5693 generic.go:356] "Generic (PLEG): container finished" podID="be4e85f9-a566-4083-b068-5a9ce3222bc6" containerID="316759f9f44b37b2c1405786415a0ad9d6ac6df3d61e64be7c55602b3536abb4" exitCode=0 Oct 31 14:06:37.798131 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:37.797673 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b74cb0f-4p8v5" event={"ID":"be4e85f9-a566-4083-b068-5a9ce3222bc6","Type":"ContainerDied","Data":"316759f9f44b37b2c1405786415a0ad9d6ac6df3d61e64be7c55602b3536abb4"} Oct 31 14:06:38.934969 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:38.934936 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b74cb0f-4p8v5" Oct 31 14:06:39.032401 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:39.032363 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be4e85f9-a566-4083-b068-5a9ce3222bc6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be4e85f9-a566-4083-b068-5a9ce3222bc6-maas-api-xks-ca\") pod \"be4e85f9-a566-4083-b068-5a9ce3222bc6\" (UID: \"be4e85f9-a566-4083-b068-5a9ce3222bc6\") " Oct 31 14:06:39.032608 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:39.032472 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be4e85f9-a566-4083-b068-5a9ce3222bc6-kube-api-access-4682x\" (UniqueName: \"kubernetes.io/projected/be4e85f9-a566-4083-b068-5a9ce3222bc6-kube-api-access-4682x\") pod \"be4e85f9-a566-4083-b068-5a9ce3222bc6\" (UID: \"be4e85f9-a566-4083-b068-5a9ce3222bc6\") " Oct 31 14:06:39.032608 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:39.032509 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be4e85f9-a566-4083-b068-5a9ce3222bc6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be4e85f9-a566-4083-b068-5a9ce3222bc6-maas-api-openshift-ca\") pod \"be4e85f9-a566-4083-b068-5a9ce3222bc6\" (UID: \"be4e85f9-a566-4083-b068-5a9ce3222bc6\") " Oct 31 14:06:39.033192 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:39.033102 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be4e85f9-a566-4083-b068-5a9ce3222bc6-maas-api-openshift-ca" pod "be4e85f9-a566-4083-b068-5a9ce3222bc6" (UID: "be4e85f9-a566-4083-b068-5a9ce3222bc6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:06:39.034663 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:39.034623 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be4e85f9-a566-4083-b068-5a9ce3222bc6-maas-api-xks-ca" pod "be4e85f9-a566-4083-b068-5a9ce3222bc6" (UID: "be4e85f9-a566-4083-b068-5a9ce3222bc6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:06:39.034790 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:39.034766 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be4e85f9-a566-4083-b068-5a9ce3222bc6-kube-api-access-4682x" pod "be4e85f9-a566-4083-b068-5a9ce3222bc6" (UID: "be4e85f9-a566-4083-b068-5a9ce3222bc6"). InnerVolumeSpecName "kube-api-access-4682x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:06:39.134103 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:39.134011 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4682x\" (UniqueName: \"kubernetes.io/projected/be4e85f9-a566-4083-b068-5a9ce3222bc6-kube-api-access-4682x\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:06:39.134103 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:39.134043 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be4e85f9-a566-4083-b068-5a9ce3222bc6-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:06:39.134103 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:39.134057 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be4e85f9-a566-4083-b068-5a9ce3222bc6-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:06:39.807958 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:39.807875 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b74cb0f-4p8v5" Oct 31 14:06:39.807958 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:39.807898 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b74cb0f-4p8v5" event={"ID":"be4e85f9-a566-4083-b068-5a9ce3222bc6","Type":"ContainerDied","Data":"83de1f0ab266230dbe79cff66d94422c2503aeb94345dba0520b58a387933981"} Oct 31 14:06:39.807958 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:06:39.807940 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83de1f0ab266230dbe79cff66d94422c2503aeb94345dba0520b58a387933981" Oct 31 14:06:48.686940 ip-10-0-131-157 kubenswrapper[5693]: E1031 14:06:48.686903 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 14:07:02.042625 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:02.042589 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36d45a9d-8ffwn"] Oct 31 14:07:02.043194 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:02.043172 5693 cpu_manager.go:414] "containerMap: removing container" podUID="be4e85f9-a566-4083-b068-5a9ce3222bc6" containerName="revoke-keys" Oct 31 14:07:02.043291 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:02.043199 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be4e85f9-a566-4083-b068-5a9ce3222bc6" containerName="revoke-keys" Oct 31 14:07:02.043354 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:02.043338 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="be4e85f9-a566-4083-b068-5a9ce3222bc6" containerName="revoke-keys" Oct 31 14:07:02.046458 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:02.046439 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d45a9d-8ffwn" Oct 31 14:07:02.049981 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:02.049958 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:07:02.050963 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:02.050946 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 14:07:02.056358 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:02.056323 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36d45a9d-8ffwn"] Oct 31 14:07:02.153919 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:02.153880 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnk8b\" (UniqueName: \"kubernetes.io/projected/70e9e98e-ef42-4867-bd8a-e89747807a13-kube-api-access-nnk8b\") pod \"maas-api-revoke-sub-36d45a9d-8ffwn\" (UID: \"70e9e98e-ef42-4867-bd8a-e89747807a13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d45a9d-8ffwn" Oct 31 14:07:02.154145 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:02.154036 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70e9e98e-ef42-4867-bd8a-e89747807a13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36d45a9d-8ffwn\" (UID: \"70e9e98e-ef42-4867-bd8a-e89747807a13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d45a9d-8ffwn" Oct 31 14:07:02.154145 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:02.154072 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70e9e98e-ef42-4867-bd8a-e89747807a13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36d45a9d-8ffwn\" (UID: \"70e9e98e-ef42-4867-bd8a-e89747807a13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d45a9d-8ffwn" Oct 31 14:07:02.255161 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:02.255126 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70e9e98e-ef42-4867-bd8a-e89747807a13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36d45a9d-8ffwn\" (UID: \"70e9e98e-ef42-4867-bd8a-e89747807a13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d45a9d-8ffwn" Oct 31 14:07:02.255161 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:02.255162 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70e9e98e-ef42-4867-bd8a-e89747807a13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36d45a9d-8ffwn\" (UID: \"70e9e98e-ef42-4867-bd8a-e89747807a13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d45a9d-8ffwn" Oct 31 14:07:02.255371 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:02.255249 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnk8b\" (UniqueName: \"kubernetes.io/projected/70e9e98e-ef42-4867-bd8a-e89747807a13-kube-api-access-nnk8b\") pod \"maas-api-revoke-sub-36d45a9d-8ffwn\" (UID: \"70e9e98e-ef42-4867-bd8a-e89747807a13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d45a9d-8ffwn" Oct 31 14:07:02.255924 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:02.255890 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70e9e98e-ef42-4867-bd8a-e89747807a13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36d45a9d-8ffwn\" (UID: \"70e9e98e-ef42-4867-bd8a-e89747807a13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d45a9d-8ffwn" Oct 31 14:07:02.257670 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:02.257651 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70e9e98e-ef42-4867-bd8a-e89747807a13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36d45a9d-8ffwn\" (UID: \"70e9e98e-ef42-4867-bd8a-e89747807a13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d45a9d-8ffwn" Oct 31 14:07:02.267809 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:02.267787 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnk8b\" (UniqueName: \"kubernetes.io/projected/70e9e98e-ef42-4867-bd8a-e89747807a13-kube-api-access-nnk8b\") pod \"maas-api-revoke-sub-36d45a9d-8ffwn\" (UID: \"70e9e98e-ef42-4867-bd8a-e89747807a13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d45a9d-8ffwn" Oct 31 14:07:02.360561 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:02.360484 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d45a9d-8ffwn" Oct 31 14:07:02.640285 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:02.640255 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36d45a9d-8ffwn"] Oct 31 14:07:02.646115 ip-10-0-131-157 kubenswrapper[5693]: W1031 14:07:02.643794 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70e9e98e_ef42_4867_bd8a_e89747807a13.slice/crio-cc70468d8f0c0906130e7d1b806912f15c648f2f895df1882958ce58b98cab93 WatchSource:0}: Error finding container cc70468d8f0c0906130e7d1b806912f15c648f2f895df1882958ce58b98cab93: Status 404 returned error can't find the container with id cc70468d8f0c0906130e7d1b806912f15c648f2f895df1882958ce58b98cab93 Oct 31 14:07:02.951766 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:02.951728 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d45a9d-8ffwn" event={"ID":"70e9e98e-ef42-4867-bd8a-e89747807a13","Type":"ContainerStarted","Data":"cc70468d8f0c0906130e7d1b806912f15c648f2f895df1882958ce58b98cab93"} Oct 31 14:07:03.957701 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:03.957668 5693 generic.go:356] "Generic (PLEG): container finished" podID="70e9e98e-ef42-4867-bd8a-e89747807a13" containerID="6557d7d516a953ac14692a0317359ab42d8c7ed544f9d1b10ea091855cae1253" exitCode=0 Oct 31 14:07:03.958119 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:03.957751 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d45a9d-8ffwn" event={"ID":"70e9e98e-ef42-4867-bd8a-e89747807a13","Type":"ContainerDied","Data":"6557d7d516a953ac14692a0317359ab42d8c7ed544f9d1b10ea091855cae1253"} Oct 31 14:07:05.095542 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:05.095513 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d45a9d-8ffwn" Oct 31 14:07:05.288965 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:05.288879 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70e9e98e-ef42-4867-bd8a-e89747807a13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70e9e98e-ef42-4867-bd8a-e89747807a13-maas-api-xks-ca\") pod \"70e9e98e-ef42-4867-bd8a-e89747807a13\" (UID: \"70e9e98e-ef42-4867-bd8a-e89747807a13\") " Oct 31 14:07:05.288965 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:05.288936 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70e9e98e-ef42-4867-bd8a-e89747807a13-kube-api-access-nnk8b\" (UniqueName: \"kubernetes.io/projected/70e9e98e-ef42-4867-bd8a-e89747807a13-kube-api-access-nnk8b\") pod \"70e9e98e-ef42-4867-bd8a-e89747807a13\" (UID: \"70e9e98e-ef42-4867-bd8a-e89747807a13\") " Oct 31 14:07:05.289165 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:05.289023 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70e9e98e-ef42-4867-bd8a-e89747807a13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70e9e98e-ef42-4867-bd8a-e89747807a13-maas-api-openshift-ca\") pod \"70e9e98e-ef42-4867-bd8a-e89747807a13\" (UID: \"70e9e98e-ef42-4867-bd8a-e89747807a13\") " Oct 31 14:07:05.289423 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:05.289398 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70e9e98e-ef42-4867-bd8a-e89747807a13-maas-api-openshift-ca" pod "70e9e98e-ef42-4867-bd8a-e89747807a13" (UID: "70e9e98e-ef42-4867-bd8a-e89747807a13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:07:05.291330 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:05.291306 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70e9e98e-ef42-4867-bd8a-e89747807a13-kube-api-access-nnk8b" pod "70e9e98e-ef42-4867-bd8a-e89747807a13" (UID: "70e9e98e-ef42-4867-bd8a-e89747807a13"). InnerVolumeSpecName "kube-api-access-nnk8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:07:05.291482 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:05.291332 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70e9e98e-ef42-4867-bd8a-e89747807a13-maas-api-xks-ca" pod "70e9e98e-ef42-4867-bd8a-e89747807a13" (UID: "70e9e98e-ef42-4867-bd8a-e89747807a13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:07:05.390060 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:05.390027 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70e9e98e-ef42-4867-bd8a-e89747807a13-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:07:05.390060 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:05.390054 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnk8b\" (UniqueName: \"kubernetes.io/projected/70e9e98e-ef42-4867-bd8a-e89747807a13-kube-api-access-nnk8b\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:07:05.390060 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:05.390067 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70e9e98e-ef42-4867-bd8a-e89747807a13-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:07:05.967847 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:05.967820 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d45a9d-8ffwn" Oct 31 14:07:05.967847 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:05.967830 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d45a9d-8ffwn" event={"ID":"70e9e98e-ef42-4867-bd8a-e89747807a13","Type":"ContainerDied","Data":"cc70468d8f0c0906130e7d1b806912f15c648f2f895df1882958ce58b98cab93"} Oct 31 14:07:05.968076 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:05.967872 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc70468d8f0c0906130e7d1b806912f15c648f2f895df1882958ce58b98cab93" Oct 31 14:07:12.256264 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:12.256183 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1def42e9-989cb"] Oct 31 14:07:12.256829 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:12.256812 5693 cpu_manager.go:414] "containerMap: removing container" podUID="70e9e98e-ef42-4867-bd8a-e89747807a13" containerName="revoke-keys" Oct 31 14:07:12.256880 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:12.256834 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70e9e98e-ef42-4867-bd8a-e89747807a13" containerName="revoke-keys" Oct 31 14:07:12.256952 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:12.256940 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="70e9e98e-ef42-4867-bd8a-e89747807a13" containerName="revoke-keys" Oct 31 14:07:12.260692 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:12.260671 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1def42e9-989cb" Oct 31 14:07:12.266969 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:12.266006 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:07:12.266969 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:12.266199 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 14:07:12.269463 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:12.269394 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1def42e9-989cb"] Oct 31 14:07:12.345995 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:12.345961 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7bjm\" (UniqueName: \"kubernetes.io/projected/09b4b740-e954-4876-909f-04f206aa9162-kube-api-access-m7bjm\") pod \"maas-api-revoke-sub-1def42e9-989cb\" (UID: \"09b4b740-e954-4876-909f-04f206aa9162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1def42e9-989cb" Oct 31 14:07:12.346170 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:12.346018 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b4b740-e954-4876-909f-04f206aa9162-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1def42e9-989cb\" (UID: \"09b4b740-e954-4876-909f-04f206aa9162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1def42e9-989cb" Oct 31 14:07:12.346170 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:12.346110 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b4b740-e954-4876-909f-04f206aa9162-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1def42e9-989cb\" (UID: \"09b4b740-e954-4876-909f-04f206aa9162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1def42e9-989cb" Oct 31 14:07:12.446776 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:12.446736 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b4b740-e954-4876-909f-04f206aa9162-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1def42e9-989cb\" (UID: \"09b4b740-e954-4876-909f-04f206aa9162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1def42e9-989cb" Oct 31 14:07:12.446961 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:12.446784 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b4b740-e954-4876-909f-04f206aa9162-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1def42e9-989cb\" (UID: \"09b4b740-e954-4876-909f-04f206aa9162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1def42e9-989cb" Oct 31 14:07:12.446961 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:12.446888 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7bjm\" (UniqueName: \"kubernetes.io/projected/09b4b740-e954-4876-909f-04f206aa9162-kube-api-access-m7bjm\") pod \"maas-api-revoke-sub-1def42e9-989cb\" (UID: \"09b4b740-e954-4876-909f-04f206aa9162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1def42e9-989cb" Oct 31 14:07:12.447563 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:12.447499 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b4b740-e954-4876-909f-04f206aa9162-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1def42e9-989cb\" (UID: \"09b4b740-e954-4876-909f-04f206aa9162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1def42e9-989cb" Oct 31 14:07:12.449344 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:12.449320 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b4b740-e954-4876-909f-04f206aa9162-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1def42e9-989cb\" (UID: \"09b4b740-e954-4876-909f-04f206aa9162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1def42e9-989cb" Oct 31 14:07:12.459277 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:12.459254 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7bjm\" (UniqueName: \"kubernetes.io/projected/09b4b740-e954-4876-909f-04f206aa9162-kube-api-access-m7bjm\") pod \"maas-api-revoke-sub-1def42e9-989cb\" (UID: \"09b4b740-e954-4876-909f-04f206aa9162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1def42e9-989cb" Oct 31 14:07:12.579572 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:12.579496 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1def42e9-989cb" Oct 31 14:07:12.859250 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:12.859171 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1def42e9-989cb"] Oct 31 14:07:12.863230 ip-10-0-131-157 kubenswrapper[5693]: W1031 14:07:12.863201 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09b4b740_e954_4876_909f_04f206aa9162.slice/crio-438131a411f7ff10700e8a99416413ae87db7910f9df4b6da3b37e4a26d094be WatchSource:0}: Error finding container 438131a411f7ff10700e8a99416413ae87db7910f9df4b6da3b37e4a26d094be: Status 404 returned error can't find the container with id 438131a411f7ff10700e8a99416413ae87db7910f9df4b6da3b37e4a26d094be Oct 31 14:07:13.008444 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:13.008410 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1def42e9-989cb" event={"ID":"09b4b740-e954-4876-909f-04f206aa9162","Type":"ContainerStarted","Data":"438131a411f7ff10700e8a99416413ae87db7910f9df4b6da3b37e4a26d094be"} Oct 31 14:07:14.016663 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:14.016610 5693 generic.go:356] "Generic (PLEG): container finished" podID="09b4b740-e954-4876-909f-04f206aa9162" containerID="32ac4a8c071a6c7350676a999b8812df7937506c99979dba9d099b27c2eabf21" exitCode=0 Oct 31 14:07:14.017069 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:14.016700 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1def42e9-989cb" event={"ID":"09b4b740-e954-4876-909f-04f206aa9162","Type":"ContainerDied","Data":"32ac4a8c071a6c7350676a999b8812df7937506c99979dba9d099b27c2eabf21"} Oct 31 14:07:15.390841 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:15.390811 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1def42e9-989cb" Oct 31 14:07:15.500669 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:15.499094 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09b4b740-e954-4876-909f-04f206aa9162-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b4b740-e954-4876-909f-04f206aa9162-maas-api-xks-ca\") pod \"09b4b740-e954-4876-909f-04f206aa9162\" (UID: \"09b4b740-e954-4876-909f-04f206aa9162\") " Oct 31 14:07:15.500669 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:15.499263 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09b4b740-e954-4876-909f-04f206aa9162-kube-api-access-m7bjm\" (UniqueName: \"kubernetes.io/projected/09b4b740-e954-4876-909f-04f206aa9162-kube-api-access-m7bjm\") pod \"09b4b740-e954-4876-909f-04f206aa9162\" (UID: \"09b4b740-e954-4876-909f-04f206aa9162\") " Oct 31 14:07:15.500669 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:15.499350 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09b4b740-e954-4876-909f-04f206aa9162-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b4b740-e954-4876-909f-04f206aa9162-maas-api-openshift-ca\") pod \"09b4b740-e954-4876-909f-04f206aa9162\" (UID: \"09b4b740-e954-4876-909f-04f206aa9162\") " Oct 31 14:07:15.513129 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:15.513079 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09b4b740-e954-4876-909f-04f206aa9162-maas-api-openshift-ca" pod "09b4b740-e954-4876-909f-04f206aa9162" (UID: "09b4b740-e954-4876-909f-04f206aa9162"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:07:15.557929 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:15.548945 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09b4b740-e954-4876-909f-04f206aa9162-kube-api-access-m7bjm" pod "09b4b740-e954-4876-909f-04f206aa9162" (UID: "09b4b740-e954-4876-909f-04f206aa9162"). InnerVolumeSpecName "kube-api-access-m7bjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:07:15.557929 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:15.557869 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09b4b740-e954-4876-909f-04f206aa9162-maas-api-xks-ca" pod "09b4b740-e954-4876-909f-04f206aa9162" (UID: "09b4b740-e954-4876-909f-04f206aa9162"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:07:15.619723 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:15.619261 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7bjm\" (UniqueName: \"kubernetes.io/projected/09b4b740-e954-4876-909f-04f206aa9162-kube-api-access-m7bjm\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:07:15.619723 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:15.619304 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b4b740-e954-4876-909f-04f206aa9162-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:07:15.619723 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:15.619323 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b4b740-e954-4876-909f-04f206aa9162-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:07:16.036854 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:16.036822 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1def42e9-989cb" Oct 31 14:07:16.037034 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:16.036853 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1def42e9-989cb" event={"ID":"09b4b740-e954-4876-909f-04f206aa9162","Type":"ContainerDied","Data":"438131a411f7ff10700e8a99416413ae87db7910f9df4b6da3b37e4a26d094be"} Oct 31 14:07:16.037034 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:16.036901 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="438131a411f7ff10700e8a99416413ae87db7910f9df4b6da3b37e4a26d094be" Oct 31 14:07:18.686522 ip-10-0-131-157 kubenswrapper[5693]: E1031 14:07:18.686485 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 14:07:46.774508 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:46.774474 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e8be16b-cjzpw"] Oct 31 14:07:46.777128 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:46.774857 5693 cpu_manager.go:414] "containerMap: removing container" podUID="09b4b740-e954-4876-909f-04f206aa9162" containerName="revoke-keys" Oct 31 14:07:46.777128 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:46.774871 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09b4b740-e954-4876-909f-04f206aa9162" containerName="revoke-keys" Oct 31 14:07:46.777128 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:46.774937 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="09b4b740-e954-4876-909f-04f206aa9162" containerName="revoke-keys" Oct 31 14:07:46.777892 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:46.777877 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e8be16b-cjzpw" Oct 31 14:07:46.782223 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:46.782199 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:07:46.782369 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:46.782257 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 14:07:46.787288 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:46.787248 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e8be16b-cjzpw"] Oct 31 14:07:46.889956 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:46.889922 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a718057-f877-459f-a205-18857d1e7f0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e8be16b-cjzpw\" (UID: \"4a718057-f877-459f-a205-18857d1e7f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e8be16b-cjzpw" Oct 31 14:07:46.890119 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:46.889973 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a718057-f877-459f-a205-18857d1e7f0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e8be16b-cjzpw\" (UID: \"4a718057-f877-459f-a205-18857d1e7f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e8be16b-cjzpw" Oct 31 14:07:46.890119 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:46.890067 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqkdn\" (UniqueName: \"kubernetes.io/projected/4a718057-f877-459f-a205-18857d1e7f0c-kube-api-access-vqkdn\") pod \"maas-api-revoke-sub-1e8be16b-cjzpw\" (UID: \"4a718057-f877-459f-a205-18857d1e7f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e8be16b-cjzpw" Oct 31 14:07:46.991262 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:46.991227 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqkdn\" (UniqueName: \"kubernetes.io/projected/4a718057-f877-459f-a205-18857d1e7f0c-kube-api-access-vqkdn\") pod \"maas-api-revoke-sub-1e8be16b-cjzpw\" (UID: \"4a718057-f877-459f-a205-18857d1e7f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e8be16b-cjzpw" Oct 31 14:07:46.991465 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:46.991291 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a718057-f877-459f-a205-18857d1e7f0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e8be16b-cjzpw\" (UID: \"4a718057-f877-459f-a205-18857d1e7f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e8be16b-cjzpw" Oct 31 14:07:46.991465 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:46.991341 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a718057-f877-459f-a205-18857d1e7f0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e8be16b-cjzpw\" (UID: \"4a718057-f877-459f-a205-18857d1e7f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e8be16b-cjzpw" Oct 31 14:07:46.992127 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:46.992108 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a718057-f877-459f-a205-18857d1e7f0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e8be16b-cjzpw\" (UID: \"4a718057-f877-459f-a205-18857d1e7f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e8be16b-cjzpw" Oct 31 14:07:46.993873 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:46.993856 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a718057-f877-459f-a205-18857d1e7f0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e8be16b-cjzpw\" (UID: \"4a718057-f877-459f-a205-18857d1e7f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e8be16b-cjzpw" Oct 31 14:07:47.004019 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:47.003999 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqkdn\" (UniqueName: \"kubernetes.io/projected/4a718057-f877-459f-a205-18857d1e7f0c-kube-api-access-vqkdn\") pod \"maas-api-revoke-sub-1e8be16b-cjzpw\" (UID: \"4a718057-f877-459f-a205-18857d1e7f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e8be16b-cjzpw" Oct 31 14:07:47.093517 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:47.093445 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e8be16b-cjzpw" Oct 31 14:07:47.367717 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:47.367691 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e8be16b-cjzpw"] Oct 31 14:07:47.370329 ip-10-0-131-157 kubenswrapper[5693]: W1031 14:07:47.370303 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a718057_f877_459f_a205_18857d1e7f0c.slice/crio-e8a7454e04e0ebea58e31fd6498b7364bd39d320e4a0f8f7a9bb9c26a3cc10f0 WatchSource:0}: Error finding container e8a7454e04e0ebea58e31fd6498b7364bd39d320e4a0f8f7a9bb9c26a3cc10f0: Status 404 returned error can't find the container with id e8a7454e04e0ebea58e31fd6498b7364bd39d320e4a0f8f7a9bb9c26a3cc10f0 Oct 31 14:07:48.255414 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:48.255378 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e8be16b-cjzpw" event={"ID":"4a718057-f877-459f-a205-18857d1e7f0c","Type":"ContainerStarted","Data":"48aa0f56fa839f51568cc87b3604278d45a6eacafca948381d3a0469c9abc586"} Oct 31 14:07:48.255870 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:48.255441 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e8be16b-cjzpw" event={"ID":"4a718057-f877-459f-a205-18857d1e7f0c","Type":"ContainerStarted","Data":"e8a7454e04e0ebea58e31fd6498b7364bd39d320e4a0f8f7a9bb9c26a3cc10f0"} Oct 31 14:07:48.275624 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:48.275573 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e8be16b-cjzpw" podStartSLOduration=2.275558357 podStartE2EDuration="2.275558357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:07:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:07:48.27318982 +0000 UTC m=+5718.980776116" watchObservedRunningTime="2026-10-31 14:07:48.275558357 +0000 UTC m=+5718.983144651" Oct 31 14:07:48.686046 ip-10-0-131-157 kubenswrapper[5693]: E1031 14:07:48.686003 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 14:07:49.261063 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:49.261032 5693 generic.go:356] "Generic (PLEG): container finished" podID="4a718057-f877-459f-a205-18857d1e7f0c" containerID="48aa0f56fa839f51568cc87b3604278d45a6eacafca948381d3a0469c9abc586" exitCode=0 Oct 31 14:07:49.261504 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:49.261130 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e8be16b-cjzpw" event={"ID":"4a718057-f877-459f-a205-18857d1e7f0c","Type":"ContainerDied","Data":"48aa0f56fa839f51568cc87b3604278d45a6eacafca948381d3a0469c9abc586"} Oct 31 14:07:50.404284 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:50.404258 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e8be16b-cjzpw" Oct 31 14:07:50.522960 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:50.522883 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a718057-f877-459f-a205-18857d1e7f0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a718057-f877-459f-a205-18857d1e7f0c-maas-api-xks-ca\") pod \"4a718057-f877-459f-a205-18857d1e7f0c\" (UID: \"4a718057-f877-459f-a205-18857d1e7f0c\") " Oct 31 14:07:50.523125 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:50.522961 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a718057-f877-459f-a205-18857d1e7f0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a718057-f877-459f-a205-18857d1e7f0c-maas-api-openshift-ca\") pod \"4a718057-f877-459f-a205-18857d1e7f0c\" (UID: \"4a718057-f877-459f-a205-18857d1e7f0c\") " Oct 31 14:07:50.523125 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:50.523036 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a718057-f877-459f-a205-18857d1e7f0c-kube-api-access-vqkdn\" (UniqueName: \"kubernetes.io/projected/4a718057-f877-459f-a205-18857d1e7f0c-kube-api-access-vqkdn\") pod \"4a718057-f877-459f-a205-18857d1e7f0c\" (UID: \"4a718057-f877-459f-a205-18857d1e7f0c\") " Oct 31 14:07:50.523349 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:50.523323 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a718057-f877-459f-a205-18857d1e7f0c-maas-api-openshift-ca" pod "4a718057-f877-459f-a205-18857d1e7f0c" (UID: "4a718057-f877-459f-a205-18857d1e7f0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:07:50.525285 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:50.525253 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a718057-f877-459f-a205-18857d1e7f0c-maas-api-xks-ca" pod "4a718057-f877-459f-a205-18857d1e7f0c" (UID: "4a718057-f877-459f-a205-18857d1e7f0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:07:50.525285 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:50.525273 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a718057-f877-459f-a205-18857d1e7f0c-kube-api-access-vqkdn" pod "4a718057-f877-459f-a205-18857d1e7f0c" (UID: "4a718057-f877-459f-a205-18857d1e7f0c"). InnerVolumeSpecName "kube-api-access-vqkdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:07:50.623999 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:50.623967 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a718057-f877-459f-a205-18857d1e7f0c-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:07:50.623999 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:50.623996 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a718057-f877-459f-a205-18857d1e7f0c-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:07:50.623999 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:50.624008 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqkdn\" (UniqueName: \"kubernetes.io/projected/4a718057-f877-459f-a205-18857d1e7f0c-kube-api-access-vqkdn\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:07:51.271799 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:51.271755 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e8be16b-cjzpw" event={"ID":"4a718057-f877-459f-a205-18857d1e7f0c","Type":"ContainerDied","Data":"e8a7454e04e0ebea58e31fd6498b7364bd39d320e4a0f8f7a9bb9c26a3cc10f0"} Oct 31 14:07:51.271958 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:51.271815 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8a7454e04e0ebea58e31fd6498b7364bd39d320e4a0f8f7a9bb9c26a3cc10f0" Oct 31 14:07:51.271958 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:51.271771 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e8be16b-cjzpw" Oct 31 14:07:56.999758 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:56.999716 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa2f15ae-zlgnk"] Oct 31 14:07:57.000649 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:57.000620 5693 cpu_manager.go:414] "containerMap: removing container" podUID="4a718057-f877-459f-a205-18857d1e7f0c" containerName="revoke-keys" Oct 31 14:07:57.000649 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:57.000657 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a718057-f877-459f-a205-18857d1e7f0c" containerName="revoke-keys" Oct 31 14:07:57.000906 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:57.000771 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a718057-f877-459f-a205-18857d1e7f0c" containerName="revoke-keys" Oct 31 14:07:57.004094 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:57.004076 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2f15ae-zlgnk" Oct 31 14:07:57.017706 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:57.010543 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:07:57.017706 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:57.013829 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 14:07:57.017706 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:57.014202 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa2f15ae-zlgnk"] Oct 31 14:07:57.083760 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:57.083725 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c302de7-718b-45a7-8be3-2ecc34622c5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa2f15ae-zlgnk\" (UID: \"5c302de7-718b-45a7-8be3-2ecc34622c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2f15ae-zlgnk" Oct 31 14:07:57.083965 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:57.083792 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxkkm\" (UniqueName: \"kubernetes.io/projected/5c302de7-718b-45a7-8be3-2ecc34622c5d-kube-api-access-dxkkm\") pod \"maas-api-revoke-sub-fa2f15ae-zlgnk\" (UID: \"5c302de7-718b-45a7-8be3-2ecc34622c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2f15ae-zlgnk" Oct 31 14:07:57.084012 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:57.083970 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c302de7-718b-45a7-8be3-2ecc34622c5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa2f15ae-zlgnk\" (UID: \"5c302de7-718b-45a7-8be3-2ecc34622c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2f15ae-zlgnk" Oct 31 14:07:57.185328 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:57.185288 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c302de7-718b-45a7-8be3-2ecc34622c5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa2f15ae-zlgnk\" (UID: \"5c302de7-718b-45a7-8be3-2ecc34622c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2f15ae-zlgnk" Oct 31 14:07:57.185554 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:57.185368 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c302de7-718b-45a7-8be3-2ecc34622c5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa2f15ae-zlgnk\" (UID: \"5c302de7-718b-45a7-8be3-2ecc34622c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2f15ae-zlgnk" Oct 31 14:07:57.185554 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:57.185397 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxkkm\" (UniqueName: \"kubernetes.io/projected/5c302de7-718b-45a7-8be3-2ecc34622c5d-kube-api-access-dxkkm\") pod \"maas-api-revoke-sub-fa2f15ae-zlgnk\" (UID: \"5c302de7-718b-45a7-8be3-2ecc34622c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2f15ae-zlgnk" Oct 31 14:07:57.186120 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:57.186099 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c302de7-718b-45a7-8be3-2ecc34622c5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa2f15ae-zlgnk\" (UID: \"5c302de7-718b-45a7-8be3-2ecc34622c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2f15ae-zlgnk" Oct 31 14:07:57.187967 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:57.187941 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c302de7-718b-45a7-8be3-2ecc34622c5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa2f15ae-zlgnk\" (UID: \"5c302de7-718b-45a7-8be3-2ecc34622c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2f15ae-zlgnk" Oct 31 14:07:57.197834 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:57.197809 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxkkm\" (UniqueName: \"kubernetes.io/projected/5c302de7-718b-45a7-8be3-2ecc34622c5d-kube-api-access-dxkkm\") pod \"maas-api-revoke-sub-fa2f15ae-zlgnk\" (UID: \"5c302de7-718b-45a7-8be3-2ecc34622c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2f15ae-zlgnk" Oct 31 14:07:57.320501 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:57.320401 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2f15ae-zlgnk" Oct 31 14:07:57.597261 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:57.597031 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa2f15ae-zlgnk"] Oct 31 14:07:57.599554 ip-10-0-131-157 kubenswrapper[5693]: W1031 14:07:57.599527 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c302de7_718b_45a7_8be3_2ecc34622c5d.slice/crio-c4d415594a5199031784268a0dd62c8d23cdcaee6bf625c5d8ccc730a3ad1088 WatchSource:0}: Error finding container c4d415594a5199031784268a0dd62c8d23cdcaee6bf625c5d8ccc730a3ad1088: Status 404 returned error can't find the container with id c4d415594a5199031784268a0dd62c8d23cdcaee6bf625c5d8ccc730a3ad1088 Oct 31 14:07:58.305961 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:58.305923 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2f15ae-zlgnk" event={"ID":"5c302de7-718b-45a7-8be3-2ecc34622c5d","Type":"ContainerStarted","Data":"9a29846e739ac80b9b889bd1ee914354b18f1057754274abdaaa7449719f7b47"} Oct 31 14:07:58.306453 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:58.305980 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2f15ae-zlgnk" event={"ID":"5c302de7-718b-45a7-8be3-2ecc34622c5d","Type":"ContainerStarted","Data":"c4d415594a5199031784268a0dd62c8d23cdcaee6bf625c5d8ccc730a3ad1088"} Oct 31 14:07:58.327084 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:58.327036 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2f15ae-zlgnk" podStartSLOduration=2.327020998 podStartE2EDuration="2.327020998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:07:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:07:58.324902333 +0000 UTC m=+5729.032488629" watchObservedRunningTime="2026-10-31 14:07:58.327020998 +0000 UTC m=+5729.034607292" Oct 31 14:07:59.313910 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:59.313872 5693 generic.go:356] "Generic (PLEG): container finished" podID="5c302de7-718b-45a7-8be3-2ecc34622c5d" containerID="9a29846e739ac80b9b889bd1ee914354b18f1057754274abdaaa7449719f7b47" exitCode=0 Oct 31 14:07:59.314441 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:07:59.313941 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2f15ae-zlgnk" event={"ID":"5c302de7-718b-45a7-8be3-2ecc34622c5d","Type":"ContainerDied","Data":"9a29846e739ac80b9b889bd1ee914354b18f1057754274abdaaa7449719f7b47"} Oct 31 14:08:00.656503 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:00.656464 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2f15ae-zlgnk" Oct 31 14:08:00.664587 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:00.664562 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c302de7-718b-45a7-8be3-2ecc34622c5d-kube-api-access-dxkkm\" (UniqueName: \"kubernetes.io/projected/5c302de7-718b-45a7-8be3-2ecc34622c5d-kube-api-access-dxkkm\") pod \"5c302de7-718b-45a7-8be3-2ecc34622c5d\" (UID: \"5c302de7-718b-45a7-8be3-2ecc34622c5d\") " Oct 31 14:08:00.664765 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:00.664631 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c302de7-718b-45a7-8be3-2ecc34622c5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c302de7-718b-45a7-8be3-2ecc34622c5d-maas-api-openshift-ca\") pod \"5c302de7-718b-45a7-8be3-2ecc34622c5d\" (UID: \"5c302de7-718b-45a7-8be3-2ecc34622c5d\") " Oct 31 14:08:00.664839 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:00.664783 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c302de7-718b-45a7-8be3-2ecc34622c5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c302de7-718b-45a7-8be3-2ecc34622c5d-maas-api-xks-ca\") pod \"5c302de7-718b-45a7-8be3-2ecc34622c5d\" (UID: \"5c302de7-718b-45a7-8be3-2ecc34622c5d\") " Oct 31 14:08:00.665088 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:00.665002 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c302de7-718b-45a7-8be3-2ecc34622c5d-maas-api-openshift-ca" pod "5c302de7-718b-45a7-8be3-2ecc34622c5d" (UID: "5c302de7-718b-45a7-8be3-2ecc34622c5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:08:00.665226 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:00.665125 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c302de7-718b-45a7-8be3-2ecc34622c5d-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:08:00.666910 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:00.666886 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c302de7-718b-45a7-8be3-2ecc34622c5d-kube-api-access-dxkkm" pod "5c302de7-718b-45a7-8be3-2ecc34622c5d" (UID: "5c302de7-718b-45a7-8be3-2ecc34622c5d"). InnerVolumeSpecName "kube-api-access-dxkkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:08:00.667689 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:00.667663 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c302de7-718b-45a7-8be3-2ecc34622c5d-maas-api-xks-ca" pod "5c302de7-718b-45a7-8be3-2ecc34622c5d" (UID: "5c302de7-718b-45a7-8be3-2ecc34622c5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:08:00.765841 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:00.765798 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c302de7-718b-45a7-8be3-2ecc34622c5d-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:08:00.765841 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:00.765830 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxkkm\" (UniqueName: \"kubernetes.io/projected/5c302de7-718b-45a7-8be3-2ecc34622c5d-kube-api-access-dxkkm\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:08:01.344962 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:01.344929 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2f15ae-zlgnk" Oct 31 14:08:01.345145 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:01.344958 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2f15ae-zlgnk" event={"ID":"5c302de7-718b-45a7-8be3-2ecc34622c5d","Type":"ContainerDied","Data":"c4d415594a5199031784268a0dd62c8d23cdcaee6bf625c5d8ccc730a3ad1088"} Oct 31 14:08:01.345145 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:01.345033 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4d415594a5199031784268a0dd62c8d23cdcaee6bf625c5d8ccc730a3ad1088" Oct 31 14:08:07.217508 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:07.217471 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d96f6927-dh89l"] Oct 31 14:08:07.217931 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:07.217866 5693 cpu_manager.go:414] "containerMap: removing container" podUID="5c302de7-718b-45a7-8be3-2ecc34622c5d" containerName="revoke-keys" Oct 31 14:08:07.217931 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:07.217878 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c302de7-718b-45a7-8be3-2ecc34622c5d" containerName="revoke-keys" Oct 31 14:08:07.218006 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:07.217939 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c302de7-718b-45a7-8be3-2ecc34622c5d" containerName="revoke-keys" Oct 31 14:08:07.221136 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:07.221119 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96f6927-dh89l" Oct 31 14:08:07.227371 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:07.227344 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:08:07.228856 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:07.228834 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 14:08:07.231271 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:07.231247 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d96f6927-dh89l"] Oct 31 14:08:07.327855 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:07.327741 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp6pq\" (UniqueName: \"kubernetes.io/projected/59907327-7199-4288-bd92-b64520eeff73-kube-api-access-gp6pq\") pod \"maas-api-revoke-sub-d96f6927-dh89l\" (UID: \"59907327-7199-4288-bd92-b64520eeff73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96f6927-dh89l" Oct 31 14:08:07.328042 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:07.327907 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59907327-7199-4288-bd92-b64520eeff73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d96f6927-dh89l\" (UID: \"59907327-7199-4288-bd92-b64520eeff73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96f6927-dh89l" Oct 31 14:08:07.328042 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:07.327970 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59907327-7199-4288-bd92-b64520eeff73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d96f6927-dh89l\" (UID: \"59907327-7199-4288-bd92-b64520eeff73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96f6927-dh89l" Oct 31 14:08:07.429191 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:07.429144 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59907327-7199-4288-bd92-b64520eeff73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d96f6927-dh89l\" (UID: \"59907327-7199-4288-bd92-b64520eeff73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96f6927-dh89l" Oct 31 14:08:07.429191 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:07.429192 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp6pq\" (UniqueName: \"kubernetes.io/projected/59907327-7199-4288-bd92-b64520eeff73-kube-api-access-gp6pq\") pod \"maas-api-revoke-sub-d96f6927-dh89l\" (UID: \"59907327-7199-4288-bd92-b64520eeff73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96f6927-dh89l" Oct 31 14:08:07.429426 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:07.429318 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59907327-7199-4288-bd92-b64520eeff73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d96f6927-dh89l\" (UID: \"59907327-7199-4288-bd92-b64520eeff73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96f6927-dh89l" Oct 31 14:08:07.429952 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:07.429931 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59907327-7199-4288-bd92-b64520eeff73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d96f6927-dh89l\" (UID: \"59907327-7199-4288-bd92-b64520eeff73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96f6927-dh89l" Oct 31 14:08:07.431675 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:07.431657 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59907327-7199-4288-bd92-b64520eeff73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d96f6927-dh89l\" (UID: \"59907327-7199-4288-bd92-b64520eeff73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96f6927-dh89l" Oct 31 14:08:07.441429 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:07.441401 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp6pq\" (UniqueName: \"kubernetes.io/projected/59907327-7199-4288-bd92-b64520eeff73-kube-api-access-gp6pq\") pod \"maas-api-revoke-sub-d96f6927-dh89l\" (UID: \"59907327-7199-4288-bd92-b64520eeff73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96f6927-dh89l" Oct 31 14:08:07.535390 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:07.535358 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96f6927-dh89l" Oct 31 14:08:07.810610 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:07.810580 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d96f6927-dh89l"] Oct 31 14:08:07.813059 ip-10-0-131-157 kubenswrapper[5693]: W1031 14:08:07.813031 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59907327_7199_4288_bd92_b64520eeff73.slice/crio-66cb57723c52c577255312d7e269160a3aa466fd50f5783e4445d5e42a2cdbcc WatchSource:0}: Error finding container 66cb57723c52c577255312d7e269160a3aa466fd50f5783e4445d5e42a2cdbcc: Status 404 returned error can't find the container with id 66cb57723c52c577255312d7e269160a3aa466fd50f5783e4445d5e42a2cdbcc Oct 31 14:08:08.378510 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:08.378467 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96f6927-dh89l" event={"ID":"59907327-7199-4288-bd92-b64520eeff73","Type":"ContainerStarted","Data":"3ae5c216c0a3448b99df8a6b9777bba7649b8206781372b3912376e97576d8cf"} Oct 31 14:08:08.378914 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:08.378526 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96f6927-dh89l" event={"ID":"59907327-7199-4288-bd92-b64520eeff73","Type":"ContainerStarted","Data":"66cb57723c52c577255312d7e269160a3aa466fd50f5783e4445d5e42a2cdbcc"} Oct 31 14:08:08.397197 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:08.397149 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96f6927-dh89l" podStartSLOduration=1.397136336 podStartE2EDuration="1.397136336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:08:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:08:08.395614206 +0000 UTC m=+5739.103200502" watchObservedRunningTime="2026-10-31 14:08:08.397136336 +0000 UTC m=+5739.104722699" Oct 31 14:08:09.384626 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:09.384594 5693 generic.go:356] "Generic (PLEG): container finished" podID="59907327-7199-4288-bd92-b64520eeff73" containerID="3ae5c216c0a3448b99df8a6b9777bba7649b8206781372b3912376e97576d8cf" exitCode=0 Oct 31 14:08:09.385079 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:09.384683 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96f6927-dh89l" event={"ID":"59907327-7199-4288-bd92-b64520eeff73","Type":"ContainerDied","Data":"3ae5c216c0a3448b99df8a6b9777bba7649b8206781372b3912376e97576d8cf"} Oct 31 14:08:10.526300 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:10.526279 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96f6927-dh89l" Oct 31 14:08:10.557718 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:10.557691 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59907327-7199-4288-bd92-b64520eeff73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59907327-7199-4288-bd92-b64520eeff73-maas-api-openshift-ca\") pod \"59907327-7199-4288-bd92-b64520eeff73\" (UID: \"59907327-7199-4288-bd92-b64520eeff73\") " Oct 31 14:08:10.557920 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:10.557830 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59907327-7199-4288-bd92-b64520eeff73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59907327-7199-4288-bd92-b64520eeff73-maas-api-xks-ca\") pod \"59907327-7199-4288-bd92-b64520eeff73\" (UID: \"59907327-7199-4288-bd92-b64520eeff73\") " Oct 31 14:08:10.557998 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:10.557920 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59907327-7199-4288-bd92-b64520eeff73-kube-api-access-gp6pq\" (UniqueName: \"kubernetes.io/projected/59907327-7199-4288-bd92-b64520eeff73-kube-api-access-gp6pq\") pod \"59907327-7199-4288-bd92-b64520eeff73\" (UID: \"59907327-7199-4288-bd92-b64520eeff73\") " Oct 31 14:08:10.558132 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:10.558111 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59907327-7199-4288-bd92-b64520eeff73-maas-api-openshift-ca" pod "59907327-7199-4288-bd92-b64520eeff73" (UID: "59907327-7199-4288-bd92-b64520eeff73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:08:10.558282 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:10.558264 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59907327-7199-4288-bd92-b64520eeff73-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:08:10.560140 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:10.560112 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59907327-7199-4288-bd92-b64520eeff73-kube-api-access-gp6pq" pod "59907327-7199-4288-bd92-b64520eeff73" (UID: "59907327-7199-4288-bd92-b64520eeff73"). InnerVolumeSpecName "kube-api-access-gp6pq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:08:10.560266 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:10.560246 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59907327-7199-4288-bd92-b64520eeff73-maas-api-xks-ca" pod "59907327-7199-4288-bd92-b64520eeff73" (UID: "59907327-7199-4288-bd92-b64520eeff73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:08:10.658966 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:10.658933 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59907327-7199-4288-bd92-b64520eeff73-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:08:10.658966 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:10.658964 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp6pq\" (UniqueName: \"kubernetes.io/projected/59907327-7199-4288-bd92-b64520eeff73-kube-api-access-gp6pq\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:08:11.394901 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:11.394865 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96f6927-dh89l" Oct 31 14:08:11.394901 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:11.394888 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96f6927-dh89l" event={"ID":"59907327-7199-4288-bd92-b64520eeff73","Type":"ContainerDied","Data":"66cb57723c52c577255312d7e269160a3aa466fd50f5783e4445d5e42a2cdbcc"} Oct 31 14:08:11.395131 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:11.394936 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66cb57723c52c577255312d7e269160a3aa466fd50f5783e4445d5e42a2cdbcc" Oct 31 14:08:17.431093 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:17.431061 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85719217-ffg8w"] Oct 31 14:08:17.431513 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:17.431450 5693 cpu_manager.go:414] "containerMap: removing container" podUID="59907327-7199-4288-bd92-b64520eeff73" containerName="revoke-keys" Oct 31 14:08:17.431513 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:17.431462 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59907327-7199-4288-bd92-b64520eeff73" containerName="revoke-keys" Oct 31 14:08:17.431587 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:17.431540 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="59907327-7199-4288-bd92-b64520eeff73" containerName="revoke-keys" Oct 31 14:08:17.434697 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:17.434677 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85719217-ffg8w" Oct 31 14:08:17.442037 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:17.442010 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:08:17.442148 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:17.442113 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 14:08:17.444356 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:17.444325 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85719217-ffg8w"] Oct 31 14:08:17.526973 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:17.526940 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f0de58d-6bbb-4c36-93c3-394defa7884b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85719217-ffg8w\" (UID: \"5f0de58d-6bbb-4c36-93c3-394defa7884b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85719217-ffg8w" Oct 31 14:08:17.527199 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:17.527175 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f0de58d-6bbb-4c36-93c3-394defa7884b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85719217-ffg8w\" (UID: \"5f0de58d-6bbb-4c36-93c3-394defa7884b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85719217-ffg8w" Oct 31 14:08:17.527297 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:17.527236 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbmrf\" (UniqueName: \"kubernetes.io/projected/5f0de58d-6bbb-4c36-93c3-394defa7884b-kube-api-access-kbmrf\") pod \"maas-api-revoke-sub-85719217-ffg8w\" (UID: \"5f0de58d-6bbb-4c36-93c3-394defa7884b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85719217-ffg8w" Oct 31 14:08:17.628259 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:17.628220 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f0de58d-6bbb-4c36-93c3-394defa7884b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85719217-ffg8w\" (UID: \"5f0de58d-6bbb-4c36-93c3-394defa7884b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85719217-ffg8w" Oct 31 14:08:17.628524 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:17.628283 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbmrf\" (UniqueName: \"kubernetes.io/projected/5f0de58d-6bbb-4c36-93c3-394defa7884b-kube-api-access-kbmrf\") pod \"maas-api-revoke-sub-85719217-ffg8w\" (UID: \"5f0de58d-6bbb-4c36-93c3-394defa7884b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85719217-ffg8w" Oct 31 14:08:17.628524 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:17.628335 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f0de58d-6bbb-4c36-93c3-394defa7884b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85719217-ffg8w\" (UID: \"5f0de58d-6bbb-4c36-93c3-394defa7884b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85719217-ffg8w" Oct 31 14:08:17.628954 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:17.628933 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f0de58d-6bbb-4c36-93c3-394defa7884b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85719217-ffg8w\" (UID: \"5f0de58d-6bbb-4c36-93c3-394defa7884b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85719217-ffg8w" Oct 31 14:08:17.630933 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:17.630908 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f0de58d-6bbb-4c36-93c3-394defa7884b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85719217-ffg8w\" (UID: \"5f0de58d-6bbb-4c36-93c3-394defa7884b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85719217-ffg8w" Oct 31 14:08:17.640518 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:17.640493 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbmrf\" (UniqueName: \"kubernetes.io/projected/5f0de58d-6bbb-4c36-93c3-394defa7884b-kube-api-access-kbmrf\") pod \"maas-api-revoke-sub-85719217-ffg8w\" (UID: \"5f0de58d-6bbb-4c36-93c3-394defa7884b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85719217-ffg8w" Oct 31 14:08:17.749840 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:17.749755 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85719217-ffg8w" Oct 31 14:08:18.029570 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:18.029545 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85719217-ffg8w"] Oct 31 14:08:18.032111 ip-10-0-131-157 kubenswrapper[5693]: W1031 14:08:18.032086 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f0de58d_6bbb_4c36_93c3_394defa7884b.slice/crio-8f7c0a3d256ccdcb73ae1f060baf68f84031ba4a7d9ae3c34399b1a8e5f5fbae WatchSource:0}: Error finding container 8f7c0a3d256ccdcb73ae1f060baf68f84031ba4a7d9ae3c34399b1a8e5f5fbae: Status 404 returned error can't find the container with id 8f7c0a3d256ccdcb73ae1f060baf68f84031ba4a7d9ae3c34399b1a8e5f5fbae Oct 31 14:08:18.431322 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:18.431280 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85719217-ffg8w" event={"ID":"5f0de58d-6bbb-4c36-93c3-394defa7884b","Type":"ContainerStarted","Data":"805e6a822277ecd73bd583bf533976b28e0c11491f8e57d696819b5809dd8f03"} Oct 31 14:08:18.431867 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:18.431346 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85719217-ffg8w" event={"ID":"5f0de58d-6bbb-4c36-93c3-394defa7884b","Type":"ContainerStarted","Data":"8f7c0a3d256ccdcb73ae1f060baf68f84031ba4a7d9ae3c34399b1a8e5f5fbae"} Oct 31 14:08:18.452319 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:18.452256 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85719217-ffg8w" podStartSLOduration=1.452236414 podStartE2EDuration="1.452236414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:08:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:08:18.449911523 +0000 UTC m=+5749.157497817" watchObservedRunningTime="2026-10-31 14:08:18.452236414 +0000 UTC m=+5749.159822710" Oct 31 14:08:18.690821 ip-10-0-131-157 kubenswrapper[5693]: E1031 14:08:18.690770 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 14:08:19.451319 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:19.451192 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85719217-ffg8w" event={"ID":"5f0de58d-6bbb-4c36-93c3-394defa7884b","Type":"ContainerDied","Data":"805e6a822277ecd73bd583bf533976b28e0c11491f8e57d696819b5809dd8f03"} Oct 31 14:08:19.451319 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:19.451197 5693 generic.go:356] "Generic (PLEG): container finished" podID="5f0de58d-6bbb-4c36-93c3-394defa7884b" containerID="805e6a822277ecd73bd583bf533976b28e0c11491f8e57d696819b5809dd8f03" exitCode=0 Oct 31 14:08:20.871509 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:20.871486 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85719217-ffg8w" Oct 31 14:08:20.969588 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:20.969556 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f0de58d-6bbb-4c36-93c3-394defa7884b-kube-api-access-kbmrf\" (UniqueName: \"kubernetes.io/projected/5f0de58d-6bbb-4c36-93c3-394defa7884b-kube-api-access-kbmrf\") pod \"5f0de58d-6bbb-4c36-93c3-394defa7884b\" (UID: \"5f0de58d-6bbb-4c36-93c3-394defa7884b\") " Oct 31 14:08:20.969808 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:20.969616 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f0de58d-6bbb-4c36-93c3-394defa7884b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f0de58d-6bbb-4c36-93c3-394defa7884b-maas-api-xks-ca\") pod \"5f0de58d-6bbb-4c36-93c3-394defa7884b\" (UID: \"5f0de58d-6bbb-4c36-93c3-394defa7884b\") " Oct 31 14:08:20.969808 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:20.969743 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f0de58d-6bbb-4c36-93c3-394defa7884b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f0de58d-6bbb-4c36-93c3-394defa7884b-maas-api-openshift-ca\") pod \"5f0de58d-6bbb-4c36-93c3-394defa7884b\" (UID: \"5f0de58d-6bbb-4c36-93c3-394defa7884b\") " Oct 31 14:08:20.970139 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:20.970099 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f0de58d-6bbb-4c36-93c3-394defa7884b-maas-api-openshift-ca" pod "5f0de58d-6bbb-4c36-93c3-394defa7884b" (UID: "5f0de58d-6bbb-4c36-93c3-394defa7884b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:08:20.972025 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:20.971999 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f0de58d-6bbb-4c36-93c3-394defa7884b-kube-api-access-kbmrf" pod "5f0de58d-6bbb-4c36-93c3-394defa7884b" (UID: "5f0de58d-6bbb-4c36-93c3-394defa7884b"). InnerVolumeSpecName "kube-api-access-kbmrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:08:20.972146 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:20.972063 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f0de58d-6bbb-4c36-93c3-394defa7884b-maas-api-xks-ca" pod "5f0de58d-6bbb-4c36-93c3-394defa7884b" (UID: "5f0de58d-6bbb-4c36-93c3-394defa7884b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:08:21.071448 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:21.071410 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbmrf\" (UniqueName: \"kubernetes.io/projected/5f0de58d-6bbb-4c36-93c3-394defa7884b-kube-api-access-kbmrf\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:08:21.071448 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:21.071448 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f0de58d-6bbb-4c36-93c3-394defa7884b-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:08:21.071745 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:21.071462 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f0de58d-6bbb-4c36-93c3-394defa7884b-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:08:21.463193 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:21.463164 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85719217-ffg8w" Oct 31 14:08:21.463364 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:21.463163 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85719217-ffg8w" event={"ID":"5f0de58d-6bbb-4c36-93c3-394defa7884b","Type":"ContainerDied","Data":"8f7c0a3d256ccdcb73ae1f060baf68f84031ba4a7d9ae3c34399b1a8e5f5fbae"} Oct 31 14:08:21.463364 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:21.463289 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f7c0a3d256ccdcb73ae1f060baf68f84031ba4a7d9ae3c34399b1a8e5f5fbae" Oct 31 14:08:27.668021 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:27.667975 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6c23f0c-8bbh5"] Oct 31 14:08:27.668846 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:27.668782 5693 cpu_manager.go:414] "containerMap: removing container" podUID="5f0de58d-6bbb-4c36-93c3-394defa7884b" containerName="revoke-keys" Oct 31 14:08:27.668846 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:27.668802 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f0de58d-6bbb-4c36-93c3-394defa7884b" containerName="revoke-keys" Oct 31 14:08:27.669274 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:27.669036 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f0de58d-6bbb-4c36-93c3-394defa7884b" containerName="revoke-keys" Oct 31 14:08:27.675024 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:27.675000 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c23f0c-8bbh5" Oct 31 14:08:27.679823 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:27.679795 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 14:08:27.679976 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:27.679835 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:08:27.680194 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:27.680166 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6c23f0c-8bbh5"] Oct 31 14:08:27.736533 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:27.736498 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a696330-db1b-490f-b47f-45adbaae2426-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6c23f0c-8bbh5\" (UID: \"7a696330-db1b-490f-b47f-45adbaae2426\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c23f0c-8bbh5" Oct 31 14:08:27.736739 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:27.736563 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cckw7\" (UniqueName: \"kubernetes.io/projected/7a696330-db1b-490f-b47f-45adbaae2426-kube-api-access-cckw7\") pod \"maas-api-revoke-sub-a6c23f0c-8bbh5\" (UID: \"7a696330-db1b-490f-b47f-45adbaae2426\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c23f0c-8bbh5" Oct 31 14:08:27.736739 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:27.736687 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a696330-db1b-490f-b47f-45adbaae2426-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6c23f0c-8bbh5\" (UID: \"7a696330-db1b-490f-b47f-45adbaae2426\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c23f0c-8bbh5" Oct 31 14:08:27.837663 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:27.837602 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a696330-db1b-490f-b47f-45adbaae2426-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6c23f0c-8bbh5\" (UID: \"7a696330-db1b-490f-b47f-45adbaae2426\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c23f0c-8bbh5" Oct 31 14:08:27.837845 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:27.837713 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a696330-db1b-490f-b47f-45adbaae2426-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6c23f0c-8bbh5\" (UID: \"7a696330-db1b-490f-b47f-45adbaae2426\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c23f0c-8bbh5" Oct 31 14:08:27.837845 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:27.837753 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cckw7\" (UniqueName: \"kubernetes.io/projected/7a696330-db1b-490f-b47f-45adbaae2426-kube-api-access-cckw7\") pod \"maas-api-revoke-sub-a6c23f0c-8bbh5\" (UID: \"7a696330-db1b-490f-b47f-45adbaae2426\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c23f0c-8bbh5" Oct 31 14:08:27.838292 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:27.838263 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a696330-db1b-490f-b47f-45adbaae2426-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6c23f0c-8bbh5\" (UID: \"7a696330-db1b-490f-b47f-45adbaae2426\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c23f0c-8bbh5" Oct 31 14:08:27.840205 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:27.840185 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a696330-db1b-490f-b47f-45adbaae2426-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6c23f0c-8bbh5\" (UID: \"7a696330-db1b-490f-b47f-45adbaae2426\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c23f0c-8bbh5" Oct 31 14:08:27.850053 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:27.850021 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cckw7\" (UniqueName: \"kubernetes.io/projected/7a696330-db1b-490f-b47f-45adbaae2426-kube-api-access-cckw7\") pod \"maas-api-revoke-sub-a6c23f0c-8bbh5\" (UID: \"7a696330-db1b-490f-b47f-45adbaae2426\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c23f0c-8bbh5" Oct 31 14:08:27.992130 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:27.992050 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c23f0c-8bbh5" Oct 31 14:08:28.267613 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:28.267576 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6c23f0c-8bbh5"] Oct 31 14:08:28.269889 ip-10-0-131-157 kubenswrapper[5693]: W1031 14:08:28.269860 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a696330_db1b_490f_b47f_45adbaae2426.slice/crio-9087534e380555b8b25a6fddb53cfc31e73d49cdfd7320668e2f2ccd77ffea16 WatchSource:0}: Error finding container 9087534e380555b8b25a6fddb53cfc31e73d49cdfd7320668e2f2ccd77ffea16: Status 404 returned error can't find the container with id 9087534e380555b8b25a6fddb53cfc31e73d49cdfd7320668e2f2ccd77ffea16 Oct 31 14:08:28.498956 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:28.498918 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c23f0c-8bbh5" event={"ID":"7a696330-db1b-490f-b47f-45adbaae2426","Type":"ContainerStarted","Data":"9087534e380555b8b25a6fddb53cfc31e73d49cdfd7320668e2f2ccd77ffea16"} Oct 31 14:08:29.534005 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:29.533968 5693 generic.go:356] "Generic (PLEG): container finished" podID="7a696330-db1b-490f-b47f-45adbaae2426" containerID="ad86fc56b5951733a65ccefffb6696c37c9c30b5f78f930348a758eb1b44029f" exitCode=0 Oct 31 14:08:29.538731 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:29.538699 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c23f0c-8bbh5" event={"ID":"7a696330-db1b-490f-b47f-45adbaae2426","Type":"ContainerDied","Data":"ad86fc56b5951733a65ccefffb6696c37c9c30b5f78f930348a758eb1b44029f"} Oct 31 14:08:30.678711 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:30.678688 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c23f0c-8bbh5" Oct 31 14:08:30.772387 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:30.772360 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a696330-db1b-490f-b47f-45adbaae2426-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a696330-db1b-490f-b47f-45adbaae2426-maas-api-xks-ca\") pod \"7a696330-db1b-490f-b47f-45adbaae2426\" (UID: \"7a696330-db1b-490f-b47f-45adbaae2426\") " Oct 31 14:08:30.772557 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:30.772420 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a696330-db1b-490f-b47f-45adbaae2426-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a696330-db1b-490f-b47f-45adbaae2426-maas-api-openshift-ca\") pod \"7a696330-db1b-490f-b47f-45adbaae2426\" (UID: \"7a696330-db1b-490f-b47f-45adbaae2426\") " Oct 31 14:08:30.772557 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:30.772501 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a696330-db1b-490f-b47f-45adbaae2426-kube-api-access-cckw7\" (UniqueName: \"kubernetes.io/projected/7a696330-db1b-490f-b47f-45adbaae2426-kube-api-access-cckw7\") pod \"7a696330-db1b-490f-b47f-45adbaae2426\" (UID: \"7a696330-db1b-490f-b47f-45adbaae2426\") " Oct 31 14:08:30.772814 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:30.772790 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a696330-db1b-490f-b47f-45adbaae2426-maas-api-openshift-ca" pod "7a696330-db1b-490f-b47f-45adbaae2426" (UID: "7a696330-db1b-490f-b47f-45adbaae2426"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:08:30.774742 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:30.774708 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a696330-db1b-490f-b47f-45adbaae2426-maas-api-xks-ca" pod "7a696330-db1b-490f-b47f-45adbaae2426" (UID: "7a696330-db1b-490f-b47f-45adbaae2426"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:08:30.774853 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:30.774751 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a696330-db1b-490f-b47f-45adbaae2426-kube-api-access-cckw7" pod "7a696330-db1b-490f-b47f-45adbaae2426" (UID: "7a696330-db1b-490f-b47f-45adbaae2426"). InnerVolumeSpecName "kube-api-access-cckw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:08:30.873386 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:30.873309 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cckw7\" (UniqueName: \"kubernetes.io/projected/7a696330-db1b-490f-b47f-45adbaae2426-kube-api-access-cckw7\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:08:30.873386 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:30.873338 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a696330-db1b-490f-b47f-45adbaae2426-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:08:30.873386 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:30.873347 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a696330-db1b-490f-b47f-45adbaae2426-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:08:31.550238 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:31.550210 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c23f0c-8bbh5" Oct 31 14:08:31.550423 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:31.550239 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c23f0c-8bbh5" event={"ID":"7a696330-db1b-490f-b47f-45adbaae2426","Type":"ContainerDied","Data":"9087534e380555b8b25a6fddb53cfc31e73d49cdfd7320668e2f2ccd77ffea16"} Oct 31 14:08:31.550423 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:31.550288 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9087534e380555b8b25a6fddb53cfc31e73d49cdfd7320668e2f2ccd77ffea16" Oct 31 14:08:37.898853 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:37.898818 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70872326-8klbv"] Oct 31 14:08:37.899244 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:37.899217 5693 cpu_manager.go:414] "containerMap: removing container" podUID="7a696330-db1b-490f-b47f-45adbaae2426" containerName="revoke-keys" Oct 31 14:08:37.899244 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:37.899234 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a696330-db1b-490f-b47f-45adbaae2426" containerName="revoke-keys" Oct 31 14:08:37.899341 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:37.899313 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a696330-db1b-490f-b47f-45adbaae2426" containerName="revoke-keys" Oct 31 14:08:37.902346 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:37.902317 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70872326-8klbv" Oct 31 14:08:37.908100 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:37.907901 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:08:37.909934 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:37.909909 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 14:08:37.913832 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:37.913793 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70872326-8klbv"] Oct 31 14:08:37.939174 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:37.939141 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/431e9a13-7749-4031-8fef-f8f7d6320b61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70872326-8klbv\" (UID: \"431e9a13-7749-4031-8fef-f8f7d6320b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70872326-8klbv" Oct 31 14:08:37.939327 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:37.939198 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6jlp\" (UniqueName: \"kubernetes.io/projected/431e9a13-7749-4031-8fef-f8f7d6320b61-kube-api-access-h6jlp\") pod \"maas-api-revoke-sub-70872326-8klbv\" (UID: \"431e9a13-7749-4031-8fef-f8f7d6320b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70872326-8klbv" Oct 31 14:08:37.939371 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:37.939339 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/431e9a13-7749-4031-8fef-f8f7d6320b61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70872326-8klbv\" (UID: \"431e9a13-7749-4031-8fef-f8f7d6320b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70872326-8klbv" Oct 31 14:08:38.040140 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:38.040106 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/431e9a13-7749-4031-8fef-f8f7d6320b61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70872326-8klbv\" (UID: \"431e9a13-7749-4031-8fef-f8f7d6320b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70872326-8klbv" Oct 31 14:08:38.040340 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:38.040197 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/431e9a13-7749-4031-8fef-f8f7d6320b61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70872326-8klbv\" (UID: \"431e9a13-7749-4031-8fef-f8f7d6320b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70872326-8klbv" Oct 31 14:08:38.040340 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:38.040229 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6jlp\" (UniqueName: \"kubernetes.io/projected/431e9a13-7749-4031-8fef-f8f7d6320b61-kube-api-access-h6jlp\") pod \"maas-api-revoke-sub-70872326-8klbv\" (UID: \"431e9a13-7749-4031-8fef-f8f7d6320b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70872326-8klbv" Oct 31 14:08:38.040795 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:38.040772 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/431e9a13-7749-4031-8fef-f8f7d6320b61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70872326-8klbv\" (UID: \"431e9a13-7749-4031-8fef-f8f7d6320b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70872326-8klbv" Oct 31 14:08:38.042701 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:38.042680 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/431e9a13-7749-4031-8fef-f8f7d6320b61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70872326-8klbv\" (UID: \"431e9a13-7749-4031-8fef-f8f7d6320b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70872326-8klbv" Oct 31 14:08:38.052584 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:38.052564 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6jlp\" (UniqueName: \"kubernetes.io/projected/431e9a13-7749-4031-8fef-f8f7d6320b61-kube-api-access-h6jlp\") pod \"maas-api-revoke-sub-70872326-8klbv\" (UID: \"431e9a13-7749-4031-8fef-f8f7d6320b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70872326-8klbv" Oct 31 14:08:38.218902 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:38.218813 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70872326-8klbv" Oct 31 14:08:38.494250 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:38.494225 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70872326-8klbv"] Oct 31 14:08:38.496476 ip-10-0-131-157 kubenswrapper[5693]: W1031 14:08:38.496446 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod431e9a13_7749_4031_8fef_f8f7d6320b61.slice/crio-6fd66cf0c91179f5367b2132cf33decb7faa8c27f0084531580bc631cf3f3654 WatchSource:0}: Error finding container 6fd66cf0c91179f5367b2132cf33decb7faa8c27f0084531580bc631cf3f3654: Status 404 returned error can't find the container with id 6fd66cf0c91179f5367b2132cf33decb7faa8c27f0084531580bc631cf3f3654 Oct 31 14:08:38.584817 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:38.584784 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70872326-8klbv" event={"ID":"431e9a13-7749-4031-8fef-f8f7d6320b61","Type":"ContainerStarted","Data":"6fd66cf0c91179f5367b2132cf33decb7faa8c27f0084531580bc631cf3f3654"} Oct 31 14:08:39.592780 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:39.592685 5693 generic.go:356] "Generic (PLEG): container finished" podID="431e9a13-7749-4031-8fef-f8f7d6320b61" containerID="c471061344f3aa338973f438c27bf8bee43fe90a089ae74752f763347943ed19" exitCode=0 Oct 31 14:08:39.592780 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:39.592726 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70872326-8klbv" event={"ID":"431e9a13-7749-4031-8fef-f8f7d6320b61","Type":"ContainerDied","Data":"c471061344f3aa338973f438c27bf8bee43fe90a089ae74752f763347943ed19"} Oct 31 14:08:40.971568 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:40.971540 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70872326-8klbv" Oct 31 14:08:41.075675 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:41.075617 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/431e9a13-7749-4031-8fef-f8f7d6320b61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/431e9a13-7749-4031-8fef-f8f7d6320b61-maas-api-xks-ca\") pod \"431e9a13-7749-4031-8fef-f8f7d6320b61\" (UID: \"431e9a13-7749-4031-8fef-f8f7d6320b61\") " Oct 31 14:08:41.085668 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:41.075955 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/431e9a13-7749-4031-8fef-f8f7d6320b61-kube-api-access-h6jlp\" (UniqueName: \"kubernetes.io/projected/431e9a13-7749-4031-8fef-f8f7d6320b61-kube-api-access-h6jlp\") pod \"431e9a13-7749-4031-8fef-f8f7d6320b61\" (UID: \"431e9a13-7749-4031-8fef-f8f7d6320b61\") " Oct 31 14:08:41.085668 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:41.076023 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/431e9a13-7749-4031-8fef-f8f7d6320b61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/431e9a13-7749-4031-8fef-f8f7d6320b61-maas-api-openshift-ca\") pod \"431e9a13-7749-4031-8fef-f8f7d6320b61\" (UID: \"431e9a13-7749-4031-8fef-f8f7d6320b61\") " Oct 31 14:08:41.086413 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:41.086092 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/431e9a13-7749-4031-8fef-f8f7d6320b61-maas-api-openshift-ca" pod "431e9a13-7749-4031-8fef-f8f7d6320b61" (UID: "431e9a13-7749-4031-8fef-f8f7d6320b61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:08:41.107849 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:41.107800 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/431e9a13-7749-4031-8fef-f8f7d6320b61-maas-api-xks-ca" pod "431e9a13-7749-4031-8fef-f8f7d6320b61" (UID: "431e9a13-7749-4031-8fef-f8f7d6320b61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:08:41.112146 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:41.112118 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/431e9a13-7749-4031-8fef-f8f7d6320b61-kube-api-access-h6jlp" pod "431e9a13-7749-4031-8fef-f8f7d6320b61" (UID: "431e9a13-7749-4031-8fef-f8f7d6320b61"). InnerVolumeSpecName "kube-api-access-h6jlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:08:41.193665 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:41.189271 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/431e9a13-7749-4031-8fef-f8f7d6320b61-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:08:41.193665 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:41.189312 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6jlp\" (UniqueName: \"kubernetes.io/projected/431e9a13-7749-4031-8fef-f8f7d6320b61-kube-api-access-h6jlp\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:08:41.193665 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:41.189331 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/431e9a13-7749-4031-8fef-f8f7d6320b61-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:08:41.623622 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:41.623587 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70872326-8klbv" event={"ID":"431e9a13-7749-4031-8fef-f8f7d6320b61","Type":"ContainerDied","Data":"6fd66cf0c91179f5367b2132cf33decb7faa8c27f0084531580bc631cf3f3654"} Oct 31 14:08:41.623831 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:41.623634 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fd66cf0c91179f5367b2132cf33decb7faa8c27f0084531580bc631cf3f3654" Oct 31 14:08:41.623831 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:41.623597 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70872326-8klbv" Oct 31 14:08:48.172502 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:48.172467 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04fd1e22-9fwmf"] Oct 31 14:08:48.172940 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:48.172912 5693 cpu_manager.go:414] "containerMap: removing container" podUID="431e9a13-7749-4031-8fef-f8f7d6320b61" containerName="revoke-keys" Oct 31 14:08:48.172940 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:48.172926 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="431e9a13-7749-4031-8fef-f8f7d6320b61" containerName="revoke-keys" Oct 31 14:08:48.173019 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:48.173002 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="431e9a13-7749-4031-8fef-f8f7d6320b61" containerName="revoke-keys" Oct 31 14:08:48.176046 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:48.176030 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fd1e22-9fwmf" Oct 31 14:08:48.185531 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:48.185499 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:08:48.185778 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:48.185694 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 14:08:48.186945 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:48.186920 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04fd1e22-9fwmf"] Oct 31 14:08:48.263382 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:48.263347 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10a5af21-791f-4678-81e3-a8c4d7f4f6d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04fd1e22-9fwmf\" (UID: \"10a5af21-791f-4678-81e3-a8c4d7f4f6d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fd1e22-9fwmf" Oct 31 14:08:48.263382 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:48.263388 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnvdh\" (UniqueName: \"kubernetes.io/projected/10a5af21-791f-4678-81e3-a8c4d7f4f6d6-kube-api-access-vnvdh\") pod \"maas-api-revoke-sub-04fd1e22-9fwmf\" (UID: \"10a5af21-791f-4678-81e3-a8c4d7f4f6d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fd1e22-9fwmf" Oct 31 14:08:48.263635 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:48.263488 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10a5af21-791f-4678-81e3-a8c4d7f4f6d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04fd1e22-9fwmf\" (UID: \"10a5af21-791f-4678-81e3-a8c4d7f4f6d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fd1e22-9fwmf" Oct 31 14:08:48.364182 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:48.364137 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10a5af21-791f-4678-81e3-a8c4d7f4f6d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04fd1e22-9fwmf\" (UID: \"10a5af21-791f-4678-81e3-a8c4d7f4f6d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fd1e22-9fwmf" Oct 31 14:08:48.364182 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:48.364186 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnvdh\" (UniqueName: \"kubernetes.io/projected/10a5af21-791f-4678-81e3-a8c4d7f4f6d6-kube-api-access-vnvdh\") pod \"maas-api-revoke-sub-04fd1e22-9fwmf\" (UID: \"10a5af21-791f-4678-81e3-a8c4d7f4f6d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fd1e22-9fwmf" Oct 31 14:08:48.364441 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:48.364233 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10a5af21-791f-4678-81e3-a8c4d7f4f6d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04fd1e22-9fwmf\" (UID: \"10a5af21-791f-4678-81e3-a8c4d7f4f6d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fd1e22-9fwmf" Oct 31 14:08:48.364863 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:48.364832 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10a5af21-791f-4678-81e3-a8c4d7f4f6d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04fd1e22-9fwmf\" (UID: \"10a5af21-791f-4678-81e3-a8c4d7f4f6d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fd1e22-9fwmf" Oct 31 14:08:48.366805 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:48.366786 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10a5af21-791f-4678-81e3-a8c4d7f4f6d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04fd1e22-9fwmf\" (UID: \"10a5af21-791f-4678-81e3-a8c4d7f4f6d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fd1e22-9fwmf" Oct 31 14:08:48.376953 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:48.376926 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnvdh\" (UniqueName: \"kubernetes.io/projected/10a5af21-791f-4678-81e3-a8c4d7f4f6d6-kube-api-access-vnvdh\") pod \"maas-api-revoke-sub-04fd1e22-9fwmf\" (UID: \"10a5af21-791f-4678-81e3-a8c4d7f4f6d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fd1e22-9fwmf" Oct 31 14:08:48.490316 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:48.490235 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fd1e22-9fwmf" Oct 31 14:08:48.685729 ip-10-0-131-157 kubenswrapper[5693]: E1031 14:08:48.685691 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 14:08:48.766696 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:48.766670 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04fd1e22-9fwmf"] Oct 31 14:08:48.767929 ip-10-0-131-157 kubenswrapper[5693]: W1031 14:08:48.767899 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a5af21_791f_4678_81e3_a8c4d7f4f6d6.slice/crio-4970f7c16583b678c5ed73e0ce3bfdaae6aae3eceadeedaf05e33eef3ee3e42b WatchSource:0}: Error finding container 4970f7c16583b678c5ed73e0ce3bfdaae6aae3eceadeedaf05e33eef3ee3e42b: Status 404 returned error can't find the container with id 4970f7c16583b678c5ed73e0ce3bfdaae6aae3eceadeedaf05e33eef3ee3e42b Oct 31 14:08:49.665417 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:49.665371 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fd1e22-9fwmf" event={"ID":"10a5af21-791f-4678-81e3-a8c4d7f4f6d6","Type":"ContainerStarted","Data":"b1b946db1d33fd8bf826704ce1a90949c5607e3e1fa5713dff0a5001b5556027"} Oct 31 14:08:49.665904 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:49.665487 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fd1e22-9fwmf" event={"ID":"10a5af21-791f-4678-81e3-a8c4d7f4f6d6","Type":"ContainerStarted","Data":"4970f7c16583b678c5ed73e0ce3bfdaae6aae3eceadeedaf05e33eef3ee3e42b"} Oct 31 14:08:50.675305 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:50.675266 5693 generic.go:356] "Generic (PLEG): container finished" podID="10a5af21-791f-4678-81e3-a8c4d7f4f6d6" containerID="b1b946db1d33fd8bf826704ce1a90949c5607e3e1fa5713dff0a5001b5556027" exitCode=0 Oct 31 14:08:50.675305 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:50.675314 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fd1e22-9fwmf" event={"ID":"10a5af21-791f-4678-81e3-a8c4d7f4f6d6","Type":"ContainerDied","Data":"b1b946db1d33fd8bf826704ce1a90949c5607e3e1fa5713dff0a5001b5556027"} Oct 31 14:08:51.833966 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:51.833939 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fd1e22-9fwmf" Oct 31 14:08:51.935608 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:51.935508 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10a5af21-791f-4678-81e3-a8c4d7f4f6d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10a5af21-791f-4678-81e3-a8c4d7f4f6d6-maas-api-openshift-ca\") pod \"10a5af21-791f-4678-81e3-a8c4d7f4f6d6\" (UID: \"10a5af21-791f-4678-81e3-a8c4d7f4f6d6\") " Oct 31 14:08:51.935813 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:51.935698 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10a5af21-791f-4678-81e3-a8c4d7f4f6d6-kube-api-access-vnvdh\" (UniqueName: \"kubernetes.io/projected/10a5af21-791f-4678-81e3-a8c4d7f4f6d6-kube-api-access-vnvdh\") pod \"10a5af21-791f-4678-81e3-a8c4d7f4f6d6\" (UID: \"10a5af21-791f-4678-81e3-a8c4d7f4f6d6\") " Oct 31 14:08:51.935813 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:51.935764 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10a5af21-791f-4678-81e3-a8c4d7f4f6d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10a5af21-791f-4678-81e3-a8c4d7f4f6d6-maas-api-xks-ca\") pod \"10a5af21-791f-4678-81e3-a8c4d7f4f6d6\" (UID: \"10a5af21-791f-4678-81e3-a8c4d7f4f6d6\") " Oct 31 14:08:51.935997 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:51.935976 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10a5af21-791f-4678-81e3-a8c4d7f4f6d6-maas-api-openshift-ca" pod "10a5af21-791f-4678-81e3-a8c4d7f4f6d6" (UID: "10a5af21-791f-4678-81e3-a8c4d7f4f6d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:08:51.936136 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:51.936115 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10a5af21-791f-4678-81e3-a8c4d7f4f6d6-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:08:51.937976 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:51.937952 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10a5af21-791f-4678-81e3-a8c4d7f4f6d6-maas-api-xks-ca" pod "10a5af21-791f-4678-81e3-a8c4d7f4f6d6" (UID: "10a5af21-791f-4678-81e3-a8c4d7f4f6d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:08:51.938046 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:51.937980 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10a5af21-791f-4678-81e3-a8c4d7f4f6d6-kube-api-access-vnvdh" pod "10a5af21-791f-4678-81e3-a8c4d7f4f6d6" (UID: "10a5af21-791f-4678-81e3-a8c4d7f4f6d6"). InnerVolumeSpecName "kube-api-access-vnvdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:08:52.037122 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:52.037068 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnvdh\" (UniqueName: \"kubernetes.io/projected/10a5af21-791f-4678-81e3-a8c4d7f4f6d6-kube-api-access-vnvdh\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:08:52.037122 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:52.037117 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10a5af21-791f-4678-81e3-a8c4d7f4f6d6-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:08:52.686619 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:52.686588 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fd1e22-9fwmf" Oct 31 14:08:52.686830 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:52.686633 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fd1e22-9fwmf" event={"ID":"10a5af21-791f-4678-81e3-a8c4d7f4f6d6","Type":"ContainerDied","Data":"4970f7c16583b678c5ed73e0ce3bfdaae6aae3eceadeedaf05e33eef3ee3e42b"} Oct 31 14:08:52.686830 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:52.686707 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4970f7c16583b678c5ed73e0ce3bfdaae6aae3eceadeedaf05e33eef3ee3e42b" Oct 31 14:08:58.428487 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:58.428446 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08e8a68b-zr7lm"] Oct 31 14:08:58.429121 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:58.428858 5693 cpu_manager.go:414] "containerMap: removing container" podUID="10a5af21-791f-4678-81e3-a8c4d7f4f6d6" containerName="revoke-keys" Oct 31 14:08:58.429121 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:58.428870 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10a5af21-791f-4678-81e3-a8c4d7f4f6d6" containerName="revoke-keys" Oct 31 14:08:58.429121 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:58.428975 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="10a5af21-791f-4678-81e3-a8c4d7f4f6d6" containerName="revoke-keys" Oct 31 14:08:58.432241 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:58.432219 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e8a68b-zr7lm" Oct 31 14:08:58.437767 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:58.437716 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:08:58.437934 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:58.437794 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 14:08:58.444149 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:58.444123 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08e8a68b-zr7lm"] Oct 31 14:08:58.602795 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:58.602764 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9tpv\" (UniqueName: \"kubernetes.io/projected/b4bc49af-5c7e-44e2-8c9f-e9bd79bad331-kube-api-access-p9tpv\") pod \"maas-api-revoke-sub-08e8a68b-zr7lm\" (UID: \"b4bc49af-5c7e-44e2-8c9f-e9bd79bad331\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e8a68b-zr7lm" Oct 31 14:08:58.602976 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:58.602902 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4bc49af-5c7e-44e2-8c9f-e9bd79bad331-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08e8a68b-zr7lm\" (UID: \"b4bc49af-5c7e-44e2-8c9f-e9bd79bad331\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e8a68b-zr7lm" Oct 31 14:08:58.602976 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:58.602928 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4bc49af-5c7e-44e2-8c9f-e9bd79bad331-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08e8a68b-zr7lm\" (UID: \"b4bc49af-5c7e-44e2-8c9f-e9bd79bad331\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e8a68b-zr7lm" Oct 31 14:08:58.703761 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:58.703662 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9tpv\" (UniqueName: \"kubernetes.io/projected/b4bc49af-5c7e-44e2-8c9f-e9bd79bad331-kube-api-access-p9tpv\") pod \"maas-api-revoke-sub-08e8a68b-zr7lm\" (UID: \"b4bc49af-5c7e-44e2-8c9f-e9bd79bad331\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e8a68b-zr7lm" Oct 31 14:08:58.703947 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:58.703772 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4bc49af-5c7e-44e2-8c9f-e9bd79bad331-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08e8a68b-zr7lm\" (UID: \"b4bc49af-5c7e-44e2-8c9f-e9bd79bad331\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e8a68b-zr7lm" Oct 31 14:08:58.703947 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:58.703804 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4bc49af-5c7e-44e2-8c9f-e9bd79bad331-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08e8a68b-zr7lm\" (UID: \"b4bc49af-5c7e-44e2-8c9f-e9bd79bad331\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e8a68b-zr7lm" Oct 31 14:08:58.704435 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:58.704413 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4bc49af-5c7e-44e2-8c9f-e9bd79bad331-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08e8a68b-zr7lm\" (UID: \"b4bc49af-5c7e-44e2-8c9f-e9bd79bad331\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e8a68b-zr7lm" Oct 31 14:08:58.706469 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:58.706448 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4bc49af-5c7e-44e2-8c9f-e9bd79bad331-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08e8a68b-zr7lm\" (UID: \"b4bc49af-5c7e-44e2-8c9f-e9bd79bad331\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e8a68b-zr7lm" Oct 31 14:08:58.716589 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:58.716567 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9tpv\" (UniqueName: \"kubernetes.io/projected/b4bc49af-5c7e-44e2-8c9f-e9bd79bad331-kube-api-access-p9tpv\") pod \"maas-api-revoke-sub-08e8a68b-zr7lm\" (UID: \"b4bc49af-5c7e-44e2-8c9f-e9bd79bad331\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e8a68b-zr7lm" Oct 31 14:08:58.746007 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:58.745986 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e8a68b-zr7lm" Oct 31 14:08:59.021613 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:59.021590 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08e8a68b-zr7lm"] Oct 31 14:08:59.023757 ip-10-0-131-157 kubenswrapper[5693]: W1031 14:08:59.023729 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4bc49af_5c7e_44e2_8c9f_e9bd79bad331.slice/crio-38ce39fe8bce87a0198d7d600e96c51899357a41fd9a494eaccd3fc7e1ad24e5 WatchSource:0}: Error finding container 38ce39fe8bce87a0198d7d600e96c51899357a41fd9a494eaccd3fc7e1ad24e5: Status 404 returned error can't find the container with id 38ce39fe8bce87a0198d7d600e96c51899357a41fd9a494eaccd3fc7e1ad24e5 Oct 31 14:08:59.725663 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:59.725054 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e8a68b-zr7lm" event={"ID":"b4bc49af-5c7e-44e2-8c9f-e9bd79bad331","Type":"ContainerStarted","Data":"02cb095380c2965edb3f878a11cf1df0b980bb98171da938b3ba6416a704f08a"} Oct 31 14:08:59.725663 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:59.725132 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e8a68b-zr7lm" event={"ID":"b4bc49af-5c7e-44e2-8c9f-e9bd79bad331","Type":"ContainerStarted","Data":"38ce39fe8bce87a0198d7d600e96c51899357a41fd9a494eaccd3fc7e1ad24e5"} Oct 31 14:08:59.752667 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:08:59.751614 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e8a68b-zr7lm" podStartSLOduration=1.751594355 podStartE2EDuration="1.751594355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:08:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:08:59.750208019 +0000 UTC m=+5790.457794318" watchObservedRunningTime="2026-10-31 14:08:59.751594355 +0000 UTC m=+5790.459180654" Oct 31 14:09:00.752980 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:00.752929 5693 generic.go:356] "Generic (PLEG): container finished" podID="b4bc49af-5c7e-44e2-8c9f-e9bd79bad331" containerID="02cb095380c2965edb3f878a11cf1df0b980bb98171da938b3ba6416a704f08a" exitCode=0 Oct 31 14:09:00.754027 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:00.754000 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e8a68b-zr7lm" event={"ID":"b4bc49af-5c7e-44e2-8c9f-e9bd79bad331","Type":"ContainerDied","Data":"02cb095380c2965edb3f878a11cf1df0b980bb98171da938b3ba6416a704f08a"} Oct 31 14:09:01.994869 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:01.994523 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e8a68b-zr7lm" Oct 31 14:09:02.061068 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:02.060976 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4bc49af-5c7e-44e2-8c9f-e9bd79bad331-kube-api-access-p9tpv\" (UniqueName: \"kubernetes.io/projected/b4bc49af-5c7e-44e2-8c9f-e9bd79bad331-kube-api-access-p9tpv\") pod \"b4bc49af-5c7e-44e2-8c9f-e9bd79bad331\" (UID: \"b4bc49af-5c7e-44e2-8c9f-e9bd79bad331\") " Oct 31 14:09:02.061068 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:02.061037 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4bc49af-5c7e-44e2-8c9f-e9bd79bad331-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4bc49af-5c7e-44e2-8c9f-e9bd79bad331-maas-api-xks-ca\") pod \"b4bc49af-5c7e-44e2-8c9f-e9bd79bad331\" (UID: \"b4bc49af-5c7e-44e2-8c9f-e9bd79bad331\") " Oct 31 14:09:02.063566 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:02.063533 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4bc49af-5c7e-44e2-8c9f-e9bd79bad331-kube-api-access-p9tpv" pod "b4bc49af-5c7e-44e2-8c9f-e9bd79bad331" (UID: "b4bc49af-5c7e-44e2-8c9f-e9bd79bad331"). InnerVolumeSpecName "kube-api-access-p9tpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:09:02.063748 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:02.063620 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4bc49af-5c7e-44e2-8c9f-e9bd79bad331-maas-api-xks-ca" pod "b4bc49af-5c7e-44e2-8c9f-e9bd79bad331" (UID: "b4bc49af-5c7e-44e2-8c9f-e9bd79bad331"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:09:02.162528 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:02.162493 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4bc49af-5c7e-44e2-8c9f-e9bd79bad331-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4bc49af-5c7e-44e2-8c9f-e9bd79bad331-maas-api-openshift-ca\") pod \"b4bc49af-5c7e-44e2-8c9f-e9bd79bad331\" (UID: \"b4bc49af-5c7e-44e2-8c9f-e9bd79bad331\") " Oct 31 14:09:02.162735 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:02.162722 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9tpv\" (UniqueName: \"kubernetes.io/projected/b4bc49af-5c7e-44e2-8c9f-e9bd79bad331-kube-api-access-p9tpv\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:09:02.162807 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:02.162738 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4bc49af-5c7e-44e2-8c9f-e9bd79bad331-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:09:02.162887 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:02.162866 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4bc49af-5c7e-44e2-8c9f-e9bd79bad331-maas-api-openshift-ca" pod "b4bc49af-5c7e-44e2-8c9f-e9bd79bad331" (UID: "b4bc49af-5c7e-44e2-8c9f-e9bd79bad331"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:09:02.263203 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:02.263172 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4bc49af-5c7e-44e2-8c9f-e9bd79bad331-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:09:02.768706 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:02.768675 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e8a68b-zr7lm" Oct 31 14:09:02.768917 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:02.768674 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e8a68b-zr7lm" event={"ID":"b4bc49af-5c7e-44e2-8c9f-e9bd79bad331","Type":"ContainerDied","Data":"38ce39fe8bce87a0198d7d600e96c51899357a41fd9a494eaccd3fc7e1ad24e5"} Oct 31 14:09:02.768917 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:02.768792 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38ce39fe8bce87a0198d7d600e96c51899357a41fd9a494eaccd3fc7e1ad24e5" Oct 31 14:09:08.621058 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:08.621026 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a1ea772-fz8vh"] Oct 31 14:09:08.621487 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:08.621429 5693 cpu_manager.go:414] "containerMap: removing container" podUID="b4bc49af-5c7e-44e2-8c9f-e9bd79bad331" containerName="revoke-keys" Oct 31 14:09:08.621487 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:08.621440 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4bc49af-5c7e-44e2-8c9f-e9bd79bad331" containerName="revoke-keys" Oct 31 14:09:08.621570 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:08.621520 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4bc49af-5c7e-44e2-8c9f-e9bd79bad331" containerName="revoke-keys" Oct 31 14:09:08.624872 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:08.624852 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1ea772-fz8vh" Oct 31 14:09:08.631704 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:08.631674 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:09:08.633250 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:08.633226 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 14:09:08.636477 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:08.636451 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a1ea772-fz8vh"] Oct 31 14:09:08.720726 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:08.720693 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmz8h\" (UniqueName: \"kubernetes.io/projected/fab68ac5-4445-4775-afad-050f248e165a-kube-api-access-lmz8h\") pod \"maas-api-revoke-sub-2a1ea772-fz8vh\" (UID: \"fab68ac5-4445-4775-afad-050f248e165a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1ea772-fz8vh" Oct 31 14:09:08.720931 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:08.720734 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fab68ac5-4445-4775-afad-050f248e165a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a1ea772-fz8vh\" (UID: \"fab68ac5-4445-4775-afad-050f248e165a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1ea772-fz8vh" Oct 31 14:09:08.720931 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:08.720818 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fab68ac5-4445-4775-afad-050f248e165a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a1ea772-fz8vh\" (UID: \"fab68ac5-4445-4775-afad-050f248e165a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1ea772-fz8vh" Oct 31 14:09:08.822229 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:08.822196 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmz8h\" (UniqueName: \"kubernetes.io/projected/fab68ac5-4445-4775-afad-050f248e165a-kube-api-access-lmz8h\") pod \"maas-api-revoke-sub-2a1ea772-fz8vh\" (UID: \"fab68ac5-4445-4775-afad-050f248e165a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1ea772-fz8vh" Oct 31 14:09:08.822433 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:08.822236 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fab68ac5-4445-4775-afad-050f248e165a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a1ea772-fz8vh\" (UID: \"fab68ac5-4445-4775-afad-050f248e165a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1ea772-fz8vh" Oct 31 14:09:08.822433 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:08.822266 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fab68ac5-4445-4775-afad-050f248e165a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a1ea772-fz8vh\" (UID: \"fab68ac5-4445-4775-afad-050f248e165a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1ea772-fz8vh" Oct 31 14:09:08.823032 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:08.823012 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fab68ac5-4445-4775-afad-050f248e165a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a1ea772-fz8vh\" (UID: \"fab68ac5-4445-4775-afad-050f248e165a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1ea772-fz8vh" Oct 31 14:09:08.824721 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:08.824704 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fab68ac5-4445-4775-afad-050f248e165a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a1ea772-fz8vh\" (UID: \"fab68ac5-4445-4775-afad-050f248e165a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1ea772-fz8vh" Oct 31 14:09:08.835084 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:08.835063 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmz8h\" (UniqueName: \"kubernetes.io/projected/fab68ac5-4445-4775-afad-050f248e165a-kube-api-access-lmz8h\") pod \"maas-api-revoke-sub-2a1ea772-fz8vh\" (UID: \"fab68ac5-4445-4775-afad-050f248e165a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1ea772-fz8vh" Oct 31 14:09:08.943589 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:08.943558 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1ea772-fz8vh" Oct 31 14:09:09.219283 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:09.219258 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a1ea772-fz8vh"] Oct 31 14:09:09.221451 ip-10-0-131-157 kubenswrapper[5693]: W1031 14:09:09.221423 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfab68ac5_4445_4775_afad_050f248e165a.slice/crio-74c1e8d65e565374f52c91da30b9ceb15625221b171f509446c3f680337d6c1f WatchSource:0}: Error finding container 74c1e8d65e565374f52c91da30b9ceb15625221b171f509446c3f680337d6c1f: Status 404 returned error can't find the container with id 74c1e8d65e565374f52c91da30b9ceb15625221b171f509446c3f680337d6c1f Oct 31 14:09:09.808106 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:09.808060 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1ea772-fz8vh" event={"ID":"fab68ac5-4445-4775-afad-050f248e165a","Type":"ContainerStarted","Data":"995b3049d7a011c1eb40eea6b79d4a0e53a3ad31eb5ebf870758b5401d4f8aa9"} Oct 31 14:09:09.808507 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:09.808139 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1ea772-fz8vh" event={"ID":"fab68ac5-4445-4775-afad-050f248e165a","Type":"ContainerStarted","Data":"74c1e8d65e565374f52c91da30b9ceb15625221b171f509446c3f680337d6c1f"} Oct 31 14:09:09.829535 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:09.829477 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1ea772-fz8vh" podStartSLOduration=1.8294600380000001 podStartE2EDuration="1.829460038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:09:09.825851646 +0000 UTC m=+5800.533437952" watchObservedRunningTime="2026-10-31 14:09:09.829460038 +0000 UTC m=+5800.537046335" Oct 31 14:09:10.846911 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:10.846875 5693 generic.go:356] "Generic (PLEG): container finished" podID="fab68ac5-4445-4775-afad-050f248e165a" containerID="995b3049d7a011c1eb40eea6b79d4a0e53a3ad31eb5ebf870758b5401d4f8aa9" exitCode=0 Oct 31 14:09:10.847463 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:10.846944 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1ea772-fz8vh" event={"ID":"fab68ac5-4445-4775-afad-050f248e165a","Type":"ContainerDied","Data":"995b3049d7a011c1eb40eea6b79d4a0e53a3ad31eb5ebf870758b5401d4f8aa9"} Oct 31 14:09:11.968910 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:11.968832 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fv482 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 14:09:11.968910 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:11.968901 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" podUID="c0bdd69c-691b-47b2-ba17-2e0ac7185643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 14:09:12.170378 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:12.170351 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1ea772-fz8vh" Oct 31 14:09:12.262368 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:12.262334 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fab68ac5-4445-4775-afad-050f248e165a-kube-api-access-lmz8h\" (UniqueName: \"kubernetes.io/projected/fab68ac5-4445-4775-afad-050f248e165a-kube-api-access-lmz8h\") pod \"fab68ac5-4445-4775-afad-050f248e165a\" (UID: \"fab68ac5-4445-4775-afad-050f248e165a\") " Oct 31 14:09:12.262588 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:12.262435 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fab68ac5-4445-4775-afad-050f248e165a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fab68ac5-4445-4775-afad-050f248e165a-maas-api-openshift-ca\") pod \"fab68ac5-4445-4775-afad-050f248e165a\" (UID: \"fab68ac5-4445-4775-afad-050f248e165a\") " Oct 31 14:09:12.262588 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:12.262481 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fab68ac5-4445-4775-afad-050f248e165a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fab68ac5-4445-4775-afad-050f248e165a-maas-api-xks-ca\") pod \"fab68ac5-4445-4775-afad-050f248e165a\" (UID: \"fab68ac5-4445-4775-afad-050f248e165a\") " Oct 31 14:09:12.262850 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:12.262815 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fab68ac5-4445-4775-afad-050f248e165a-maas-api-openshift-ca" pod "fab68ac5-4445-4775-afad-050f248e165a" (UID: "fab68ac5-4445-4775-afad-050f248e165a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:09:12.264880 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:12.264848 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fab68ac5-4445-4775-afad-050f248e165a-maas-api-xks-ca" pod "fab68ac5-4445-4775-afad-050f248e165a" (UID: "fab68ac5-4445-4775-afad-050f248e165a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:09:12.265025 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:12.264855 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fab68ac5-4445-4775-afad-050f248e165a-kube-api-access-lmz8h" pod "fab68ac5-4445-4775-afad-050f248e165a" (UID: "fab68ac5-4445-4775-afad-050f248e165a"). InnerVolumeSpecName "kube-api-access-lmz8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:09:12.363712 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:12.363628 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmz8h\" (UniqueName: \"kubernetes.io/projected/fab68ac5-4445-4775-afad-050f248e165a-kube-api-access-lmz8h\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:09:12.363712 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:12.363710 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fab68ac5-4445-4775-afad-050f248e165a-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:09:12.363712 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:12.363721 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fab68ac5-4445-4775-afad-050f248e165a-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:09:12.862433 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:12.862394 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1ea772-fz8vh" Oct 31 14:09:12.862433 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:12.862425 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1ea772-fz8vh" event={"ID":"fab68ac5-4445-4775-afad-050f248e165a","Type":"ContainerDied","Data":"74c1e8d65e565374f52c91da30b9ceb15625221b171f509446c3f680337d6c1f"} Oct 31 14:09:12.862681 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:12.862496 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74c1e8d65e565374f52c91da30b9ceb15625221b171f509446c3f680337d6c1f" Oct 31 14:09:18.687005 ip-10-0-131-157 kubenswrapper[5693]: E1031 14:09:18.686971 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 14:09:18.853252 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:18.853221 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c942d9e3-gj6rg"] Oct 31 14:09:18.853654 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:18.853626 5693 cpu_manager.go:414] "containerMap: removing container" podUID="fab68ac5-4445-4775-afad-050f248e165a" containerName="revoke-keys" Oct 31 14:09:18.853707 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:18.853659 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fab68ac5-4445-4775-afad-050f248e165a" containerName="revoke-keys" Oct 31 14:09:18.853756 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:18.853746 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="fab68ac5-4445-4775-afad-050f248e165a" containerName="revoke-keys" Oct 31 14:09:18.856786 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:18.856770 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c942d9e3-gj6rg" Oct 31 14:09:18.863057 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:18.862599 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:09:18.864318 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:18.864287 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 14:09:18.867346 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:18.867324 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c942d9e3-gj6rg"] Oct 31 14:09:18.920437 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:18.920405 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c10bf276-e22f-4a40-bf07-d2d4486fe634-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c942d9e3-gj6rg\" (UID: \"c10bf276-e22f-4a40-bf07-d2d4486fe634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c942d9e3-gj6rg" Oct 31 14:09:18.920613 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:18.920444 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c10bf276-e22f-4a40-bf07-d2d4486fe634-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c942d9e3-gj6rg\" (UID: \"c10bf276-e22f-4a40-bf07-d2d4486fe634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c942d9e3-gj6rg" Oct 31 14:09:18.920613 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:18.920467 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2nhq\" (UniqueName: \"kubernetes.io/projected/c10bf276-e22f-4a40-bf07-d2d4486fe634-kube-api-access-l2nhq\") pod \"maas-api-revoke-sub-c942d9e3-gj6rg\" (UID: \"c10bf276-e22f-4a40-bf07-d2d4486fe634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c942d9e3-gj6rg" Oct 31 14:09:19.021314 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:19.021229 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c10bf276-e22f-4a40-bf07-d2d4486fe634-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c942d9e3-gj6rg\" (UID: \"c10bf276-e22f-4a40-bf07-d2d4486fe634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c942d9e3-gj6rg" Oct 31 14:09:19.021314 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:19.021270 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2nhq\" (UniqueName: \"kubernetes.io/projected/c10bf276-e22f-4a40-bf07-d2d4486fe634-kube-api-access-l2nhq\") pod \"maas-api-revoke-sub-c942d9e3-gj6rg\" (UID: \"c10bf276-e22f-4a40-bf07-d2d4486fe634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c942d9e3-gj6rg" Oct 31 14:09:19.021522 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:19.021356 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c10bf276-e22f-4a40-bf07-d2d4486fe634-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c942d9e3-gj6rg\" (UID: \"c10bf276-e22f-4a40-bf07-d2d4486fe634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c942d9e3-gj6rg" Oct 31 14:09:19.022033 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:19.022006 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c10bf276-e22f-4a40-bf07-d2d4486fe634-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c942d9e3-gj6rg\" (UID: \"c10bf276-e22f-4a40-bf07-d2d4486fe634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c942d9e3-gj6rg" Oct 31 14:09:19.023824 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:19.023800 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c10bf276-e22f-4a40-bf07-d2d4486fe634-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c942d9e3-gj6rg\" (UID: \"c10bf276-e22f-4a40-bf07-d2d4486fe634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c942d9e3-gj6rg" Oct 31 14:09:19.033871 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:19.033850 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2nhq\" (UniqueName: \"kubernetes.io/projected/c10bf276-e22f-4a40-bf07-d2d4486fe634-kube-api-access-l2nhq\") pod \"maas-api-revoke-sub-c942d9e3-gj6rg\" (UID: \"c10bf276-e22f-4a40-bf07-d2d4486fe634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c942d9e3-gj6rg" Oct 31 14:09:19.171806 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:19.171767 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c942d9e3-gj6rg" Oct 31 14:09:19.447001 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:19.446974 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c942d9e3-gj6rg"] Oct 31 14:09:19.449069 ip-10-0-131-157 kubenswrapper[5693]: W1031 14:09:19.449039 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc10bf276_e22f_4a40_bf07_d2d4486fe634.slice/crio-ca891c210b54581467ddae0ad4289d9314d6f82b05c9ac03ae82255db2d315a1 WatchSource:0}: Error finding container ca891c210b54581467ddae0ad4289d9314d6f82b05c9ac03ae82255db2d315a1: Status 404 returned error can't find the container with id ca891c210b54581467ddae0ad4289d9314d6f82b05c9ac03ae82255db2d315a1 Oct 31 14:09:19.899704 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:19.899667 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c942d9e3-gj6rg" event={"ID":"c10bf276-e22f-4a40-bf07-d2d4486fe634","Type":"ContainerStarted","Data":"fe2da91dd20e0f896742396e90561bd0931b692dfe5942019ce6be72aaa73173"} Oct 31 14:09:19.900237 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:19.899726 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c942d9e3-gj6rg" event={"ID":"c10bf276-e22f-4a40-bf07-d2d4486fe634","Type":"ContainerStarted","Data":"ca891c210b54581467ddae0ad4289d9314d6f82b05c9ac03ae82255db2d315a1"} Oct 31 14:09:19.918198 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:19.918137 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c942d9e3-gj6rg" podStartSLOduration=1.918118314 podStartE2EDuration="1.918118314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:09:19.917302843 +0000 UTC m=+5810.624889140" watchObservedRunningTime="2026-10-31 14:09:19.918118314 +0000 UTC m=+5810.625704610" Oct 31 14:09:20.932785 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:20.932728 5693 generic.go:356] "Generic (PLEG): container finished" podID="c10bf276-e22f-4a40-bf07-d2d4486fe634" containerID="fe2da91dd20e0f896742396e90561bd0931b692dfe5942019ce6be72aaa73173" exitCode=0 Oct 31 14:09:20.933559 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:20.933530 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c942d9e3-gj6rg" event={"ID":"c10bf276-e22f-4a40-bf07-d2d4486fe634","Type":"ContainerDied","Data":"fe2da91dd20e0f896742396e90561bd0931b692dfe5942019ce6be72aaa73173"} Oct 31 14:09:22.271271 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:22.271247 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c942d9e3-gj6rg" Oct 31 14:09:22.362068 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:22.361825 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c10bf276-e22f-4a40-bf07-d2d4486fe634-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c10bf276-e22f-4a40-bf07-d2d4486fe634-maas-api-xks-ca\") pod \"c10bf276-e22f-4a40-bf07-d2d4486fe634\" (UID: \"c10bf276-e22f-4a40-bf07-d2d4486fe634\") " Oct 31 14:09:22.362068 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:22.361882 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c10bf276-e22f-4a40-bf07-d2d4486fe634-kube-api-access-l2nhq\" (UniqueName: \"kubernetes.io/projected/c10bf276-e22f-4a40-bf07-d2d4486fe634-kube-api-access-l2nhq\") pod \"c10bf276-e22f-4a40-bf07-d2d4486fe634\" (UID: \"c10bf276-e22f-4a40-bf07-d2d4486fe634\") " Oct 31 14:09:22.362068 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:22.362029 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c10bf276-e22f-4a40-bf07-d2d4486fe634-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c10bf276-e22f-4a40-bf07-d2d4486fe634-maas-api-openshift-ca\") pod \"c10bf276-e22f-4a40-bf07-d2d4486fe634\" (UID: \"c10bf276-e22f-4a40-bf07-d2d4486fe634\") " Oct 31 14:09:22.363459 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:22.362748 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c10bf276-e22f-4a40-bf07-d2d4486fe634-maas-api-openshift-ca" pod "c10bf276-e22f-4a40-bf07-d2d4486fe634" (UID: "c10bf276-e22f-4a40-bf07-d2d4486fe634"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:09:22.366028 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:22.365997 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c10bf276-e22f-4a40-bf07-d2d4486fe634-maas-api-xks-ca" pod "c10bf276-e22f-4a40-bf07-d2d4486fe634" (UID: "c10bf276-e22f-4a40-bf07-d2d4486fe634"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:09:22.366444 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:22.366419 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c10bf276-e22f-4a40-bf07-d2d4486fe634-kube-api-access-l2nhq" pod "c10bf276-e22f-4a40-bf07-d2d4486fe634" (UID: "c10bf276-e22f-4a40-bf07-d2d4486fe634"). InnerVolumeSpecName "kube-api-access-l2nhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:09:22.463665 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:22.463606 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c10bf276-e22f-4a40-bf07-d2d4486fe634-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:09:22.463866 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:22.463673 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c10bf276-e22f-4a40-bf07-d2d4486fe634-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:09:22.463866 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:22.463689 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2nhq\" (UniqueName: \"kubernetes.io/projected/c10bf276-e22f-4a40-bf07-d2d4486fe634-kube-api-access-l2nhq\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:09:22.952594 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:22.952560 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c942d9e3-gj6rg" event={"ID":"c10bf276-e22f-4a40-bf07-d2d4486fe634","Type":"ContainerDied","Data":"ca891c210b54581467ddae0ad4289d9314d6f82b05c9ac03ae82255db2d315a1"} Oct 31 14:09:22.952796 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:22.952598 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c942d9e3-gj6rg" Oct 31 14:09:22.952796 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:22.952609 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca891c210b54581467ddae0ad4289d9314d6f82b05c9ac03ae82255db2d315a1" Oct 31 14:09:29.071892 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:29.071855 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e5de419-pxq76"] Oct 31 14:09:29.072309 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:29.072290 5693 cpu_manager.go:414] "containerMap: removing container" podUID="c10bf276-e22f-4a40-bf07-d2d4486fe634" containerName="revoke-keys" Oct 31 14:09:29.072309 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:29.072304 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c10bf276-e22f-4a40-bf07-d2d4486fe634" containerName="revoke-keys" Oct 31 14:09:29.072400 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:29.072387 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="c10bf276-e22f-4a40-bf07-d2d4486fe634" containerName="revoke-keys" Oct 31 14:09:29.075823 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:29.075799 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5de419-pxq76" Oct 31 14:09:29.083324 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:29.082902 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:09:29.083324 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:29.082920 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 14:09:29.086714 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:29.086679 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e5de419-pxq76"] Oct 31 14:09:29.126276 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:29.126243 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b3ba40a-7371-467d-8061-e184db3c6f25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e5de419-pxq76\" (UID: \"6b3ba40a-7371-467d-8061-e184db3c6f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5de419-pxq76" Oct 31 14:09:29.126276 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:29.126283 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b3ba40a-7371-467d-8061-e184db3c6f25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e5de419-pxq76\" (UID: \"6b3ba40a-7371-467d-8061-e184db3c6f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5de419-pxq76" Oct 31 14:09:29.126533 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:29.126389 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5xz8\" (UniqueName: \"kubernetes.io/projected/6b3ba40a-7371-467d-8061-e184db3c6f25-kube-api-access-n5xz8\") pod \"maas-api-revoke-sub-7e5de419-pxq76\" (UID: \"6b3ba40a-7371-467d-8061-e184db3c6f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5de419-pxq76" Oct 31 14:09:29.227359 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:29.227329 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5xz8\" (UniqueName: \"kubernetes.io/projected/6b3ba40a-7371-467d-8061-e184db3c6f25-kube-api-access-n5xz8\") pod \"maas-api-revoke-sub-7e5de419-pxq76\" (UID: \"6b3ba40a-7371-467d-8061-e184db3c6f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5de419-pxq76" Oct 31 14:09:29.227572 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:29.227397 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b3ba40a-7371-467d-8061-e184db3c6f25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e5de419-pxq76\" (UID: \"6b3ba40a-7371-467d-8061-e184db3c6f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5de419-pxq76" Oct 31 14:09:29.227572 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:29.227420 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b3ba40a-7371-467d-8061-e184db3c6f25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e5de419-pxq76\" (UID: \"6b3ba40a-7371-467d-8061-e184db3c6f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5de419-pxq76" Oct 31 14:09:29.228114 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:29.228090 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b3ba40a-7371-467d-8061-e184db3c6f25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e5de419-pxq76\" (UID: \"6b3ba40a-7371-467d-8061-e184db3c6f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5de419-pxq76" Oct 31 14:09:29.229851 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:29.229828 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b3ba40a-7371-467d-8061-e184db3c6f25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e5de419-pxq76\" (UID: \"6b3ba40a-7371-467d-8061-e184db3c6f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5de419-pxq76" Oct 31 14:09:29.239659 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:29.239623 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5xz8\" (UniqueName: \"kubernetes.io/projected/6b3ba40a-7371-467d-8061-e184db3c6f25-kube-api-access-n5xz8\") pod \"maas-api-revoke-sub-7e5de419-pxq76\" (UID: \"6b3ba40a-7371-467d-8061-e184db3c6f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5de419-pxq76" Oct 31 14:09:29.390909 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:29.390828 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5de419-pxq76" Oct 31 14:09:29.668848 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:29.668825 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e5de419-pxq76"] Oct 31 14:09:29.670801 ip-10-0-131-157 kubenswrapper[5693]: W1031 14:09:29.670778 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b3ba40a_7371_467d_8061_e184db3c6f25.slice/crio-9176b2f739f96f5c5b1922821d456a8d61ca55bf3c5edc17c34069f6de760e8f WatchSource:0}: Error finding container 9176b2f739f96f5c5b1922821d456a8d61ca55bf3c5edc17c34069f6de760e8f: Status 404 returned error can't find the container with id 9176b2f739f96f5c5b1922821d456a8d61ca55bf3c5edc17c34069f6de760e8f Oct 31 14:09:29.994566 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:29.994526 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5de419-pxq76" event={"ID":"6b3ba40a-7371-467d-8061-e184db3c6f25","Type":"ContainerStarted","Data":"9176b2f739f96f5c5b1922821d456a8d61ca55bf3c5edc17c34069f6de760e8f"} Oct 31 14:09:31.048598 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:31.048165 5693 generic.go:356] "Generic (PLEG): container finished" podID="6b3ba40a-7371-467d-8061-e184db3c6f25" containerID="32c2eef1a0dbbabb5ea017d7c887dafe85359caa7112b21e5999f9ca6a0480bd" exitCode=0 Oct 31 14:09:31.048598 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:31.048211 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5de419-pxq76" event={"ID":"6b3ba40a-7371-467d-8061-e184db3c6f25","Type":"ContainerDied","Data":"32c2eef1a0dbbabb5ea017d7c887dafe85359caa7112b21e5999f9ca6a0480bd"} Oct 31 14:09:32.263218 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:32.262825 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-95zp2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 14:09:32.263218 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:32.262893 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-95zp2" podUID="f942c1e3-8982-4e26-9c30-82ac4265a1eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 14:09:32.449666 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:32.448920 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5de419-pxq76" Oct 31 14:09:32.466340 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:32.466313 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b3ba40a-7371-467d-8061-e184db3c6f25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b3ba40a-7371-467d-8061-e184db3c6f25-maas-api-xks-ca\") pod \"6b3ba40a-7371-467d-8061-e184db3c6f25\" (UID: \"6b3ba40a-7371-467d-8061-e184db3c6f25\") " Oct 31 14:09:32.466539 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:32.466366 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b3ba40a-7371-467d-8061-e184db3c6f25-kube-api-access-n5xz8\" (UniqueName: \"kubernetes.io/projected/6b3ba40a-7371-467d-8061-e184db3c6f25-kube-api-access-n5xz8\") pod \"6b3ba40a-7371-467d-8061-e184db3c6f25\" (UID: \"6b3ba40a-7371-467d-8061-e184db3c6f25\") " Oct 31 14:09:32.466539 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:32.466428 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b3ba40a-7371-467d-8061-e184db3c6f25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b3ba40a-7371-467d-8061-e184db3c6f25-maas-api-openshift-ca\") pod \"6b3ba40a-7371-467d-8061-e184db3c6f25\" (UID: \"6b3ba40a-7371-467d-8061-e184db3c6f25\") " Oct 31 14:09:32.466889 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:32.466867 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b3ba40a-7371-467d-8061-e184db3c6f25-maas-api-openshift-ca" pod "6b3ba40a-7371-467d-8061-e184db3c6f25" (UID: "6b3ba40a-7371-467d-8061-e184db3c6f25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:09:32.468885 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:32.468855 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b3ba40a-7371-467d-8061-e184db3c6f25-maas-api-xks-ca" pod "6b3ba40a-7371-467d-8061-e184db3c6f25" (UID: "6b3ba40a-7371-467d-8061-e184db3c6f25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:09:32.469060 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:32.469039 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b3ba40a-7371-467d-8061-e184db3c6f25-kube-api-access-n5xz8" pod "6b3ba40a-7371-467d-8061-e184db3c6f25" (UID: "6b3ba40a-7371-467d-8061-e184db3c6f25"). InnerVolumeSpecName "kube-api-access-n5xz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:09:32.567437 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:32.567407 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b3ba40a-7371-467d-8061-e184db3c6f25-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:09:32.567437 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:32.567435 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5xz8\" (UniqueName: \"kubernetes.io/projected/6b3ba40a-7371-467d-8061-e184db3c6f25-kube-api-access-n5xz8\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:09:32.567437 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:32.567445 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b3ba40a-7371-467d-8061-e184db3c6f25-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:09:33.062350 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:33.062319 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5de419-pxq76" Oct 31 14:09:33.062528 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:33.062322 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5de419-pxq76" event={"ID":"6b3ba40a-7371-467d-8061-e184db3c6f25","Type":"ContainerDied","Data":"9176b2f739f96f5c5b1922821d456a8d61ca55bf3c5edc17c34069f6de760e8f"} Oct 31 14:09:33.062528 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:33.062473 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9176b2f739f96f5c5b1922821d456a8d61ca55bf3c5edc17c34069f6de760e8f" Oct 31 14:09:39.298370 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:39.298335 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c5efdfe-5n92f"] Oct 31 14:09:39.298915 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:39.298895 5693 cpu_manager.go:414] "containerMap: removing container" podUID="6b3ba40a-7371-467d-8061-e184db3c6f25" containerName="revoke-keys" Oct 31 14:09:39.299013 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:39.298918 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b3ba40a-7371-467d-8061-e184db3c6f25" containerName="revoke-keys" Oct 31 14:09:39.299074 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:39.299056 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b3ba40a-7371-467d-8061-e184db3c6f25" containerName="revoke-keys" Oct 31 14:09:39.302377 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:39.302356 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5efdfe-5n92f" Oct 31 14:09:39.310492 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:39.309014 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:09:39.310492 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:39.309119 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 14:09:39.311739 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:39.311712 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c5efdfe-5n92f"] Oct 31 14:09:39.331531 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:39.331493 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67d7d10f-bcae-4c92-aaf9-8243055dd450-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c5efdfe-5n92f\" (UID: \"67d7d10f-bcae-4c92-aaf9-8243055dd450\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5efdfe-5n92f" Oct 31 14:09:39.331749 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:39.331556 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67d7d10f-bcae-4c92-aaf9-8243055dd450-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c5efdfe-5n92f\" (UID: \"67d7d10f-bcae-4c92-aaf9-8243055dd450\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5efdfe-5n92f" Oct 31 14:09:39.331749 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:39.331659 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87std\" (UniqueName: \"kubernetes.io/projected/67d7d10f-bcae-4c92-aaf9-8243055dd450-kube-api-access-87std\") pod \"maas-api-revoke-sub-4c5efdfe-5n92f\" (UID: \"67d7d10f-bcae-4c92-aaf9-8243055dd450\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5efdfe-5n92f" Oct 31 14:09:39.432968 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:39.432935 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87std\" (UniqueName: \"kubernetes.io/projected/67d7d10f-bcae-4c92-aaf9-8243055dd450-kube-api-access-87std\") pod \"maas-api-revoke-sub-4c5efdfe-5n92f\" (UID: \"67d7d10f-bcae-4c92-aaf9-8243055dd450\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5efdfe-5n92f" Oct 31 14:09:39.433162 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:39.433099 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67d7d10f-bcae-4c92-aaf9-8243055dd450-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c5efdfe-5n92f\" (UID: \"67d7d10f-bcae-4c92-aaf9-8243055dd450\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5efdfe-5n92f" Oct 31 14:09:39.433162 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:39.433155 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67d7d10f-bcae-4c92-aaf9-8243055dd450-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c5efdfe-5n92f\" (UID: \"67d7d10f-bcae-4c92-aaf9-8243055dd450\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5efdfe-5n92f" Oct 31 14:09:39.433882 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:39.433854 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67d7d10f-bcae-4c92-aaf9-8243055dd450-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c5efdfe-5n92f\" (UID: \"67d7d10f-bcae-4c92-aaf9-8243055dd450\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5efdfe-5n92f" Oct 31 14:09:39.435708 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:39.435690 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67d7d10f-bcae-4c92-aaf9-8243055dd450-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c5efdfe-5n92f\" (UID: \"67d7d10f-bcae-4c92-aaf9-8243055dd450\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5efdfe-5n92f" Oct 31 14:09:39.445225 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:39.445204 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87std\" (UniqueName: \"kubernetes.io/projected/67d7d10f-bcae-4c92-aaf9-8243055dd450-kube-api-access-87std\") pod \"maas-api-revoke-sub-4c5efdfe-5n92f\" (UID: \"67d7d10f-bcae-4c92-aaf9-8243055dd450\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5efdfe-5n92f" Oct 31 14:09:39.618346 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:39.618260 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5efdfe-5n92f" Oct 31 14:09:39.894468 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:39.894443 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c5efdfe-5n92f"] Oct 31 14:09:39.896708 ip-10-0-131-157 kubenswrapper[5693]: W1031 14:09:39.896681 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d7d10f_bcae_4c92_aaf9_8243055dd450.slice/crio-b93394e052e166f6e6a5262090bc2eaed92dc7d48c50f421062521d3b518b63e WatchSource:0}: Error finding container b93394e052e166f6e6a5262090bc2eaed92dc7d48c50f421062521d3b518b63e: Status 404 returned error can't find the container with id b93394e052e166f6e6a5262090bc2eaed92dc7d48c50f421062521d3b518b63e Oct 31 14:09:40.100965 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:40.100928 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5efdfe-5n92f" event={"ID":"67d7d10f-bcae-4c92-aaf9-8243055dd450","Type":"ContainerStarted","Data":"b93394e052e166f6e6a5262090bc2eaed92dc7d48c50f421062521d3b518b63e"} Oct 31 14:09:41.126865 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:41.126129 5693 generic.go:356] "Generic (PLEG): container finished" podID="67d7d10f-bcae-4c92-aaf9-8243055dd450" containerID="cfb104b0503fe4c29ba80fe64c00f455f1816584442ecb751e67d7ff547051f2" exitCode=0 Oct 31 14:09:41.126865 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:41.126266 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5efdfe-5n92f" event={"ID":"67d7d10f-bcae-4c92-aaf9-8243055dd450","Type":"ContainerDied","Data":"cfb104b0503fe4c29ba80fe64c00f455f1816584442ecb751e67d7ff547051f2"} Oct 31 14:09:41.962823 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:41.962781 5693 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fv482 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 14:09:41.963044 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:41.962841 5693 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fv482" podUID="c0bdd69c-691b-47b2-ba17-2e0ac7185643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 14:09:42.266743 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:42.266713 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5efdfe-5n92f" Oct 31 14:09:42.371503 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:42.371466 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67d7d10f-bcae-4c92-aaf9-8243055dd450-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67d7d10f-bcae-4c92-aaf9-8243055dd450-maas-api-openshift-ca\") pod \"67d7d10f-bcae-4c92-aaf9-8243055dd450\" (UID: \"67d7d10f-bcae-4c92-aaf9-8243055dd450\") " Oct 31 14:09:42.371712 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:42.371520 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67d7d10f-bcae-4c92-aaf9-8243055dd450-kube-api-access-87std\" (UniqueName: \"kubernetes.io/projected/67d7d10f-bcae-4c92-aaf9-8243055dd450-kube-api-access-87std\") pod \"67d7d10f-bcae-4c92-aaf9-8243055dd450\" (UID: \"67d7d10f-bcae-4c92-aaf9-8243055dd450\") " Oct 31 14:09:42.371712 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:42.371603 5693 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67d7d10f-bcae-4c92-aaf9-8243055dd450-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67d7d10f-bcae-4c92-aaf9-8243055dd450-maas-api-xks-ca\") pod \"67d7d10f-bcae-4c92-aaf9-8243055dd450\" (UID: \"67d7d10f-bcae-4c92-aaf9-8243055dd450\") " Oct 31 14:09:42.371974 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:42.371948 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67d7d10f-bcae-4c92-aaf9-8243055dd450-maas-api-openshift-ca" pod "67d7d10f-bcae-4c92-aaf9-8243055dd450" (UID: "67d7d10f-bcae-4c92-aaf9-8243055dd450"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:09:42.373905 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:42.373883 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67d7d10f-bcae-4c92-aaf9-8243055dd450-maas-api-xks-ca" pod "67d7d10f-bcae-4c92-aaf9-8243055dd450" (UID: "67d7d10f-bcae-4c92-aaf9-8243055dd450"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:09:42.373905 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:42.373889 5693 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d7d10f-bcae-4c92-aaf9-8243055dd450-kube-api-access-87std" pod "67d7d10f-bcae-4c92-aaf9-8243055dd450" (UID: "67d7d10f-bcae-4c92-aaf9-8243055dd450"). InnerVolumeSpecName "kube-api-access-87std". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:09:42.472516 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:42.472484 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67d7d10f-bcae-4c92-aaf9-8243055dd450-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:09:42.472516 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:42.472509 5693 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87std\" (UniqueName: \"kubernetes.io/projected/67d7d10f-bcae-4c92-aaf9-8243055dd450-kube-api-access-87std\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:09:42.472516 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:42.472519 5693 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67d7d10f-bcae-4c92-aaf9-8243055dd450-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:09:43.137121 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:43.137085 5693 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5efdfe-5n92f" Oct 31 14:09:43.137121 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:43.137116 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5efdfe-5n92f" event={"ID":"67d7d10f-bcae-4c92-aaf9-8243055dd450","Type":"ContainerDied","Data":"b93394e052e166f6e6a5262090bc2eaed92dc7d48c50f421062521d3b518b63e"} Oct 31 14:09:43.137358 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:43.137189 5693 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b93394e052e166f6e6a5262090bc2eaed92dc7d48c50f421062521d3b518b63e" Oct 31 14:09:48.686415 ip-10-0-131-157 kubenswrapper[5693]: E1031 14:09:48.686382 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 14:09:59.749759 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:09:59.749720 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-9c9ccc756-vngkm_56840e04-370d-44ef-92a1-46d4bb15f9ec/authorino/0.log" Oct 31 14:10:11.530515 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:11.530427 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6df7b9f889-66dn6_5c3a62d1-ddd7-4459-8342-2ed0eedd3d8d/manager/0.log" Oct 31 14:10:13.045284 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:13.045254 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-2wppt_75062797-634c-4d93-9c96-7c0060e0f684/proxy/0.log" Oct 31 14:10:14.360820 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:14.360792 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-04fd1e22-9fwmf_10a5af21-791f-4678-81e3-a8c4d7f4f6d6/revoke-keys/0.log" Oct 31 14:10:14.531262 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:14.531235 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-08e8a68b-zr7lm_b4bc49af-5c7e-44e2-8c9f-e9bd79bad331/revoke-keys/0.log" Oct 31 14:10:14.704470 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:14.704436 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-11881524-dvn9m_208b7c2f-0a32-4cfc-8100-26764ae45e26/revoke-keys/0.log" Oct 31 14:10:14.878558 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:14.878531 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-1def42e9-989cb_09b4b740-e954-4876-909f-04f206aa9162/revoke-keys/0.log" Oct 31 14:10:15.047517 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:15.047428 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-1e8be16b-cjzpw_4a718057-f877-459f-a205-18857d1e7f0c/revoke-keys/0.log" Oct 31 14:10:15.221511 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:15.221477 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2a1ea772-fz8vh_fab68ac5-4445-4775-afad-050f248e165a/revoke-keys/0.log" Oct 31 14:10:15.400573 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:15.400541 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-36d45a9d-8ffwn_70e9e98e-ef42-4867-bd8a-e89747807a13/revoke-keys/0.log" Oct 31 14:10:15.575530 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:15.575506 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-4c5efdfe-5n92f_67d7d10f-bcae-4c92-aaf9-8243055dd450/revoke-keys/0.log" Oct 31 14:10:15.743269 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:15.743191 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-5b74cb0f-4p8v5_be4e85f9-a566-4083-b068-5a9ce3222bc6/revoke-keys/0.log" Oct 31 14:10:15.908121 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:15.908095 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-70872326-8klbv_431e9a13-7749-4031-8fef-f8f7d6320b61/revoke-keys/0.log" Oct 31 14:10:16.080967 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:16.080891 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7e5de419-pxq76_6b3ba40a-7371-467d-8061-e184db3c6f25/revoke-keys/0.log" Oct 31 14:10:16.257234 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:16.257210 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-85719217-ffg8w_5f0de58d-6bbb-4c36-93c3-394defa7884b/revoke-keys/0.log" Oct 31 14:10:16.433330 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:16.433306 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a6c23f0c-8bbh5_7a696330-db1b-490f-b47f-45adbaae2426/revoke-keys/0.log" Oct 31 14:10:16.602604 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:16.602572 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-abecc70c-k6hx5_c05c1144-fffc-47f3-8b46-9b708b7d3c5d/revoke-keys/0.log" Oct 31 14:10:16.785733 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:16.785632 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c6f2c4d4-852rr_3b3398fa-0028-4349-abd3-99d1cfe8247c/revoke-keys/0.log" Oct 31 14:10:16.962467 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:16.962436 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c942d9e3-gj6rg_c10bf276-e22f-4a40-bf07-d2d4486fe634/revoke-keys/0.log" Oct 31 14:10:17.142516 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:17.142419 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d96f6927-dh89l_59907327-7199-4288-bd92-b64520eeff73/revoke-keys/0.log" Oct 31 14:10:17.324905 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:17.324876 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f9dcc22e-hcz8n_909bf5d0-6f7f-487d-9a4a-b3db681d664b/revoke-keys/0.log" Oct 31 14:10:17.495169 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:17.495140 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-fa2f15ae-zlgnk_5c302de7-718b-45a7-8be3-2ecc34622c5d/revoke-keys/0.log" Oct 31 14:10:17.688929 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:17.688796 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-rqmck_b8aff5f5-e566-4b4d-9360-a33187a1119c/postgres/0.log" Oct 31 14:10:18.685719 ip-10-0-131-157 kubenswrapper[5693]: E1031 14:10:18.685677 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 14:10:19.566062 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:19.566032 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-9c9ccc756-vngkm_56840e04-370d-44ef-92a1-46d4bb15f9ec/authorino/0.log" Oct 31 14:10:20.150831 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:20.150800 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-b2vpf_86e98514-c296-467f-ba33-cbcdf689b48b/kuadrant-console-plugin/0.log" Oct 31 14:10:21.951305 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:21.951270 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-6zvmg_f02c2b48-73f3-45a7-a329-1b797c681c6a/discovery/0.log" Oct 31 14:10:22.134470 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:22.134438 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54cf69b775-b227x_42b4c824-4d0e-4551-869e-8f369356e2d9/kube-auth-proxy/0.log" Oct 31 14:10:22.869220 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:22.869187 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fc8c56547-95zp2_f942c1e3-8982-4e26-9c30-82ac4265a1eb/istio-proxy/0.log" Oct 31 14:10:23.055897 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:23.055855 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fc8c56547-bc5lr_073cc2ff-2dc7-4376-8449-f45cd637eeea/istio-proxy/0.log" Oct 31 14:10:23.645318 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:23.645280 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fc8c56547-fv482_c0bdd69c-691b-47b2-ba17-2e0ac7185643/istio-proxy/0.log" Oct 31 14:10:25.937070 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:25.937020 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-wzbx5_adaef1d1-7d48-4517-af6b-0a5ec03b958c/main/0.log" Oct 31 14:10:26.116769 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:26.116737 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-j8sln_f6597c0b-d1c0-4000-b098-520ea8eae932/main/0.log" Oct 31 14:10:26.471600 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:26.471568 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-lfzgn_80aab26c-df22-4263-bc1e-4e066eae2903/main/0.log" Oct 31 14:10:26.645188 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:26.645156 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85tlqg_61487058-ab9d-4537-ab76-cf431dd17a7f/main/0.log" Oct 31 14:10:27.187801 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:27.187763 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fwm_66c6ec5a-c0da-4bd7-a314-0c90ccec2425/main/0.log" Oct 31 14:10:33.024050 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:33.024013 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6f2c4d4-852rr"] Oct 31 14:10:33.026144 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:33.026109 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6f2c4d4-852rr"] Oct 31 14:10:33.406884 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:33.406852 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b3398fa-0028-4349-abd3-99d1cfe8247c" path="/var/lib/kubelet/pods/3b3398fa-0028-4349-abd3-99d1cfe8247c/volumes" Oct 31 14:10:42.230921 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:42.230885 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wgrhs_ed7130e5-e8ea-4ade-97ba-2c7b7880aa7a/global-pull-secret-syncer/0.log" Oct 31 14:10:42.564913 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:42.564830 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-crhzp_43421968-0e5b-4e27-be79-104aae5f72e0/konnectivity-agent/0.log" Oct 31 14:10:42.591620 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:42.591590 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal_72de75e4a93b82a0133be1d6ed9e0a51/haproxy/0.log" Oct 31 14:10:43.021962 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:43.021935 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9dcc22e-hcz8n"] Oct 31 14:10:43.025155 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:43.025131 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9dcc22e-hcz8n"] Oct 31 14:10:43.405978 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:43.405948 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="909bf5d0-6f7f-487d-9a4a-b3db681d664b" path="/var/lib/kubelet/pods/909bf5d0-6f7f-487d-9a4a-b3db681d664b/volumes" Oct 31 14:10:48.686100 ip-10-0-131-157 kubenswrapper[5693]: E1031 14:10:48.686063 5693 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9t68r" containerName="extractor" Oct 31 14:10:49.366134 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:49.366101 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-9c9ccc756-vngkm_56840e04-370d-44ef-92a1-46d4bb15f9ec/authorino/0.log" Oct 31 14:10:49.885370 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:49.885340 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-b2vpf_86e98514-c296-467f-ba33-cbcdf689b48b/kuadrant-console-plugin/0.log" Oct 31 14:10:52.186379 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:52.186348 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-847b6d5877-sxkd8_25a02fd6-8ab0-4cca-94cd-d4082f874170/metrics-server/0.log" Oct 31 14:10:52.213617 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:52.213591 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-zbjvn_3015bfb0-80c8-4c01-9053-85e5ff2318fb/monitoring-plugin/0.log" Oct 31 14:10:52.424480 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:52.424451 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gqmhl_4b6af1c5-6045-4e4c-aa87-2473d19bd000/node-exporter/0.log" Oct 31 14:10:52.465361 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:52.465292 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gqmhl_4b6af1c5-6045-4e4c-aa87-2473d19bd000/kube-rbac-proxy/0.log" Oct 31 14:10:52.491152 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:52.491131 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gqmhl_4b6af1c5-6045-4e4c-aa87-2473d19bd000/init-textfile/0.log" Oct 31 14:10:52.861177 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:52.861107 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-227cc_ddebd541-4d2f-4861-9dd0-0ee66b03aaf2/prometheus-operator/0.log" Oct 31 14:10:52.878831 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:52.878804 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-227cc_ddebd541-4d2f-4861-9dd0-0ee66b03aaf2/kube-rbac-proxy/0.log" Oct 31 14:10:52.943759 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:52.943729 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f7b7cfbff-7br98_c49bcfc5-1bc7-46fc-85ca-5be1fa784116/telemeter-client/0.log" Oct 31 14:10:52.973333 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:52.973306 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f7b7cfbff-7br98_c49bcfc5-1bc7-46fc-85ca-5be1fa784116/reload/0.log" Oct 31 14:10:53.011194 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:53.011167 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f7b7cfbff-7br98_c49bcfc5-1bc7-46fc-85ca-5be1fa784116/kube-rbac-proxy/0.log" Oct 31 14:10:53.630202 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:53.630171 5693 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jr74b/perf-node-gather-daemonset-zsdph"] Oct 31 14:10:53.630722 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:53.630697 5693 cpu_manager.go:414] "containerMap: removing container" podUID="67d7d10f-bcae-4c92-aaf9-8243055dd450" containerName="revoke-keys" Oct 31 14:10:53.630722 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:53.630721 5693 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67d7d10f-bcae-4c92-aaf9-8243055dd450" containerName="revoke-keys" Oct 31 14:10:53.630978 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:53.630850 5693 memory_manager.go:365] "RemoveStaleState removing state" podUID="67d7d10f-bcae-4c92-aaf9-8243055dd450" containerName="revoke-keys" Oct 31 14:10:53.634376 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:53.634356 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-zsdph" Oct 31 14:10:53.643215 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:53.643192 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jr74b/perf-node-gather-daemonset-zsdph"] Oct 31 14:10:53.703443 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:53.703404 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jr74b\"/\"openshift-service-ca.crt\"" Oct 31 14:10:53.703613 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:53.703512 5693 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jr74b\"/\"kube-root-ca.crt\"" Oct 31 14:10:53.732398 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:53.732374 5693 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jr74b\"/\"default-dockercfg-c88j5\"" Oct 31 14:10:53.806848 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:53.806813 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/20fc708c-bb08-4638-b7b4-b4e4a42fde72-podres\") pod \"perf-node-gather-daemonset-zsdph\" (UID: \"20fc708c-bb08-4638-b7b4-b4e4a42fde72\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-zsdph" Oct 31 14:10:53.807018 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:53.806874 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/20fc708c-bb08-4638-b7b4-b4e4a42fde72-lib-modules\") pod \"perf-node-gather-daemonset-zsdph\" (UID: \"20fc708c-bb08-4638-b7b4-b4e4a42fde72\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-zsdph" Oct 31 14:10:53.807018 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:53.806942 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20fc708c-bb08-4638-b7b4-b4e4a42fde72-sys\") pod \"perf-node-gather-daemonset-zsdph\" (UID: \"20fc708c-bb08-4638-b7b4-b4e4a42fde72\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-zsdph" Oct 31 14:10:53.807018 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:53.807006 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb8nw\" (UniqueName: \"kubernetes.io/projected/20fc708c-bb08-4638-b7b4-b4e4a42fde72-kube-api-access-tb8nw\") pod \"perf-node-gather-daemonset-zsdph\" (UID: \"20fc708c-bb08-4638-b7b4-b4e4a42fde72\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-zsdph" Oct 31 14:10:53.807134 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:53.807028 5693 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/20fc708c-bb08-4638-b7b4-b4e4a42fde72-proc\") pod \"perf-node-gather-daemonset-zsdph\" (UID: \"20fc708c-bb08-4638-b7b4-b4e4a42fde72\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-zsdph" Oct 31 14:10:53.907923 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:53.907889 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/20fc708c-bb08-4638-b7b4-b4e4a42fde72-lib-modules\") pod \"perf-node-gather-daemonset-zsdph\" (UID: \"20fc708c-bb08-4638-b7b4-b4e4a42fde72\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-zsdph" Oct 31 14:10:53.908094 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:53.907929 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20fc708c-bb08-4638-b7b4-b4e4a42fde72-sys\") pod \"perf-node-gather-daemonset-zsdph\" (UID: \"20fc708c-bb08-4638-b7b4-b4e4a42fde72\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-zsdph" Oct 31 14:10:53.908094 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:53.907975 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb8nw\" (UniqueName: \"kubernetes.io/projected/20fc708c-bb08-4638-b7b4-b4e4a42fde72-kube-api-access-tb8nw\") pod \"perf-node-gather-daemonset-zsdph\" (UID: \"20fc708c-bb08-4638-b7b4-b4e4a42fde72\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-zsdph" Oct 31 14:10:53.908094 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:53.907994 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/20fc708c-bb08-4638-b7b4-b4e4a42fde72-proc\") pod \"perf-node-gather-daemonset-zsdph\" (UID: \"20fc708c-bb08-4638-b7b4-b4e4a42fde72\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-zsdph" Oct 31 14:10:53.908094 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:53.908034 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/20fc708c-bb08-4638-b7b4-b4e4a42fde72-lib-modules\") pod \"perf-node-gather-daemonset-zsdph\" (UID: \"20fc708c-bb08-4638-b7b4-b4e4a42fde72\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-zsdph" Oct 31 14:10:53.908094 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:53.908050 5693 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/20fc708c-bb08-4638-b7b4-b4e4a42fde72-podres\") pod \"perf-node-gather-daemonset-zsdph\" (UID: \"20fc708c-bb08-4638-b7b4-b4e4a42fde72\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-zsdph" Oct 31 14:10:53.908094 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:53.908085 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20fc708c-bb08-4638-b7b4-b4e4a42fde72-sys\") pod \"perf-node-gather-daemonset-zsdph\" (UID: \"20fc708c-bb08-4638-b7b4-b4e4a42fde72\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-zsdph" Oct 31 14:10:53.908359 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:53.908114 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/20fc708c-bb08-4638-b7b4-b4e4a42fde72-podres\") pod \"perf-node-gather-daemonset-zsdph\" (UID: \"20fc708c-bb08-4638-b7b4-b4e4a42fde72\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-zsdph" Oct 31 14:10:53.908359 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:53.908123 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/20fc708c-bb08-4638-b7b4-b4e4a42fde72-proc\") pod \"perf-node-gather-daemonset-zsdph\" (UID: \"20fc708c-bb08-4638-b7b4-b4e4a42fde72\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-zsdph" Oct 31 14:10:53.920458 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:53.920434 5693 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb8nw\" (UniqueName: \"kubernetes.io/projected/20fc708c-bb08-4638-b7b4-b4e4a42fde72-kube-api-access-tb8nw\") pod \"perf-node-gather-daemonset-zsdph\" (UID: \"20fc708c-bb08-4638-b7b4-b4e4a42fde72\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-zsdph" Oct 31 14:10:53.949158 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:53.949128 5693 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-zsdph" Oct 31 14:10:54.226785 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:54.226762 5693 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jr74b/perf-node-gather-daemonset-zsdph"] Oct 31 14:10:54.228200 ip-10-0-131-157 kubenswrapper[5693]: W1031 14:10:54.228172 5693 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod20fc708c_bb08_4638_b7b4_b4e4a42fde72.slice/crio-dbe514f165947c9796dd7bdf9037514d550f92b9036cad77bca79b8db5c88145 WatchSource:0}: Error finding container dbe514f165947c9796dd7bdf9037514d550f92b9036cad77bca79b8db5c88145: Status 404 returned error can't find the container with id dbe514f165947c9796dd7bdf9037514d550f92b9036cad77bca79b8db5c88145 Oct 31 14:10:54.298135 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:54.298112 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6df7b9f889-66dn6_5c3a62d1-ddd7-4459-8342-2ed0eedd3d8d/manager/0.log" Oct 31 14:10:54.549030 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:54.548938 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-zsdph" event={"ID":"20fc708c-bb08-4638-b7b4-b4e4a42fde72","Type":"ContainerStarted","Data":"1d09053a9b1ad48cdba52160fa787d15ee14b6fb2f56d29ccb743ab0088ef093"} Oct 31 14:10:54.549030 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:54.549020 5693 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-zsdph" event={"ID":"20fc708c-bb08-4638-b7b4-b4e4a42fde72","Type":"ContainerStarted","Data":"dbe514f165947c9796dd7bdf9037514d550f92b9036cad77bca79b8db5c88145"} Oct 31 14:10:54.549237 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:54.549044 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-zsdph" Oct 31 14:10:54.567894 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:54.567835 5693 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-zsdph" podStartSLOduration=1.5678216360000001 podStartE2EDuration="1.567821636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 14:10:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:10:54.566451561 +0000 UTC m=+5905.274037856" watchObservedRunningTime="2026-10-31 14:10:54.567821636 +0000 UTC m=+5905.275407930" Oct 31 14:10:56.366051 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:56.366020 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-84bd68cdfb-b2vg6_84a1aeec-6740-42f9-b7da-83148de0f73a/console/0.log" Oct 31 14:10:58.328557 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:58.328511 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-2wppt_75062797-634c-4d93-9c96-7c0060e0f684/proxy/0.log" Oct 31 14:10:59.109149 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:59.109120 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bdwh8_f4a19150-beb0-4289-91ee-b1668968e6ca/dns/0.log" Oct 31 14:10:59.140460 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:59.140421 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bdwh8_f4a19150-beb0-4289-91ee-b1668968e6ca/kube-rbac-proxy/0.log" Oct 31 14:10:59.223055 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:59.223027 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5gdlb_26913ad4-6613-4f3f-9819-9be238088c0a/dns-node-resolver/0.log" Oct 31 14:10:59.540801 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:10:59.540751 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6df7b9f889-66dn6_5c3a62d1-ddd7-4459-8342-2ed0eedd3d8d/manager/0.log" Oct 31 14:11:00.127247 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:11:00.127216 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tk5vl_14dc1e6d-70e1-4ba5-bffb-d9299433756d/node-ca/0.log" Oct 31 14:11:00.564863 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:11:00.564832 5693 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-zsdph" Oct 31 14:11:02.025193 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:11:02.025156 5693 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11881524-dvn9m"] Oct 31 14:11:02.027461 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:11:02.027439 5693 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11881524-dvn9m"] Oct 31 14:11:02.926180 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:11:02.926145 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-6zvmg_f02c2b48-73f3-45a7-a329-1b797c681c6a/discovery/0.log" Oct 31 14:11:03.295499 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:11:03.295413 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54cf69b775-b227x_42b4c824-4d0e-4551-869e-8f369356e2d9/kube-auth-proxy/0.log" Oct 31 14:11:03.406467 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:11:03.406430 5693 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="208b7c2f-0a32-4cfc-8100-26764ae45e26" path="/var/lib/kubelet/pods/208b7c2f-0a32-4cfc-8100-26764ae45e26/volumes" Oct 31 14:11:03.490179 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:11:03.490153 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fc8c56547-95zp2_f942c1e3-8982-4e26-9c30-82ac4265a1eb/istio-proxy/0.log" Oct 31 14:11:03.525599 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:11:03.525567 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fc8c56547-bc5lr_073cc2ff-2dc7-4376-8449-f45cd637eeea/istio-proxy/0.log" Oct 31 14:11:03.680410 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:11:03.680382 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fc8c56547-fv482_c0bdd69c-691b-47b2-ba17-2e0ac7185643/istio-proxy/0.log" Oct 31 14:11:03.990778 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:11:03.990700 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-2wppt_75062797-634c-4d93-9c96-7c0060e0f684/proxy/0.log" Oct 31 14:11:05.258516 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:11:05.258485 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6df7b9f889-66dn6_5c3a62d1-ddd7-4459-8342-2ed0eedd3d8d/manager/0.log" Oct 31 14:11:07.541119 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:11:07.541093 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pwtgm_45870e34-5e37-443a-826b-81c9c031051c/serve-healthcheck-canary/0.log" Oct 31 14:11:08.381608 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:11:08.381560 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9t68r_97cf3735-d794-4628-9048-252f949317b9/kube-rbac-proxy/0.log" Oct 31 14:11:08.409249 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:11:08.409219 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9t68r_97cf3735-d794-4628-9048-252f949317b9/exporter/0.log" Oct 31 14:11:08.444296 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:11:08.444266 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9t68r_97cf3735-d794-4628-9048-252f949317b9/extractor/0.log" Oct 31 14:11:08.508090 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:11:08.508061 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-6zvmg_f02c2b48-73f3-45a7-a329-1b797c681c6a/discovery/0.log" Oct 31 14:11:08.519839 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:11:08.519817 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54cf69b775-b227x_42b4c824-4d0e-4551-869e-8f369356e2d9/kube-auth-proxy/0.log" Oct 31 14:11:08.621939 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:11:08.621914 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fc8c56547-95zp2_f942c1e3-8982-4e26-9c30-82ac4265a1eb/istio-proxy/0.log" Oct 31 14:11:08.643815 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:11:08.643790 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fc8c56547-bc5lr_073cc2ff-2dc7-4376-8449-f45cd637eeea/istio-proxy/0.log" Oct 31 14:11:08.717889 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:11:08.717849 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fc8c56547-fv482_c0bdd69c-691b-47b2-ba17-2e0ac7185643/istio-proxy/0.log" Oct 31 14:11:09.467971 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:11:09.467940 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-2wppt_75062797-634c-4d93-9c96-7c0060e0f684/proxy/0.log" Oct 31 14:11:11.994076 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:11:11.994042 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-5c44b469d4-vd8dj_3335d434-c4ef-4f47-90d2-d0cc63f3204c/manager/0.log" Oct 31 14:11:12.018437 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:11:12.018410 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-5c44b469d4-vd8dj_3335d434-c4ef-4f47-90d2-d0cc63f3204c/copy-manifests/0.log" Oct 31 14:11:12.589820 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:11:12.589787 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-6zvmg_f02c2b48-73f3-45a7-a329-1b797c681c6a/discovery/0.log" Oct 31 14:11:12.610601 ip-10-0-131-157 kubenswrapper[5693]: I1031 14:11:12.610581 5693 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54cf69b775-b227x_42b4c824-4d0e-4551-869e-8f369356e2d9/kube-auth-proxy/0.log"